Digital coherent transceivers have revolutionized optical fiber communications due to their superior performance offered compared to intensity modulation and direct detection based alternatives. As systems employing digital coherent transceivers seek to approach their information theoretic capacity, the use of multilevel modulation formats combined with appropriate forward error correction becomes essential. Given this context, in this tutorial paper, we therefore explore the digital signal processing (DSP) utilized in a coherent transceiver with a focus on multilevel modulation formats. By way of an introduction, we open by discussing the photonic technology required to realize a coherent transceiver. After discussing this interface between the analog optical channel and the digital domain, the rest of the paper is focused on DSP. We begin by discussing algorithms that correct for imperfections in the optical to digital conversion, including IQ imbalance and timing skew. Next, we discuss channel equalization including means for their realization for both quasi-static and dynamic channel impairments. Synchronization algorithms that correct for the difference between the transmitter and receiver oscillators both optical and electrical are then discussed and issues associated with symbol decoding highlighted. For most of the cases, we start with polarization division multiplexed quadrature phase-shift keying (PDM-QPSK) format as a basis and then discuss the extension to allow for high order multilevel formats. Finally, we conclude by discussing some of the open research challenges in the field.