In this dissertation the first published real-time implementation of a 16-QAM transmission system with FPGA-based DSP is presented. 2.5 Gb/s coherent 16-QAM data has been optically transmitted over 20 and 100 km and synchronously received by heterodyning in a real-time I&Q receiver, with BER below the threshold of a state-of-the-art FEC (7% overhead). Two techniques of feed-forward carrier phase recovery (Blind Phase Search (BPS) and QPSK partitioning (QPSKP)) were tested in a real-time transmission experiment and compared with each other. The influence of the required resolution of the analog-to-digital converter (ADC) has been investigated, which is a challenge of real-time coherent transmission systems. The influence of phase noise in 16-QAM, which is mainly contributed from laser sources, optical amplifiers, and nonlinear effects in optical fibers is also shown. Moreover, different operation points of a 16-QAM modulator were tested in real-time and an optimal condition is found which minimizes the BER.