High speed optical links suffer from inter-symbol-interference (ISI) due to their limited bandwidth. Equalisation is typically used to mitigate ISI and therefore improve the link capacity. This dissertation explores novel equalisation techniques for carrierless amplitude and phase (CAP) modulation based optical communication systems including OM4 based and plastic optical fibre (POF) based links. An 850 nm VCSEL based OM4 link using CAP-16 scheme is studied. For the first time, the CAP equaliser, is proposed to mitigate both crosstalk channel interference (CCI) and ISI in the link at the receiver side. Performance comparisons are studied between the CAP-16 scheme using CAP equaliser and a conventional equaliser, pulse amplitude modulation (PAM-4) scheme, and discrete multitone (DMT) scheme. CAP based data transmission of 112 Gb/s is achieved over 150 m OM4 fibre with this novel equaliser, while the conventional equaliser can only support over 1 m OM4 fibre and fails to recover the signals at the same data rate. In addition, this novel equaliser provides a 1.2 dB and 1.7 dB improvement in receiver sensitivity over PAM-4 and DMT schemes, respectively, at 112 Gb/s over 100 m OM4 fibre. A novel pre-CAP-equaliser solving CCI at the transmitter side is also proposed. Data transmission of 56 Gb/s over 100 m OM4 fibre is reported experimentally with an improvement of 0.7 dB in receiver sensitivity compared to using the CAP equaliser at the receiver side. A simulation study shows a 2 dB improvement in receiver sensitivity at 112 Gb/s over 100 m OM4 fibre. Furthermore, an artificial neural network (ANN) equaliser in conjunction with the CAP equaliser structure is explored in a VCSEL based OM4 fibre link in order to further mitigate the nonlinear impairments. For 112 Gb/s data transmission over 100 m OM4 fibre, a 2.4 dB improvement of receiver sensitivity is achieved compared to the CAP equaliser. In addition to the electrical equalisers, a monolithically integrated silicon optical equaliser consisting of three taps is used for 50 Gb/s data transmission. After 10 km standard single mode fibre (SSMF), error free eye diagrams at the receiver are demonstrated.
A μLED based POF link based on an APD receiver is also investigated with the CAP equaliser at the receiver side. Data transmission rates of 4 Gb/s over 25 m and 5 Gb/s over 10 m POF links are demonstrated with this equaliser while the conventional equaliser can only support 4 Gb/s over 10 m and fails to recover the signals for 5 Gb/s data transmission.