The BICEP/Keck experiment (BK) is a series of small-aperture refracting telescopes observing degree-scale Cosmic Microwave Background (CMB) polarization from the South Pole in search of a primordial $B$-mode signature. This $B$-mode signal arises from primordial gravitational waves interacting with the CMB, and has amplitude parametrized by the tensor-to-scalar ratio $r$. Since 2016, BICEP3 and the Keck Array have been observing with 4800 total antenna-coupled transition-edge sensor detectors, with frequency bands spanning 95, 150, 220, and 270 GHz. Here we present the optical performance of these receivers from 2016 to 2019, including far-field beams measured in situ with an improved chopped thermal source and instrument spectral response measured with a field-deployable Fourier Transform Spectrometer. As a pair differencing experiment, an important systematic that must be controlled is the differential beam response between the co-located, orthogonally polarized detectors. We generate per-detector far-field beam maps and the corresponding differential beam mismatch that is used to estimate the temperature-to-polarization leakage in our CMB maps and to give feedback on detector and optics fabrication. The differential beam parameters presented here were estimated using improved low-level beam map analysis techniques, including efficient removal of non-Gaussian noise as well as improved spatial masking. These techniques help minimize systematic uncertainty in the beam analysis, with the goal of constraining the bias on $r$ induced by temperature-to-polarization leakage to be subdominant to the statistical uncertainty. This is essential as we progress to higher detector counts in the next generation of CMB experiments.