Herein, we report on the charge dynamics of photovoltaic devices based on two novel small-molecule nonfullerene acceptors featuring a central ketone unit. Using ultrafast near-infrared spectroscopy with optical and photocurrent detection methods, we identify one of the key loss channels in the devices as geminate recombination (GR) of interfacial charge transfer states (CTSs). We find that the magnitude of GR is highly sensitive to the choice of solvent and annealing conditions. Interestingly, regardless of these processing conditions, the same lifetime for GR (∼130 ps) is obtained by both detection methods upon decomposing the complex broadband transient optical spectra, suggesting this time scale is inherent and independent of morphology. These observations suggest that the CTSs in the studied material blends are mostly strongly bound, and that charge generation from these states is highly inefficient. We further rationalize our results by considering the impact of the processing on the morphology of the mixed donor and acceptor domains and discuss the potential consequences of the early charge dynamics on the performance of emerging nonfullerene photovoltaic devices. Our results demonstrate that careful choice of processing conditions enables enhanced exciton harvesting and suppression of GR by more than 3 orders of magnitude.