The interplay between magnetism and superconductivity can lead to unconventional proximity and Josephson effects. A related phenomenon that has recently attracted considerable attention is the superconducting diode effect, in which a nonreciprocal critical current emerges. Although superconducting diodes based on superconductor/ferromagnet (S/F) bilayers were demonstrated more than a decade ago, the precise underlying mechanism remains unclear. While not formally linked to this effect, the Fulde-Ferrell-Larkin-Ovchinikov (FFLO) state is a plausible mechanism due to the twofold rotational symmetry breaking caused by the finite center-of-mass-momentum of the Cooper pairs. Here, we directly observe asymmetric vortex dynamics that uncover the mechanism behind the superconducting vortex diode effect in Nb/EuS (S/F) bilayers. Based on our nanoscale SQUID-on-tip (SOT) microscope and supported by in-situ transport measurements, we propose a theoretical model that captures our key results. The key conclusion of our model is that screening currents induced by the stray fields from the F layer are responsible for the measured nonreciprocal critical current. Thus, we determine the origin of the vortex diode effect, which builds a foundation for new device concepts.