The implementation of static artificial magnetic fields in ultracold atomic systems has become a powerful tool, e.g. for simulating quantum-Hall physics with charge-neutral atoms. Taking an interacting bosonic flux ladder as a minimal model, we investigate protocols for adiabatic state preparation via magnetic flux ramps. Considering the fact that it is actually the artificial vector potential (in the form of Peierls phases) that can be experimentally engineered in optical lattices, rather than the magnetic field, we find that the time required for adiabatic state preparation dramatically depends on which pattern of Peierls phases is used. This can be understood intuitively by noting that different patterns of time-dependent Peierls phases that all give rise to the same magnetic field ramp, generally lead to different artificial electric fields during the ramp. Remarkably, we find that an optimal choice allows for preparing the ground state almost instantaneously. We relate this observation to shortcuts to adiabaticity via counterdiabatic driving. Our findings open new possibilities for robust state preparation in atomic quantum simulators.