The spectroscopy of atomic gases confined in hollow-core photonic crystal fiber (HC-PCF) provides optimal atom-light coupling beyond the diffraction limit, which is desirable for various applications such as sensing, referencing, and nonlinear optics. Recently, coherent spectroscopy was carried out on highly excited Rydberg states at room temperature in a gas-filled HC-PCF. The large polarizability of the Rydberg states made it possible to detect weak electric fields inside the fiber. In this Letter, we show that by combining highly excited Rydberg states with higher-order optical modes, we can gain insight into the distribution and underlying effects of these electric fields. Comparisons between experimental findings and simulations indicate that the fields are caused by the dipole moments of atoms adsorbed on the hollow-core wall. Knowing the origin of the electric fields is an important step towards suppressing them in future HC-PCF experiments. Furthermore, a better understanding of the influence of adatoms will be advantageous for optimizing electric-field-sensitive experiments carried out in the vicinity of nearby surfaces.