© 2019 authors. Published by the American Physical Society. Published by the American Physical Society under the terms of the "https://creativecommons.org/licenses/by/4.0/" Creative Commons Attribution 4.0 International license. Further distribution of this work must maintain attribution to the author(s) and the published article's title, journal citation, and DOI. The ability to retrieve image data through hair-thin optical fibers promises to open up new applications in a range of fields, from biomedical imaging to industrial inspection. Unfortunately, small changes in mechanical deformation and temperature can completely scramble optical information, distorting any resulting images. Correction of these dynamic changes requires measurement of the fiber transmission matrix (TM) in situ immediately before imaging, which typically requires access to both the proximal and distal facets of the fiber simultaneously. As a result, TM calibration is not feasible during most realistic usage scenarios without compromising the thin form factor with bulky distal optics. Here, we introduce a new approach to determine the TM of multimode or multicore optical fibers in a reflection-mode configuration, without requiring access to the distal facet. We propose introducing a thin stack of structured metasurface reflectors at the distal facet of the fiber, to introduce wavelength-dependent, spatially heterogeneous reflectance profiles. We derive a first-order fiber model that compensates these wavelength-dependent changes in the fiber TM and show that, consequently, the reflected data at three wavelengths can be used to unambiguously reconstruct the full TM by an iterative optimization algorithm. Unlike previous approaches, our method does not require the fiber matrix to be unitary, making it applicable to physically realistic fiber systems that have non-negligible power loss. We demonstrate TM reconstruction and imaging first using simulated nonunitary fibers and noisy reflection matrices, then using larger experimentally measured TMs of a densely packed multicore fiber (MCF), and finally using experimentally measured multiwavelength TMs recorded from a step-index multimode fiber (MMF). Parallelization of multiwavelength in situ measurements could enable experimental characterization times comparable with state-of-the-art transmission-mode fiber TM experiments. Our findings pave the way for online TM calibration in situ in hair-thin optical fibers.