The recently proposed phase-matching quantum key distribution offers means to overcome the linear key rate–transmittance bound. Since the key information is encoded onto the phases of coherent states, the misalignment between the two remote reference frames would yield errors and significantly degrade the key generation rate from the ideal case. In this work, we propose a reference-frame-independent design of phase-matching quantum key distribution by introducing a high-dimensional key encoding space. With encoded phases spanning the unit circle, the error statistics at arbitrary fixed-phase-reference difference can be recovered and treated separately, from which the misalignment angle can be identified. By naturally extending the binary encoding symmetry and complementarity to high dimensions, we present a security proof of this high-dimensional phase-matching quantum key distribution and demonstrate with simulation that a 17-dimensional protocol is completely immune to any degree of fixed misalignment and robust to slow phase fluctuations. We expect the high-dimensional protocol to be a practical reference-frame independent design for general phase-encoding schemes where high-dimensional encoding is relatively easy to implement