In this paper, we consider the symmetries of the Dirac operator derived from a connection with skew-symmetric torsion. We find that the generalized conformal Killing-Yano tensors give rise to symmetry operators of the massless Dirac equation, provided an explicitly given anomaly vanishes. We show that this gives rise to symmetries of the Dirac operator in the case of strong Kahler with torsion (KT) and strong hyper-Kahler with torsion (HKT) manifolds.