The effects of high-temperature annealing on ferromagnetic Co-doped Indium Tin Oxide (ITO) thin films have been investigated using X-ray diffraction (XRD), magnetometry, and X-Ray Magnetic Circular Dichroism (XMCD). Following annealing, the magnetometry results indicate the formation of Co clusters with a significant increase in the saturation magnetization of the thin films arising from defects introduced during cluster formation. However, sum rule analysis of the element-specific XMCD results shows that the magnetic moment at the Co sites is reduced after annealing. The effects of annealing demonstrate that the ferromagnetism observed in the as-deposited Co-doped ITO films arises from intrinsic defects and cannot be related to the segregation of metallic Co clusters.