Fiber structures and pathological features, e.g., inflammation and glycosaminoglycan (GAG) deposition, are the primary determinants of aortic mechanical properties which are associated with the development of an aneurysm. This study is designed to quantify the association of tissue ultimate strength and extensibility with the structural percentage of different components, in particular, GAG, and local fiber orientation. Thoracic aortic aneurysm (TAA) tissues from eight patients were collected. Ninety-six tissue strips of thickened intima, media, and adventitia were prepared for uni-extension tests and histopathological examination. Area ratios of collagen, elastin, macrophage and GAG, and collagen fiber dispersion were quantified. Collagen, elastin, and GAG were layer-dependent and the inflammatory burden in all layers was low. The local GAG ratio was negatively associated with the collagen ratio (r2 = 0.173, p < 0.05), but positively with elastin (r2 = 0.037, p < 0.05). Higher GAG deposition resulted in larger local collagen fiber dispersion in the media and adventitia, but not in the intima. The ultimate stretch in both axial and circumferential directions was exclusively associated with elastin ratio (axial: r2 = 0.186, p = 0.04; circumferential: r2 = 0.175, p = 0.04). Multivariate analysis showed that collagen and GAG contents were both associated with ultimate strength in the circumferential direction, but not with the axial direction (collagen: slope = 27.3, GAG: slope = -18.4, r2 = 0.438, p = 0.002). GAG may play important roles in TAA material strength. Their deposition was found to be associated positively with the local collagen fiber dispersion and negatively with ultimate strength in the circumferential direction.