Early detection of esophageal neoplasia enables curative endoscopic therapy, but the current diagnostic standard of care has low sensitivity because early neoplasia is often inconspicuous with conventional white-light endoscopy. Here, we hypothesized that spectral endoscopy could enhance contrast for neoplasia in surveillance of patients with Barrett's esophagus. A custom spectral endoscope was deployed in a pilot clinical study of 20 patients to capture 715 in vivo tissue spectra matched with gold standard diagnosis from histopathology. Spectral endoscopy was sensitive to changes in neovascularization during the progression of disease; both non-dysplastic and neoplastic Barrett's esophagus showed higher blood volume relative to healthy squamous tissue (P = 0.001 and 0.02, respectively), and vessel radius appeared larger in neoplasia relative to non-dysplastic Barrett's esophagus (P = 0.06). We further developed a deep learning algorithm capable of classifying spectra of neoplasia versus non-dysplastic Barrett's esophagus with high accuracy (84.8% accuracy, 83.7% sensitivity, 85.5% specificity, 78.3% positive predictive value, and 89.4% negative predictive value). Exploiting the newly acquired library of labeled spectra to model custom color filter sets identified a potential 12-fold enhancement in contrast between neoplasia and non-dysplastic Barrett's esophagus using application-specific color filters compared with standard-of-care white-light imaging (perceptible color difference = 32.4 and 2.7, respectively). This work demonstrates the potential of endoscopic spectral imaging to extract vascular properties in Barrett's esophagus, to classify disease stages using deep learning, and to enable high-contrast endoscopy. SIGNIFICANCE: The results of this pilot first-in-human clinical trial demonstrate the potential of spectral endoscopy to reveal disease-associated vascular changes and to provide high-contrast delineation of neoplasia in the esophagus. GRAPHICAL ABSTRACT: http://cancerres.aacrjournals.org/content/canres/81/12/3415/F1.large.jpg.