Repository logo

Critiquing Protein Family Classification Models Using Sufficient Input Subsets.

Accepted version



Change log


Carter, Brandon 
Bileschi, Maxwell 
Smith, Jamie 
Sanderson, Theo 
Bryant, Drew 


In many application domains, neural networks are highly accurate and have been deployed at large scale. However, users often do not have good tools for understanding how these models arrive at their predictions. This has hindered adoption in fields such as the life and medical sciences, where researchers require that models base their decisions on underlying biological phenomena rather than peculiarities of the dataset. We propose a set of methods for critiquing deep learning models and demonstrate their application for protein family classification, a task for which high-accuracy models have considerable potential impact. Our methods extend the Sufficient Input Subsets (SIS) technique, which we use to identify subsets of features in each protein sequence that are alone sufficient for classification. Our suite of tools analyzes these subsets to shed light on the decision-making criteria employed by models trained on this task. These tools show that while deep models may perform classification for biologically relevant reasons, their behavior varies considerably across the choice of network architecture and parameter initialization. While the techniques that we develop are specific to the protein sequence classification task, the approach taken generalizes to a broad set of scientific contexts in which model interpretability is essential.



interpretability, machine learning, model selection, neural networks, protein classification, protein domain, Computational Biology, Deep Learning, Humans, Machine Learning, Models, Biological, Multigene Family, Neural Networks, Computer, Proteins

Journal Title

J Comput Biol

Conference Name

Journal ISSN


Volume Title



Mary Ann Liebert Inc


All rights reserved
Simons Foundation (598399)