There is a growing demand to be able to “explain” machine learning (ML) systems’ decisions and actions to human users, par-ticularly when used in contexts where deci-sions have substantial implications for those affected and where there is a requirement for political accountability or legal compli-ance. Explainability is often discussed as a technical challenge in designing ML systems and decision procedures, to improve under-standing of what is typically a “black box” phenomenon. But some of the most difficult challenges are non-technical and raise ques-tions about the broader accountability of organizations using ML in their decision-making.