The endoplasmic reticulum (ER)-localized Hsp70 chaperone BiP contributes to protein folding homeostasis by engaging unfolded client proteins in a process that is tightly coupled to ATP binding and hydrolysis. The inverse correlation between BiP AMPylation and the burden of unfolded ER proteins suggests a post-translational mechanism for adjusting BiP's activity to changing levels of ER stress, but the underlying molecular details are unexplored. We present biochemical and crystallographic studies indicating that irrespective of the identity of the bound nucleotide AMPylation biases BiP towards a conformation normally attained by the ATP-bound chaperone. AMPylation does not affect the interaction between BiP and J-protein co-factors but appears to allosterically impair J protein-stimulated ATP-hydrolysis, resulting in the inability of modified BiP to attain high affinity for its substrates. These findings suggest a molecular mechanism by which AMPylation serves as a switch to inactivate BiP, limiting its interactions with substrates whilst conserving ATP.