Despite the impressive quality improvements yielded by neural machine translation (NMT) systems, controlling their translation output to adhere to user-provided terminology con- straints remains an open problem. We describe our approach to constrained neural decod- ing based on finite-state machines and multi- stack decoding which supports target-side con- straints as well as constraints with correspond- ing aligned input text spans. We demonstrate the performance of our framework on multiple translation tasks and motivate the need for constrained decoding with attentions as a means of reducing misplacement and duplication when translating user constraints.