The atomic geometries and transition levels of point defects and substitutional dopants in few-layer and bulk black phosphorus are calculated. The vacancy is found to reconstruct in monolayer P to leave a single dangling bond, giving a negative U defect with a +/- transition level at 0.24 eV above the valence band edge. The V(-) state forms an unusual 4-fold coordinated site. In few-layer and bulk black P, the defect becomes a positive U site. The divacancy is much more stable than the monovacancy, and it reconstructs to give no deep gap states. Substitutional dopants such as C, Si, O or S do not give rise to shallow donor or acceptor states but instead reconstruct to form non-doping sites analogous to DX or AX centers in GaAs. Impurities on black P adopt the 8-N rule of bonding, as in amorphous semiconductors, rather than simple substitutional geometries seen in tetrahedral semiconductors.