A measurement of the associated production of a top-quark pair ($tt¯$) with a vector boson ($W$, $Z$) in proton-proton collisions at a center-of-mass energy of 13 TeV is presented, using $36.1$ fb$\mathrm{<mrow\; data-mjx-texclass="ORD">-1</mrow>}$ of integrated luminosity collected by the ATLAS detector at the Large Hadron Collider. Events are selected in channels with two same- or opposite-sign leptons (electrons or muons), three leptons or four leptons, and each channel is further divided into multiple regions to maximize the sensitivity of the measurement. The $tt¯Z$ and $tt¯W$ production cross sections are simultaneously measured using a combined fit to all regions. The best-fit values of the production cross sections are $\sigma tt¯Z=0.95\pm 0.08\mathrm{stat}.\pm 0.10\mathrm{syst}.$ pb and $\sigma tt¯W=0.87\pm 0.13\mathrm{stat}.\pm 0.14\mathrm{syst}.$ pb in agreement with the Standard Model predictions. The measurement of the $tt¯Z$ cross section is used to set constraints on effective field theory operators which modify the $tt¯Z$ vertex.