Direct searches for lepton flavor violation in decays of the $Z$ boson with the ATLAS detector at the LHC are presented. Decays of the $Z$ boson into an electron or a muon and a hadronically decaying $\tau$ lepton are considered. The searches are based on a data sample of proton-proton collisions collected by the ATLAS detector in 2015 and 2016, corresponding to an integrated luminosity of 36.1 fb$\mathrm{<mrow\; data-mjx-texclass="ORD">-1</mrow>}$ at a center-of-mass energy of $s=13$ TeV. No significant excess of events above the expected background is observed, and upper limits on the branching ratios of lepton-flavor-violating decays are set at the 95% confidence level: and . This is the first limit on $B(Z\to e\tau )$ with ATLAS data. The upper limit on $B(Z\to \mu \tau )$ is combined with a previous ATLAS result based on 20.3 fb$\mathrm{<mrow\; data-mjx-texclass="ORD">-1</mrow>}$ of proton-proton collision data at a center-of-mass energy of $s=8$ TeV and the combined upper limit at 95% confidence level is .