A search is performed for heavy neutrinos in the decay of a $W$ boson into two muons and a jet. The data set corresponds to an integrated luminosity of approximately $3.0 fb-1$ of proton-proton collision data at centre-of-mass energies of 7 and $8 TeV$ collected with the LHCb experiment. Both same-sign and opposite-sign muons in the final state are considered. Data are found to be consistent with the expected background. Upper limits on the coupling of a heavy neutrino with the Standard Model neutrino are set at $95\%$ confidence level in the heavy-neutrino mass range from 5 to $50 GeV/c2$. These are of the order of $10-3$ for lepton-number-conserving decays and of the order of $10-4$ for lepton-number-violating heavy-neutrino decays.