Determining the average fraction of Lyman continuum (LyC) photons escaping high redshift galaxies is essential for understanding how reionization proceeded in the z>6 Universe. We want to measure the LyC signal from a sample of sources in the Chandra Deep Field South (CDFS) and COSMOS fields for which ultra-deep VIMOS spectroscopy as well as multi-wavelength Hubble Space Telescope (HST) imaging are available. We select a sample of 46 galaxies at $z\sim 4$ from the VIMOS Ultra Deep Survey (VUDS) database, such that the VUDS spectra contain the LyC part of the spectra, that is, the rest-frame range $880-910\backslash AA$. Taking advantage of the HST imaging, we apply a careful cleaning procedure and reject all the sources showing nearby clumps with different colours, that could potentially be lower-redshift interlopers. After this procedure, the sample is reduced to 33 galaxies. We measure the ratio between ionizing flux (LyC at $895\backslash AA$) and non-ionizing emission (at $\sim 1500\backslash AA$) for all individual sources. We also produce a normalized stacked spectrum of all sources. Assuming an intrinsic average $L\nu (1470)/L\nu (895)$ of 3, we estimate the individual and average relative escape fraction. We do not detect ionizing radiation from any individual source, although we identify a possible LyC emitter with very high Ly$\alpha$ equivalent width (EW). From the stacked spectrum and assuming a mean transmissivity for the sample, we measure a relative escape fraction $fescrel=0.09\pm 0.04$. We also look for correlations between the limits in the LyC flux and source properties and find a tentative correlation between LyC flux and the EW of the Ly$\alpha$ emission line. Our results imply that the LyC flux emitted by $V=25-26$ star-forming galaxies at z$\sim$4 is at most very modest, in agreement with previous upper limits from studies based on broad and narrow band imaging.