Debris disks are the natural by-products of the planet formation process. Scattered or polarized light observations are mostly sensitive to small dust grains that are released from the grinding down of bigger planetesimals. High angular resolution observations at optical wavelengths can provide key constraints on the radial and azimuthal distribution of the small dust grains. These constraints can help us better understand where most of the dust grains are released upon collisions. We present SPHERE/ZIMPOL observations of the debris disk around HR 4796 A, and model the radial profiles along several azimuthal angles of the disk with a code that accounts for the effect of stellar radiation pressure. This enables us to derive an appropriate description for the radial and azimuthal distribution of the small dust grains. Even though we only model the radial profiles along (or close to) the semi-major axis of the disk, our best-fit model is not only in good agreement with our observations but also with previously published datasets (from near-IR to sub-mm wavelengths). We find that the reference radius is located at $76.4\pm 0.4$ au, and the disk has an eccentricity of $0.076-0.010+0.016$, with the pericenter located on the front side of the disk (north of the star). We find that small dust grains must be preferentially released near the pericenter to explain the observed brightness asymmetry. Even though parent bodies spend more time near the apocenter, the brightness asymmetry implies that collisions happen more frequently near the pericenter of the disk. Our model can successfully reproduce the shape of the outer edge of the disk, without having to invoke an outer planet shepherding the debris disk. With a simple treatment of the effect of the radiation pressure, we conclude that the parent planetesimals are located in a narrow ring of about $3.6$ au in width.