Intermetallic compounds containing f-electron elements have been prototypical materials for investigating strong electron correlations and quantum criticality (QC). Their heavy fermion ground state evoked by the magnetic f-electrons is susceptible to the onset of quantum phases, such as magnetism or superconductivity, due to the enhanced effective mass (m*) and a corresponding decrease of the Fermi temperature. However, the presence of f-electron valence fluctuations to a non-magnetic state is regarded an anathema to QC, as it usually generates a paramagnetic Fermi-liquid state with quasiparticles of moderate m*. Such systems are typically isotropic, with a characteristic energy scale T0 of the order of hundreds of kelvins that require large magnetic fields or pressures to promote a valence or magnetic instability. Here we show the discovery of a quantum critical behaviour and a Lifshitz transition under low magnetic field in an intermediate valence compound α-YbAlB4. The QC origin is attributed to the anisotropic hybridization between the conduction and localized f-electrons. These findings suggest a new route to bypass the large valence energy scale in developing the QC.