The non-agricultural employment transfer of the rural labor force has fundamentally changed the labor input in China in recent decades. A good understanding of how the off-farm employment of rural laborers affects agricultural land use in China is needed. We use the Driscoll and Kraay standard errors fixed effects model to investigate the relationship between rural laborers’ off-farm employment and agricultural land use efficiency based on a panel data of 1,961 counties in China. We find that the distribution of county-level agricultural land use efficiency is heavily skewed to the right, with many counties below the national average efficiency level. We also identify a robust U-shaped relationship between off-farm employment and the change in agricultural land use efficiency, indicating that the substitution effect of capital and technology for rural labor has changed from weak to strong. The findings have important policy implications for the joint reform of the household registration (hukou) system and the rural land use system (i.e., the Three Rights Separation Reform) in China. Using the criteria derived from empirical analysis, we also perform content analysis to assess 20 rural land use policies implemented between 2014 and 2020. We find that these documents covered the three important areas to improve rural land use efficiency, that is, rural-urban mobility, rural land rights market development, and rural land rights protection. Although a synergy among the three areas has not yet been achieved, the central government has already put in place policies to enable and support coordinated actions in the three areas. The lessons learned from China also serve as a helpful reference for addressing the challenge of rural labor loss in other developing countries.