Robotic hands have long strived to reach the performance of human hands. The physical complexity and extraordinary capabilities of the human hand, in terms of sensing, actuation, and cognitive abilities, make achieving this goal challenging. At the heart of the physical structure of the hand is its' passive behaviors. Seen most clearly in soft robotic hands, these behaviors influence and affect the mechanical, sensing, and control functionalities. With this perspective, we present a framework through which passivity in robot hands can be understood, by concretely identifying the role of passivity in the design, fabrication, and control of soft hands. In this framework we focus on the interactions between the physical hand and the: environment, internal actuation, sensor morphology, and wrist control. Taking these surrounding systems away, we are left with a passive soft hand whose behaviors emerge from external interactions. Inspired by the human hand, we define the role of these four key interacting pillars and review how state-of-the art robot hands utilize these four elements to aid functionality. We show how these pillars promote hybrid soft-rigid hands with rich behaviors, providing benefits in terms of the increased adaptability to uncertain environments, improved scalability and reduction in the cost of actuation, sensing, and control. This review provides a conceptual framework for approaching hand design and analysis through consideration of the passive behaviors. This highlights not only the advances that can be made by approaching the problem in this way but also the outstanding challenges that stem from this outlook.