Obesity is reaching epidemic proportions and imposes major negative health crises and an economic burden in both high and low income countries. The multifaceted nature of obesity represents a major health challenge, with obesity affecting a variety of different organs and increases the risk of many other noncommunicable diseases, such as type 2 diabetes, fatty liver disease, dementia, cardiovascular diseases, and even cancer. The defining organ of obesity is the adipose tissue, highlighting the need to more comprehensively understand the development and biology of this tissue to understand the pathogenesis of obesity. Adipose tissue is a miscellaneous and highly plastic endocrine organ. It comes in many different sizes and shades and is distributed throughout many different locations in the body. Though its development begins prenatally, quite uniquely, it has the capacity for unlimited growth throughout adulthood. Adipose tissue is also a highly sexually dimorphic tissue, patterning men and women in different ways, which means the risks associated with obesity are also sexually dimorphic. Recent studies show that environmental factors during prenatal and early stages of postnatal development have the capacity to programme the structure and function of adipose tissue, with implications for the development of obesity. This review summarizes the evidence for a role for early environmental factors, such as maternal malnutrition, hypoxia, and exposure to excess hormones and endocrine disruptors during gestation in the programming of adipose tissue and obesity in the offspring. We will also discuss the complexity of studying adipose tissue biology and the importance of appreciating nuances in adipose tissue, such as sexual dimorphism and divergent responses to metabolic and endocrine stimuli. Given the rising levels of obesity worldwide, understanding how environmental conditions in early life affects adipose tissue phenotype and the subsequent development of obesity is of absolute importance.