Clathrin-mediated endocytosis is an essential process that forms vesicles from the plasma membrane. Although most of the protein components of the endocytic protein machinery have been thoroughly characterized, their organization at the endocytic site is poorly understood. We developed a fluorescence microscopy method to track the average positions of yeast endocytic proteins in relation to each other with a time precision below 1 s and with a spatial precision of ~10 nm. With these data, integrated with shapes of endocytic membrane intermediates and with superresolution imaging, we could visualize the dynamic architecture of the endocytic machinery. We showed how different coat proteins are distributed within the coat structure and how the assembly dynamics of N-BAR proteins relate to membrane shape changes. Moreover, we found that the region of actin polymerization is located at the base of the endocytic invagination, with the growing ends of filaments pointing toward the plasma membrane.