In the first half of the thesis I investigate David Hilbert's early ontology of mathematics around the period 1899-1916. Hilbert's early views are of significant philosophical interest and have been largely ignored due to his later, more influential work. I suggest that, in this period Hilbert, can be understood as an early structuralist. In the second half of the thesis, I connect two important debates in the foundations of mathematics: Hale and Wright's neo-Fregean logicism and the Frege-Hilbert controversy. Using this connection, I adapt Frege's objections to Hilbert and apply them to Hale and Wright's account. By doing this, I show that the neo-Fregean logicists have long abandoned the Fregean element of their program in favor of a structuralist ontology. I conclude that our ontological conception of what exists in mathematics and what it is like constrains the foundations we use to characterise mathematical reality.