Synergistic Utilization of LLMs for Program Synthesis

Abstract

Advances in Large Language Models (LLMs) have led them to be used as black boxes in several evolutionary algorithms for program synthesis. While these methods tend to be agnostic about which model is used, they only allow for using one. This paper suggests that using a combination of LLMs to seed population-based algorithms introduces more variation and leads to a wider variety of problems that can be solved, due to leveraging the strengths of component LLMs. We test this on the PSB2 suite, using the Search, Execute, Instruct, Debug and Rank (SEIDR) algorithm. In all cases examined, we find that using a combination of LLMs leads to more problems solved and better test pass rates compared to using the best individual model. We also find that the computational cost, as measured in terms of excess programs generated, is lowered.