Overview

Large language models (LLMs) is an active area of research that has had a significant impact on both academia and industry. Both proprietary and open models, such as Code Llama, have demonstrated significant capability for code development tasks such as code completion, test generation, and code summarization.

However, the next leap will involve reasoning and planning with LLM trained on code. Reasoning is of core importance to code development and future LLM coding capabilities. The inputs to the reasoning process are multifaceted. Common ones include the source code and error logs for code translation and debugging. Additional information could be gained through static analysis of the code, such as abstract syntax tree (AST), a tree representation of the structure of the source code. Yet another source of information is the runtime profiler, where information regarding where the runtime is spent is collected.

There are many advanced reasoning techniques in the code development domain. Notable examples include self-debug and flow-engineering. Planning is also a critical component for applying LLM for code development. There are many tools that have been developed in the past to assist code development. Examples include tools for test case generation, tools for security risk identification from source code, and tools for runtime performance bottleneck analysis. How to combine the tools together to accomplish a complex task is a planning problem and a very important research direction.