Bot with Plan

In the bot-with-plan project we developed an experimental approach to enhance the capabilities of ReAct-style LLM agents, particularly focusing on improving the performance of smaller open-source language models.

LLM agents need a broad range of capabilities to execute tasks autonomously, including decomposing complex instructions, planning actions, using tools, and adjusting plans based on observations. While commercial LLMs like GPT-4 can handle these complex prompts and include features like function calling, smaller open LLMs struggle. To bridge this gap for smaller open models, we experimented with an approach that separates planning from function calling in LLM agents. This separation allows the planner module to focus solely on describing tasks and selecting appropriate tools, without the complexity of function calling details. We implemented a system where tools are responsible for translating task descriptions into specific function calls, utilizing either generic function calling models or tool-specific methods.

To achieve this, we fine-tuned 7B LLMs (Mistral 7b) on synthetic agent trajectories that were created with GPT-4 in a simulated environment. This fine-tuning allowed the smaller model to achieve planning performance comparable to GPT-4. Our approach supports easy adaptation to different application-specific tools by generating corresponding simulated tools and trajectories for planner fine-tuning. Our agent implementation extensively utilizes schema-guided generation to ensure reliable communication between agent modules and can be used with predefined tools.

As part of this project, we also created fully-functional RAG-based search tools compatible with our agent implementation. These include an internet search tool that doesn’t require an API key, a Wikipedia search based on a local search index, and a tool for summarizing retrieved documents using Llama-3-8B-Instruct.

Watch the video demonstration to see our agent implementation in action, or explore the corresponding notebook for a detailed overview.

Related articles:

• Impact of prompt masking on LLM agent planning performance
• Planner fine-tuning on synthetic agent trajectories
• Separation of planning concerns in ReAct-style LLM agents
• Schema-guided generation with open LLMs
• Schema-guided generation in LangChain agents
• Enhancing Internet Search with RAG and Open Large Language Models