Ideally, you may consider agentic functions whilst you might be creating them, as a substitute of analysis being an afterthought. For this to work, although, you want to have the ability to mock each inner and exterior dependencies of the agent you might be creating. I’m extraordinarily excited by PydanticAI as a result of it helps dependency injection from the bottom up. It’s the first framework that has allowed me to construct agentic functions in an evaluation-driven method.
On this article, I’ll speak in regards to the core challenges and exhibit creating a easy agent in an evaluation-driven manner utilizing PydanticAI.
Challenges when creating GenAI functions
Like many GenAI builders, I’ve been ready for an agentic framework that helps the complete improvement lifecycle. Every time a brand new framework comes alongside, I strive it out hoping that this would be the One — see, for instance, my articles about DSPy, Langchain, LangGraph, and Autogen.
I discover that there are core challenges {that a} software program developer faces when creating an LLM-based utility. These challenges are usually not blockers if you’re constructing a easy PoC with GenAI, however they may come to chunk you if you’re constructing LLM-powered functions in manufacturing.
What challenges?
(1) Non-determinism: In contrast to most software program APIs, calls to an LLM with the very same enter may return completely different outputs every time. How do you even start to check such an utility?
(2) LLM limitations: Foundational fashions like GPT-4, Claude, and Gemini are restricted by their coaching knowledge (e.g., no entry to enterprise confidential data), functionality (e.g., you cannot invoke enterprise APIs and databases), and cannot plan/cause.
(3) LLM flexibility: Even should you resolve to stay to LLMs from a single supplier akin to Anthropic, chances are you’ll discover that you simply want a unique LLM for every step — maybe one step of your workflow wants a low-latency small language mannequin (Haiku), one other requires nice code-generation functionality (Sonnet), and a 3rd step requires glorious contextual consciousness (Opus).
(4) Price of Change: GenAI applied sciences are shifting quick. Lately, most of the enhancements have come about in foundational mannequin capabilities. Not are the foundational fashions simply producing textual content primarily based on consumer prompts. They’re now multimodal, can generate structured outputs, and may have reminiscence. But, should you attempt to construct in an LLM-agnostic manner, you typically lose the low-level API entry that may activate these options.
To assist handle the primary drawback, of non-determinism, your software program testing wants to include an analysis framework. You’ll by no means have software program that works 100%; as a substitute, you’ll need to have the ability to design round software program that’s x% appropriate, construct guardrails and human oversight to catch the exceptions, and monitor the system in real-time to catch regressions. Key to this functionality is evaluation-driven improvement (my time period), an extension of test-driven improvement in software program.
The present workaround for all of the LLM limitations in Problem #2 is to make use of agentic architectures like RAG, present the LLM entry to instruments, and make use of patterns like Reflection, ReACT and Chain of Thought. So, your framework might want to have the power to orchestrate brokers. Nonetheless, evaluating brokers that may name exterior instruments is difficult. You want to have the ability to inject proxies for these exterior dependencies in an effort to check them individually, and consider as you construct.
To deal with problem #3, an agent wants to have the ability to invoke the capabilities of several types of foundational fashions. Your agent framework must be LLM-agnostic on the granularity of a single step of an agentic workflow. To deal with the speed of change consideration (problem #4), you need to retain the power to make low-level entry to the foundational mannequin APIs and to strip out sections of your codebase which can be not essential.
Is there a framework that meets all these standards? For the longest time, the reply was no. The closest I may get was to make use of Langchain, pytest’s dependency injection, and deepeval with one thing like this (full instance is here):
from unittest.mock import patch, Mock
from deepeval.metrics import GEvalllm_as_judge = GEval(
title="Correctness",
standards="Decide whether or not the precise output is factually appropriate primarily based on the anticipated output.",
evaluation_params=[LLMTestCaseParams.INPUT, LLMTestCaseParams.ACTUAL_OUTPUT],
mannequin='gpt-3.5-turbo'
)
@patch('lg_weather_agent.retrieve_weather_data', Mock(return_value=chicago_weather))
def eval_query_rain_today():
input_query = "Is it raining in Chicago?"
expected_output = "No, it isn't raining in Chicago proper now."
end result = lg_weather_agent.run_query(app, input_query)
actual_output = end result[-1]
print(f"Precise: {actual_output} Anticipated: {expected_output}")
test_case = LLMTestCase(
enter=input_query,
actual_output=actual_output,
expected_output=expected_output
)
llm_as_judge.measure(test_case)
print(llm_as_judge.rating)
Basically, I’d assemble a Mock object (chicago_weather within the above instance) for each LLM name and patch the decision to the LLM (retrieve_weather_data within the above instance) with the hardcoded object at any time when I wanted to mock that a part of the agentic workflow. The dependency injection is everywhere, you want a bunch of hardcoded objects, and the calling workflow turns into extraordinarily exhausting to observe. Notice that should you don’t have dependency injection, there is no such thing as a strategy to check a operate like this: clearly, the exterior service will return the present climate and there’s no strategy to decide what the proper reply is for a query akin to whether or not or not it’s raining proper now.
So … is there an agent framework that helps dependency injection, is Pythonic, gives low-level entry to LLMs, is model-agnostic, helps constructing it one eval-at-a-time, and is simple to make use of and observe?
Virtually. PydanticAI meets the primary 3 necessities; the fourth (low-level LLM entry) just isn’t doable, however the design doesn’t preclude it. In the remainder of this text, I’ll present you methods to use it to develop an agentic utility in an evaluation-driven manner.
1. Your first PydanticAI Utility
Let’s begin out by constructing a easy PydanticAI utility. This can use an LLM to reply questions on mountains:
agent = llm_utils.agent()
query = "What's the tallest mountain in British Columbia?"
print(">> ", query)
reply = agent.run_sync(query)
print(reply.knowledge)
Within the code above, I’m creating an agent (I’ll present you ways, shortly) after which calling run_sync passing within the consumer immediate, and getting again the LLM’s response. run_sync is a strategy to have the agent invoke the LLM and watch for the response. Different methods are to run the question asynchronously, or to stream its response. (Full code is right here if you wish to observe alongside).
Run the code above, and you’re going to get one thing like:
>> What's the tallest mountain in British Columbia?
The tallest mountain in British Columbia is **Mount Robson**, at 3,954 metres (12,972 ft).
To create the agent, create a mannequin after which inform the agent to make use of that Mannequin for all its steps.
import pydantic_ai
from pydantic_ai.fashions.gemini import GeminiModeldef default_model() -> pydantic_ai.fashions.Mannequin:
mannequin = GeminiModel('gemini-1.5-flash', api_key=os.getenv('GOOGLE_API_KEY'))
return mannequin
def agent() -> pydantic_ai.Agent:
return pydantic_ai.Agent(default_model())
The thought behind default_model() is to make use of a comparatively cheap however quick mannequin like Gemini Flash because the default. You possibly can then change the mannequin utilized in particular steps as essential by passing in a unique mannequin to run_sync()
PydanticAI mannequin help looks sparse, however essentially the most generally used fashions — the present frontier ones from OpenAI, Groq, Gemini, Mistral, Ollama, and Anthropic — are all supported. By means of Ollama, you will get entry to Llama3, Starcoder2, Gemma2, and Phi3. Nothing important appears to be lacking.
2. Pydantic with structured outputs
The instance within the earlier part returned free-form textual content. In most agentic workflows, you’ll need the LLM to return structured knowledge in an effort to use it instantly in packages.
Contemplating that this API is from Pydantic, returning structured output is sort of simple. Simply outline the specified output as a dataclass (full code is here):
from dataclasses import dataclass@dataclass
class Mountain:
title: str
location: str
peak: float
While you create the Agent, inform it the specified output sort:
agent = Agent(llm_utils.default_model(),
result_type=Mountain,
system_prompt=(
"You're a mountaineering information, who gives correct data to most people.",
"Present all distances and heights in meters",
"Present location as distance and path from nearest huge metropolis",
))
Notice additionally the usage of the system immediate to specify items and many others.
Operating this on three questions, we get:
>> Inform me in regards to the tallest mountain in British Columbia?
Mountain(title='Mount Robson', location='130km North of Vancouver', peak=3999.0)
>> Is Mt. Hood straightforward to climb?
Mountain(title='Mt. Hood', location='60 km east of Portland', peak=3429.0)
>> What is the tallest peak within the Enchantments?
Mountain(title='Mount Stuart', location='100 km east of Seattle', peak=3000.0)
However how good is that this agent? Is the peak of Mt. Robson appropriate? Is Mt. Stuart actually the tallest peak within the Enchantments? All of this data may have been hallucinated!
There isn’t a manner so that you can understand how good an agentic utility is except you consider the agent towards reference solutions. You cannot simply “eyeball it”. Sadly, that is the place a number of LLM frameworks fall quick — they make it actually exhausting to guage as you develop the LLM utility.
3. Consider towards reference solutions
It’s once you begin to consider towards reference solutions that PydanticAI begins to indicate its strengths. Every part is sort of Pythonic, so you may construct customized analysis metrics fairly merely.
For instance, that is how we are going to consider a returned Mountain object on three standards and create a composite rating (full code is right here):
def consider(reply: Mountain, reference_answer: Mountain) -> Tuple[float, str]:
rating = 0
cause = []
if reference_answer.title in reply.title:
rating += 0.5
cause.append("Right mountain recognized")
if reference_answer.location in reply.location:
rating += 0.25
cause.append("Right metropolis recognized")
height_error = abs(reference_answer.peak - reply.peak)
if height_error < 10:
rating += 0.25 * (10 - height_error)/10.0
cause.append(f"Top was {height_error}m off. Right reply is {reference_answer.peak}")
else:
cause.append(f"Unsuitable mountain recognized. Right reply is {reference_answer.title}")return rating, ';'.be part of(cause)
Now, we will run this on a dataset of questions and reference solutions:
questions = [
"Tell me about the tallest mountain in British Columbia?",
"Is Mt. Hood easy to climb?",
"What's the tallest peak in the Enchantments?"
]reference_answers = [
Mountain("Robson", "Vancouver", 3954),
Mountain("Hood", "Portland", 3429),
Mountain("Dragontail", "Seattle", 2690)
]
total_score = 0
for l_question, l_reference_answer in zip(questions, reference_answers):
print(">> ", l_question)
l_answer = agent.run_sync(l_question)
print(l_answer.knowledge)
l_score, l_reason = consider(l_answer.knowledge, l_reference_answer)
print(l_score, ":", l_reason)
total_score += l_score
avg_score = total_score / len(questions)
Operating this, we get:
>> Inform me in regards to the tallest mountain in British Columbia?
Mountain(title='Mount Robson', location='130 km North-East of Vancouver', peak=3999.0)
0.75 : Right mountain recognized;Right metropolis recognized;Top was 45.0m off. Right reply is 3954
>> Is Mt. Hood straightforward to climb?
Mountain(title='Mt. Hood', location='60 km east of Portland, OR', peak=3429.0)
1.0 : Right mountain recognized;Right metropolis recognized;Top was 0.0m off. Right reply is 3429
>> What is the tallest peak within the Enchantments?
Mountain(title='Dragontail Peak', location='14 km east of Leavenworth, WA', peak=3008.0)
0.5 : Right mountain recognized;Top was 318.0m off. Right reply is 2690
Common rating: 0.75
Mt. Robson’s peak is 45m off; Dragontail peak’s peak was 318m off. How would you repair this?
That’s proper. You’d use a RAG structure or arm the agent with a instrument that gives the proper peak data. Let’s use the latter method and see methods to do it with Pydantic.
Notice how evaluation-driven improvement reveals us the trail ahead to enhance our agentic utility.
4a. Utilizing a instrument
PydanticAI helps a number of methods to supply instruments to an agent. Right here, I annotate a operate to be known as at any time when it wants the peak of a mountain (full code here):
agent = Agent(llm_utils.default_model(),
result_type=Mountain,
system_prompt=(
"You're a mountaineering information, who gives correct data to most people.",
"Use the offered instrument to search for the elevation of many mountains."
"Present all distances and heights in meters",
"Present location as distance and path from nearest huge metropolis",
))
@agent.instrument
def get_height_of_mountain(ctx: RunContext[Tools], mountain_name: str) -> str:
return ctx.deps.elev_wiki.snippet(mountain_name)
The operate, although, does one thing unusual. It pulls an object known as elev_wiki out of the run-time context of the agent. This object is handed in once we name run_sync:
class Instruments:
elev_wiki: wikipedia_tool.WikipediaContent
def __init__(self):
self.elev_wiki = OnlineWikipediaContent("Checklist of mountains by elevation")instruments = Instruments() # Instruments or FakeTools
l_answer = agent.run_sync(l_question, deps=instruments) # observe how we're capable of inject
As a result of the Runtime context could be handed into each agent invocation or instrument name , we will use it to do dependency injection in PydanticAI. You’ll see this within the subsequent part.
The wiki itself simply queries Wikipedia on-line (code here) and extracts the contents of the web page and passes the suitable mountain data to the agent:
import wikipediaclass OnlineWikipediaContent(WikipediaContent):
def __init__(self, subject: str):
print(f"Will question on-line Wikipedia for data on {subject}")
self.web page = wikipedia.web page(subject)
def url(self) -> str:
return self.web page.url
def html(self) -> str:
return self.web page.html()
Certainly, once we run it, we get appropriate heights now:
Will question on-line Wikipedia for data on Checklist of mountains by elevation
>> Inform me in regards to the tallest mountain in British Columbia?
Mountain(title='Mount Robson', location='100 km west of Jasper', peak=3954.0)
0.75 : Right mountain recognized;Top was 0.0m off. Right reply is 3954
>> Is Mt. Hood straightforward to climb?
Mountain(title='Mt. Hood', location='50 km ESE of Portland, OR', peak=3429.0)
1.0 : Right mountain recognized;Right metropolis recognized;Top was 0.0m off. Right reply is 3429
>> What is the tallest peak within the Enchantments?
Mountain(title='Mount Stuart', location='Cascades, Washington, US', peak=2869.0)
0 : Unsuitable mountain recognized. Right reply is Dragontail
Common rating: 0.58
4b. Dependency injecting a mock service
Ready for the API name to Wikipedia every time throughout improvement or testing is a foul thought. As an alternative, we are going to need to mock the Wikipedia response in order that we will develop shortly and be assured of the end result we’re going to get.
Doing that may be very easy. We create a Pretend counterpart to the Wikipedia service:
class FakeWikipediaContent(WikipediaContent):
def __init__(self, subject: str):
if subject == "Checklist of mountains by elevation":
print(f"Will used cached Wikipedia data on {subject}")
self.url_ = "https://en.wikipedia.org/wiki/List_of_mountains_by_elevation"
with open("mountains.html", "rb") as ifp:
self.html_ = ifp.learn().decode("utf-8")def url(self) -> str:
return self.url_
def html(self) -> str:
return self.html_
Then, inject this faux object into the runtime context of the agent throughout improvement:
class FakeTools:
elev_wiki: wikipedia_tool.WikipediaContent
def __init__(self):
self.elev_wiki = FakeWikipediaContent("Checklist of mountains by elevation")instruments = FakeTools() # Instruments or FakeTools
l_answer = agent.run_sync(l_question, deps=instruments) # observe how we're capable of inject
This time once we run, the analysis makes use of the cached wikipedia content material:
Will used cached Wikipedia data on Checklist of mountains by elevation
>> Inform me in regards to the tallest mountain in British Columbia?
Mountain(title='Mount Robson', location='100 km west of Jasper', peak=3954.0)
0.75 : Right mountain recognized;Top was 0.0m off. Right reply is 3954
>> Is Mt. Hood straightforward to climb?
Mountain(title='Mt. Hood', location='50 km ESE of Portland, OR', peak=3429.0)
1.0 : Right mountain recognized;Right metropolis recognized;Top was 0.0m off. Right reply is 3429
>> What is the tallest peak within the Enchantments?
Mountain(title='Mount Stuart', location='Cascades, Washington, US', peak=2869.0)
0 : Unsuitable mountain recognized. Right reply is Dragontail
Common rating: 0.58
Look fastidiously on the above output — there are completely different errors from the zero-shot instance. In Part #2, the LLM picked Vancouver because the closest metropolis to Mt. Robson and Dragontail because the tallest peak within the Enchantments. These solutions occurred to be appropriate. Now, it picks Jasper and Mt. Stuart. We have to do extra work to repair these errors — however evaluation-driven improvement at the very least offers us a path of journey.
Present Limitations
PydanticAI may be very new. There are a few locations the place it may very well be improved:
- There isn’t a low-level entry to the mannequin itself. For instance, completely different foundational fashions help context caching, immediate caching, and many others. The mannequin abstraction in PydanticAI doesn’t present a strategy to set these on the mannequin. Ideally, we will work out a kwargs manner of doing such settings.
- The necessity to create two variations of agent dependencies, one actual and one faux, is sort of widespread. It might be good if we had been capable of annoate a instrument or present a easy strategy to swap between the 2 kinds of providers throughout the board.
- Throughout improvement, you don’t want logging as a lot. However once you go to run the agent, you’ll normally need to log the prompts and responses. Generally, you’ll want to log the intermediate responses. The way in which to do that appears to be a industrial product known as Logfire. An OSS, cloud-agnostic logging framework that integrates with the PydanticAI library can be splendid.
It’s doable that these exist already and I missed them, or maybe they may have been carried out by the point you might be studying this text. In both case, go away a remark for future readers.
Total, I like PydanticAI — it provides a really clear and Pythonic strategy to construct agentic functions in an evaluation-driven method.
Recommended subsequent steps:
- That is a type of weblog posts the place you’ll profit from truly working the examples as a result of it describes a technique of improvement in addition to a brand new library. This GitHub repo accommodates the PydanticAI instance I walked by means of on this put up: https://github.com/lakshmanok/lakblogs/tree/main/pydantic_ai_mountains Comply with the directions within the README to strive it out.
- Pydantic AI documentation: https://ai.pydantic.dev/
- Patching a Langchain workflow with Mock objects. My “earlier than” answer: https://github.com/lakshmanok/lakblogs/blob/main/genai_agents/eval_weather_agent.py