As a part of their current DEV Day presentation, OpenAI introduced that Immediate Caching was now obtainable for numerous fashions. On the time of writing, these fashions had been:-
GPT-4o, GPT-4o mini, o1-preview and o1-mini, in addition to fine-tuned variations of these fashions.
This information shouldn’t be underestimated, as it is going to permit builders to avoid wasting on prices and cut back software runtime latency.
API calls to supported fashions will mechanically profit from Immediate Caching on prompts longer than 1,024 tokens. The API caches the longest prefix of a immediate that has been beforehand computed, beginning at 1,024 tokens and rising in 128-token increments. For those who reuse prompts with widespread prefixes, OpenAI will mechanically apply the Immediate Caching low cost with out requiring you to vary your API integration.
As an OpenAI API developer, the one factor you will have to fret about is methods to monitor your Immediate Caching use, i.e. examine that it’s being utilized.
On this article, I’ll present you ways to try this utilizing Python, a Jupyter Pocket book and a chat completion instance.