Context caching

What is context caching ?

Context caching is the ability to cache the KV (key-value) computations of the same context between requests.

This feature helps speed up inference time by over 10 times when the request has the same context.

The speedup is about 2 times when benchmarked with evaluation datasets like M3Exam.

How does context caching is work ?

Context caching is automatically triggered when requests have the same context within the same batch size.

For example, if the endpoint has a batch size of 8 and each request has 1,024 tokens, and the requests share the same context for 900 tokens.

Instead of the system calculating 1,024 * 8 = 8,192 tokens.

The system will calculate ((1,024 - 900) * 8) + 1,024 = 2,016 tokens.

This significantly reduces the compute required and improves the endpoint's latency.

Use case

• RAG

• Few-shot prompting

• Code Co-pilot