> For the complete documentation index, see [llms.txt](https://docs.float16.cloud/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://docs.float16.cloud/getting-started/one-click-deploy/features/context-caching.md).

# Context caching

### What is context caching ?

Context caching is the ability to cache the KV (key-value) computations of the same context between requests. 

This feature helps speed up inference time by over 10 times when the request has the same context. 

The speedup is about 2 times when benchmarked with evaluation datasets like M3Exam.

### How does context caching is work ? 

Context caching is automatically triggered when requests have the same context within the same batch size. 

For example, if the endpoint has a batch size of 8 and each request has 1,024 tokens, and the requests share the same context for 900 tokens.

Instead of the system calculating 1,024 \* 8 = 8,192 tokens.

The system will calculate ((1,024 - 900) \* 8) + 1,024 = 2,016 tokens. 

This significantly reduces the compute required and improves the endpoint's latency.

### Use case

* RAG
* Few-shot prompting
* Code Co-pilot