Context

What is Context in Generative AI?

Context, simply put, is the short-term (working) memory of a language model (LLM). It is the set of all input data that the model is able to "see" and analyze at the moment it generates a response.

Context consists of three main elements:

1. System instruction (System Prompt) – e.g. "You are a helpful assistant".
2. Conversation history – the user's previous questions and the model's answers.
3. Current request – the new prompt and any attached data (documents, code).

Context Window

This is the maximum data limit, expressed in tokens, that the model can accept. If the conversation exceeds this limit, the oldest information is "forgotten" (cut off) or the model returns an error.

The size of the window is a key parameter when choosing a model for building apps:

• Claude 4.5 Opus: 200k tokens
• GPT-5.1: 400K tokens
• Gemini 3 Pro: 1,048,576 tokens

Challenges: "Lost in the Middle"

A large context window doesn't always mean perfect memory. Models tend to remember information at the beginning and end of the context better, and lose the information in the middle. This phenomenon is called "Lost in the Middle".

To test how effective the Context is, the "Needle in a Haystack" benchmark is used, which checks whether the model can pull out a precise fact hidden deep within a very long text.

Importance in automation (RAG)

When building AI apps (e.g. company chatbots), Context is the most expensive resource. To avoid overloading it, the RAG (Retrieval Augmented Generation) technique is used. Instead of putting the whole documentation into the Context, the system searches only for the fragments related to the question and "feeds" the model with them in real time.