AI Systems Move Past Simple Prompts

AI Systems Move Past Simple Prompts - ai context engineering — AI Systems Move Past Simple Prompts

Context engineering is the art and science of designing systems that understand, interpret, and respond to contextual information. It involves creating algorithms and models that enable machines, like AI assistants, to process and prioritize relevant data from user interactions and environment cues. This discipline is key for developing intelligent systems that can provide personalized and accurate responses by understanding the user’s intent and the situational context.

Building a Memory Layer for Alexa

Ricardo Ferreira, a developer at Redis, created an open-source project called Agent Memory Server (AMS) to build a memory layer on top of Redis. The goal was to allow developers to create short-term and long-term memory for applications, enabling more natural, human-like conversations. Ferreira applied this technology to build “My Jarvis,” an Alexa skill that uses an LLM to answer questions. The initial implementation worked, but as users asked more complex questions, the system began to struggle with consistency and accuracy.

One early issue was that the language model (LLM) had no concept of time. When asked to set a reminder, it often defaulted to 2021. Ferreira solved this by creating tools that explicitly provided the current date and time to the model. Another problem arose with user identity. The system would forget a user’s name after a few minutes. This was due to the stateless nature of LLMs and the limitations of short-term memory. Ferreira implemented a custom chat memory wrapper using Redis to store a buffer of the last 10 messages, allowing the model to recall previous interactions within a session.

The system eventually needed long-term memory to retain information across sessions. Ferreira implemented a content retriever using vector search, which allowed the model to find relevant past memories. This worked well, but it introduced new problems. For example, if a user asked about their favorite color, the model might answer “black” because both “favorite programming language” and “favorite color” appeared in the retrieved memories. This “context confusion” required more intentional design.

Practical Solutions for Context Confusion

Ferreira categorized the problems into context poisoning, distraction, and confusion. He realized that simply upgrading to a larger model or a more elaborate system prompt did not solve these issues. Instead, he had to pragmatically engineer the context at every stage of the pipeline. One technique was “few-shots design,” where the system prompt included examples of correct and incorrect responses to guide the model. For instance, to distinguish between a “pull request” and a “dentist appointment,” the system provided examples of how the model should interpret the word “priority.”

Another challenge was the sheer volume of data retrieved during a query. The vector search would return the top-k results, and the model would use all of them to formulate an answer. This often led to irrelevant information being included. Ferreira solved this by implementing “reranking.” Before the LLM saw the retrieved data, a separate scoring model (using Cohere) filtered out low-scoring results. This ensured that only the most relevant information was passed to the model.

Multi-tenancy was also a concern. When family members used the device, the model needed to separate their memories. Ferreira used vector search with post-filtering, tagging memories with a user ID and filtering the results based on who was currently speaking. He also had to balance the number of retrieved memories (top-k). Too few resulted in missing information; too many degraded the answer quality. This balance is a critical part of context engineering.

The Hidden Costs of Context

Ferreira’s final lesson was about the financial cost of these engineering practices. Initially, he kept a growing context window that included all past interactions. This provided a rich conversational experience but caused the token count to grow exponentially. This linear growth of context directly translated to higher API costs. He had to cap the context size using a token window to keep costs flat. However, simply deleting old messages could lose important information. He solved this by using a summarization technique, where a second LLM call condensed the history into a summary before the main answer was generated.

He also discovered the value of semantic caching. Instead of caching exact string matches, the system cached answers based on the semantic meaning of the query. If a user asked the same question in a different way, the system could retrieve the cached answer rather than making a new API call. This reduced redundant calls by more than half. However, semantic caching is not perfect; it requires careful calibration to avoid caching irrelevant results, such as when a user speaks with a heavy accent.

These engineering decisions—tools for time, retrieval for memory, filtering for relevance, and caching for cost—transformed a basic Alexa skill into a robust application. The work demonstrated that context engineering is not a one-time setup but a continuous process of refining how an AI interprets and uses information.