Building the Data Layer for AI Agents
Enterprise companies are rushing to integrate AI agents into their operations, but the data layer they rely on is often ill-prepared for the task. Fabiane Nardon, representing TOTVS, explained during a recent presentation that while her company has access to vast amounts of information, simply having data isn’t enough. The information stored in traditional transactional systems was built for human applications, not for the token-hungry, latency-sensitive loops of modern AI reasoning. She noted that these systems were never designed to handle hundreds of unpredictable queries, nor were they optimized for the probabilistic reasoning required by agents rather than the deterministic precision of traditional software.
When deploying enterprise-grade agents, organizations must decide where to retrieve information. Nardon outlined a strategy for balancing freshness and processing power by splitting data sources. For scenarios requiring fresh data, such as write operations or business rules, the transactional system remains the primary source. However, for historical analysis or semantic search capabilities that legacy databases often lack, companies should route requests to a data platform. This approach mitigates the latency issues associated with directly querying transactional databases while ensuring data consistency.
Related: Uber Launches GitFarm for Large Monorepos
Fixing Ambiguity with Semantics
One of the biggest hurdles in enterprise data is semantic ambiguity. What constitutes an “active customer” varies wildly between marketing and finance departments. Nardon argued that LLMs lack the context to intuit these differences without explicit guidance. She proposed using Semantic Web standards, specifically RDF (Resource Description Framework) and OWL (Web Ontology Language), to solve this problem. By assigning unique identifiers to concepts and defining ontologies that represent relationships between them, companies can create a context graph that helps agents understand the nuances of business terms. Studies from 2024 suggest that adding this ontology-based layer can improve LLM response precision by 40%.
The implementation involves creating a low-latency data layer within the platform, often built on Postgres and DuckDB. This layer handles real-time processing and transformation, ensuring that data is consistent and ready for retrieval. When agents query this layer, they don’t just get raw data; they receive the relevant portion of the ontology tied to that specific data product. This ensures the agent understands exactly what it is looking at, reducing hallucinations and improving the accuracy of the final output.
Related: Apache Hudi pipelines face petabyte-scale delays
Security is handled by embedding permissions directly into the tool code rather than relying on the LLM to generate queries. This prevents prompt injection attacks. Additionally, identity propagation is used to ensure that an agent can only retrieve data a specific user is authorized to see, by authenticating via OAuth and passing the user context to the tools. Nardon also addressed the high cost of tokens, particularly in regions with lower economic purchasing power. She introduced a pattern called MCP Fabric, which allows a single service to act as a gateway for hundreds of virtual MCP servers, drastically reducing deployment overhead.
To further reduce costs, the company utilizes dynamic tool search. Instead of feeding every available tool into the agent’s context—which consumes significant tokens—the agent calls a search tool to find the specific tools it needs for a given task. For data formatting, they are testing TOON, a new standard designed to represent mixed data types more compactly than JSON, potentially saving 30% to 60% in token usage while maintaining sufficient precision for the models. This layered approach to tool management ensures that agents remain efficient and cost-effective in production environments.
