Seeing Ghosts: The Importance of Indexing for Agents

All us database fanatics in the room know the importance of indexing for obtaining good query performance. The most classic example is likely the B+Tree, which allows for fast retrieval and updates to data that has a single sort order (like, say, names and numbers; but unlike, say, points in 3D space). There are lots of different kinds of indexes, some quite complicated. But they all have a similar basic structure: an index is a data structure that is computationally derived from actual user data, and it makes queries faster to execute than would otherwise be possible.

Do indexes still have a role to play in modern AI systems? Well, agentic systems are often incredibly slow, so it would be very convenient to use some kind of indexing. But fitting indexes to the modern agentic loop is not easy.

Agentic search systems — sometimes called deep research systems — have the core virtue that they keep working until the answer is good. A comically simplified agentic loop might go like this:

While (The-Answer-Is-Bad):
  Figure-Out-A-Tool-To-Run-That-Might-Improve-The-Answer
  Run-The-Tool
  Make-Answer-Better-Based-On-Tool-Results

return The-Answer-That-Is-Now-Good

At some level this is so simple and dumb that you can't believe it works. But it embodies why agents are great: they can respond flexibly to real-world data, rather than slavishly carrying out what the user asked for. It also shows why conventional indexing is hard to apply: agent work is so unstructured and so fly-by-the-seat-of-its-pants that it's hard to see how we could ever design an index that would help this loop go faster.

Well, one thing to note about indexes is that they often enable data skipping. Because the B+Tree index in a conventional database provides sorted access to, say, the account numbers of bank customers, a database query processor searching for (accountName == 'Mike') knows it can skip over all but a tiny fraction of the database. This reduction in data processing yields a massive runtime benefit. It's possible because the query processor can be 100% sure there is no relevant data in the database other than the listings the B+Tree has fetched when asked for accountName == 'Mike'.

Agents rarely enjoy this kind of total confidence about what can be ignored. Imagine you've provided your new movie script summarizer agent with a draft of 1999's Sixth Sense. (Spoiler alert for this 27-year-old movie: there's a big twist at the end!)

How will you tell the agent when it can just stop reading? Is the right time to terminate after reading 10%, or 50%? Or after reading the intro dialogue of every scene? Whatever you pick, the agent is either going to read a wasteful amount of info, or is going to miss that Bruce Willis is actually a ghost. In most cases, agents face an uncomfortable choice between delivering low resource consumption or delivering high quality.

An agentic index — a derived data structure that has global knowledge of the data being examined — is what allows the agent to break out of this choice and deliver on both.

It's one reason why Gekit's dedicated data agents, which manage datasets over the long haul and build indexes when needed, will be able to deliver much better results than a conventional task-driven agent.