What AI crawlers can tell us about the next phase of the internet
Ask a friend where to eat tonight and you get one opinion. Ask ChatGPT or Gemini and you get one answer, too, in seconds, with no need to open ten tabs and compare. Millions of people now search this way, and many have not noticed how much their habits have changed.
That shift has a quiet audience behind it: AI crawlers. These automated visitors read websites so AI tools can learn about a business. For thirty years, companies built their sites for two readers, people and search engines. A third reader has now arrived, and it is growing fast. Research by Vercel and MERJ found that AI crawlers pulled up pages nearly 1.3 billion times in a single month on Vercel’s network alone. That is more than a quarter of the volume from Googlebot, Google’s own crawler.
Manavdeep Singh, Founder and CEO of Publive, says these visitors matter because they shape how a business is described. “These crawlers are how AI learns what a business does and whether it can be trusted,” he says.
The problem starts with how websites are built. Many modern sites only come together inside a person’s browser, and most AI crawlers cannot complete that step. “A website can look perfect to a customer and almost empty to the AI that’s deciding how to describe that business,” Singh says.
AI does not leave a blank when it cannot read a page. It fills the gap with whatever it finds elsewhere, such as an aggregator, a competitor’s comparison page or an old news story. “Your customer still gets an answer about you,” Singh says. “It just isn’t yours.”
Most companies also have no way to spot the problem. Their analytics count visitors who use a browser, and crawlers do not use one. The dashboard looks healthy while the way people find the business changes underneath it.
The shift raises questions the industry has yet to answer. Search engines sent visitors back to the sites they read. AI reads far more and sends far fewer people back. Singh asks who gets the credit for the information, and who checks that the answer is accurate.
For a business, a simple first step is to ask ChatGPT or Gemini to describe it, then compare the answer with what the owner would say. Any mismatch shows where the story is going missing.
Singh’s advice is to treat a website less like a brochure and more like infrastructure, something machines can read clearly and consistently. “I think the next phase of the internet won’t reward whoever publishes the most,” he says. “It will reward whoever is understood. Being online was the first step. Being readable is the next one.”- Ends
