AI is killing the open web. Who can stop it?
My website is www.rohit.online. You can visit it from your home computer, your office computer, from anywhere in your home country, or anywhere on the planet. You don’t need to login or pay anything.
Your website is www.[YOUR-WEBSITE].com. I can visit it from my home computer, my office computer, from anywhere in my home country, or from anywhere on the planet. I don’t need to login or pay anything.
This is called the open web. You just need a browser, an internet connection, and voila! You can surf the web without hitting any paywalls or logins.
I was born and raised in a small town in India. I learned more from the open web than from my textbooks. I am who I am because of the open web.
But LLMs are killing this open web.
Google bots have scraped the open web's websites for decades. Initially, the exchange between Google and the websites seemed somewhat fair: for every two pages the bots visited, Google sent one visitor to the website. Google profited from the scraped content, and in return, websites received traffic. But this deal has changed.
Google bots still scrape the web. But now, for every six pages scraped, they send one visitor to the original websites. What has changed? Now, Google doesn’t want the users to leave Google.com. So, it answers the users directly. And where does it get the answers from? From websites.
This sounds unfair. But wait till you hear about the LLMs.
For every 250 pages that OpenAI scrapes, it sends one user to the original website. For every 6000 pages that Anthropic scrapes, it sends one user to the original website. These numbers are reliable as they come from Cloudflare, the company that knows more about internet traffic than any other company.
The products of these AI companies are called helpful assistants. Are they? Or are they bullies?
To keep the AI scrapers away, websites might have to enforce paywalls and logins. Soon, the open web will be the close web. As far as I know, we can’t do anything about it. Individuals choosing not to use AI can not solve the problem. Just like you or I can not solve global warming by riding a bicycle. We need fundamental, systematic change.
I believe the solution will not come from regulators, but from the tech world itself. Governments and lawmakers are powerful, but their structure and incentives make them ineffective. We need a technical solution that, by design, can differentiate between humans and bots.
Companies like Cloudflare are building such solutions. But they are neither as simple nor as open as the open web. And the power of the web lies in its openness and simplicity.
PS: If you know of someone making an open solution to this problem, please forward me the link at hello@rohit.online