AI tools such as ChatGPT, Perplexity, and Google AI Overviews are changing the way users retrieve information. Instead of browsing through websites, users receive responses within AI interfaces. This change poses a grave danger for Publishers whose standard revenue model based on advertisements is reliant on visits to the sites. The Advertising model based on visits collection is on the verge of collapse.
Steady Decreasing Web Traffic
AI tools eliminate the need to visit content-rich websites. Publishers relying on display advertisement revenue, would suffer traffic reduction as high as ~40%, according to a study. Advanced estimates from Gartner indicate that businesses reliant on search traffic will be impacted severely as well, ~50% decline in traffic volume predicted by 2028 as well as ~25% decrease in the utilization of search engines by 2026.
Ad Revenue Gravitation Toward AI and Walled Gardens
As AI channels become primary information sources, ad dollars are following. Users spend more time in app-like AI ecosystems, which can integrate advertising or branded content directly. We are now seeing a movement from an open web to a closed web platform that focuses on monetization.
Publisher Countermeasures: Blocking, Licensing & Pay-Per-Crawl
To counteract the loss of ad revenue, publishers are exploring new strategies:
Blocking AI crawlers by default:
Cloudflare, which powers 20% of the web, has begun blocking AI bots by default unless they are explicitly allowed. This helps maintain control over the AI-assisted content generation and prevents AI scraping without consent.
How it works:
- Cloudflare AI bot detection works using user agents and IP addresses to identify AI bots.
- When AI bots request a webpage, Cloudflare sends a 403 Forbidden response unless the publisher has allowed them.
- Cloudflare’s default “deny-first” stance safeguards AI content scraping and monetization.
Advantages:
- Content protection: Prevents AI platforms from utilizing publishers’ works for “training” without proper attribution or compensation.
- Negotiation leverage: Publishers are able to set terms for access to their content, effectively using licensing fees.
- Revenue control: High-value content can be monetized as it is protected from AI answer engines that skirt around advertisements.
Risks:
- Reduced discovery: long-term visibility on the internet can be reduced for sites that utilize AI- driven platforms that block crawlers.
- Smaller AI startups may find it hard to afford large-scale data licensing.
Pay-Per-Crawl licensing models:
Cloudflare allows publishers to monetize AI crawlers using existing web infrastructure. AI agents either pay upfront via request headers (granting a standard 200 OK) or receive a 402 Payment Required response on payment request.
How it works:
- When an AI bot accesses a webpage:
- If payment headers are made → Cloudflare allows access (returns a 200 OK).
- If payment headers are absent → Cloudflare using one of the most ignored HTTP response codes, “402 Payment Required” responds to the request, turning monetization of content a possibility.
- This allows licensing to be automated where AI firms embed billing APIs to manage each crawl’s microtransaction licensing.
Advantages:
- AI answer engines wanting your content have to pay, which allows access but ensures fair compensation.
- This is scalable covering millions of Cloudflare-hosted sites.
- Even without users clicking through, direct monetization can be created turning it into a new revenue stream.
Risks:
- In avoiding paywalled sites, AI companies diminishes their footprint on AI-generated content.
- Big data licensing may be a challenge for smaller AI startups with limited funds.
Revenue-sharing directly with AI platforms:
Some content creators are entering content-sharing agreements with services such as Perplexity, which pay for publishers’ content based on the revenue generated from usage.
How it works:
- AI engines retrieve and process publishers’ content, returning excerpts alongside citations and summaries.
- Revenue is earned based on views, clicks, or usage.
- Revenue is often split as part of larger licensing deals, and not as per-crawl agreements.
Advantages:
- Monetize existing demand: Instead of battling AI, they seek ways to profit from it.
- AI citations and backlinks may drive AI-perpetuated traffic, thus improving the publisher’s direct access to readers.
- Future-proofing: Embracing the new AI-enabled ecosystem of content discovery.
Risks:
- The increasing reliance of publishers on AI companies will enable the latter to garner greater control over traffic.
- The ability to negotiate premium contracts, as seen with larger publishers like the WSJ or NYT, does not extend to smaller sites, as these are paid nominal amounts.
- Fewer users are incentivized to click through during the content retrieval process.
In conclusion, as AI intermediaries increase, the emerging models appear to offer a way for publishers to monetize content. Regardless, there is little sympathy to be had for publishers that are caught in the crossfire. They seek to regain value in a system that is increasingly moving away from the open web.
