Online Media Brands Seek Solution to Halt Unwanted AI Crawlers with New Protocol

Publishers Deploy New Tech To Protect Content From AI Scraping

Published: October 26, 2023 | Last Updated: October 26, 2023

A coalition of prominent online media organizations is implementing a novel system designed to safeguard their intellectual property from unauthorized use by Artificial Intelligence developers. The initiative aims to address growing concerns over AI companies leveraging copyrighted content for model training without proper licensing or compensation.

The Rise of Really Simple Licensing (RSL)

Several major digital brands, Including Yahoo, Quora, and medium, are adopting a new protocol called Really Simple Licensing, or RSL. Inspired by the widely-used really Simple Syndication (RSS) standard, RSL offers a standardized way for websites to communicate licensing terms to AI crawlers. Unlike RSS, which focuses on content distribution, RSL specifically addresses the permissions surrounding AI training data.

Currently, AI crawlers rely on “robots.txt” files to determine which parts of a website they are allowed to access. Though, recent reports indicate that many AI companies have been circumventing or ignoring these directives. A 2024 study by Imperva revealed that Bad Bots now account for more than half of all internet traffic, with AI crawlers representing a significant portion of that figure.

How RSL Works: A New layer of Protection

RSL functions as an added layer of technological defense against unauthorized scraping. It provides a clear and machine-readable signal to AI crawlers, explicitly stating the licensing terms for content use. This system intends to move beyond merely blocking access and instead offering a framework for controlled, compensated access.

Tim O’Reilly, CEO of O’Reilly Media, emphasized that RSL is about ensuring fair compensation for creators. “RSL builds directly on the legacy of RSS, providing the missing licensing layer for the AI-first Internet,” O’Reilly stated.”It ensures that the creators and publishers who fuel AI innovation are not just part of the conversation but fairly compensated for the value they create.”

Who’s On Board? A Growing List of Supporters

The RSL standard has already garnered support from a diverse group of online publishers and technology companies. Key participants include Reddit, People, Internet Brands, Fastly, wikiHow, O’reilly, Daily Beast, the MIT Press, Miso, Adweek, Ranker, and Raptive.

Tony Stubblebine,CEO of medium,issued a firm statement: “If AI is trained on our writers’ work,then it needs to pay for that work. right now, AI runs on stolen content. Adopting this RSL Standard is how we force those AI companies to either pay for what they use, stop using it, or shut down.”

The Impact of AI on Web Traffic

The adoption of RSL comes at a time when online publishers are grappling with a significant decline in web traffic, partly attributed to the growing prevalence of AI-powered tools. Google’s integration of AI-generated answers into search results has been criticized for diverting clicks away from publishers’ websites. Though, Google maintains that these AI Overviews actually drive “higher quality clicks” resulting in increased engagement.

Moreover, AI chatbots like ChatGPT offer users a synthesized information experience, reducing the need to visit multiple websites for research. A recent report from Infactory indicated that publishers are losing up to 25% of their traffic to AI platforms.

Metric	Before AI Integration (Q4 2022)	Current (Q3 2023)	Change
Average Publisher Website Traffic	100,000 visits/month	75,000 visits/month	-25%
Click-Through Rate from Google Search	5.2%	3.8%	-27%
Time on Site (Average)	2:30 minutes	2:00 minutes	-13%

Complementary Approaches to AI Content Protection

Publishers are exploring multiple strategies to protect their content. In addition to RSL, some are pursuing legal action against AI companies, while others are negotiating licensing deals. Services like Tollbit aim to charge AI crawlers for access to website content, and content delivery networks like cloudflare are actively blocking AI crawlers.

RSL co-founder eckart Walther emphasizes that RSL and tools like Cloudflare are complementary. “These compensation methods can also work together. For example, a publisher might want to charge for crawling their content and then also require a royalty payment every time the content is used by an AI model to reply to a question,” Walther explained.

The future of AI and Content Licensing

The debate surrounding AI and content licensing is likely to continue evolving. As AI technology becomes more sophisticated, publishers will need to adapt their strategies to protect their intellectual property and ensure fair compensation. The RSL standard represents a significant step towards establishing a more sustainable and equitable relationship between AI developers and content creators.

Did You Know? the concept of licensing content for machine learning isn’t entirely new, with precedent in areas like text-to-speech technology where voice actors receive royalties.

Pro Tip: Publishers should regularly review their robots.txt files and consider implementing additional security measures to deter unauthorized scraping.

Frequently Asked Questions About RSL

1. What is the RSL standard? The Really simple Licensing standard is a new protocol designed to allow publishers to specify licensing terms for AI crawlers, ensuring fair compensation for content used in AI training.
2. How does RSL differ from robots.txt? Robots.txt simply blocks crawlers; RSL establishes a framework for controlled access with associated licensing fees.
3. What are the benefits of adopting RSL? RSL empowers publishers to monetize their content used in AI training and protect their intellectual property from unauthorized use.
4. Which companies are supporting RSL? A growing list of companies, including Yahoo, Quora, Medium, Reddit, and O’Reilly Media, have announced their support for the RSL standard.
5. Is RSL a complete solution to AI content scraping? While RSL is a significant step, it’s ofen used in conjunction with othre measures, such as legal action and technical blocking tools.

What are your thoughts on the future of AI and its impact on content creation? Do you think publishers are adequately prepared to protect their content in the age of AI?

What are the potential legal ramifications for AI developers who disregard robots.txt and scrape copyrighted content?

The Rising Tide of AI Scraping & Its Impact on Publishers

Online media brands are facing a growing challenge: AI crawlers – frequently enough referred to as AI scraping bots – systematically consuming their content. While search engine crawlers are essential for indexing and driving organic traffic, these unwanted AI crawlers are different. They’re primarily used to train large language models (LLMs) and other artificial intelligence applications, often without permission or compensation to the original content creators. This practice, known as content scraping, is causing significant concerns around copyright infringement, revenue loss, and website performance.

Understanding the Problem: Why AI Crawlers Are Different

Traditional web crawlers, like those from Google or Bing, generally adhere to the robots.txt file, a standard that instructs them on which parts of a website not to crawl. However, many AI crawlers disregard these directives.

Here’s a breakdown of the key differences:

* Scale: AI crawlers often operate at a much larger scale than traditional search engine bots, placing a substantial load on server resources.

* Disregard for robots.txt: Many AI developers simply ignore the established robots.txt protocol.

* Commercial Gain: The primary purpose isn’t discovery, but rather the extraction of data for commercial AI model training.

* Impact on SEO: Excessive crawling can negatively impact a website’s search engine optimization (SEO), perhaps leading to lower rankings.

Introducing the New Protocol: A Collaborative Defense

A coalition of major news publishers and technology companies are developing a new protocol aimed at curbing unauthorized AI scraping. While details are still emerging (as of September 12, 2025), the core principle revolves around a system of digital watermarks and crawler identification.

Here’s how it’s expected to work:

1. Content Watermarking: Publishers embed subtle, undetectable watermarks within their content. These watermarks don’t affect the user experience but can be detected by the protocol’s system.
2. Crawler Identification: The protocol establishes a registry of legitimate crawlers (search engines, academic researchers, etc.).
3. Access Control: Crawlers attempting to access content without proper identification or those detected scraping content with watermarks will be blocked or rate-limited.
4. Reporting Mechanism: A centralized system allows publishers to report instances of unauthorized scraping.

Key Players & Industry Support

Several prominent organizations are involved in the development of this new protocol. These include:

* NewsGuard: Known for its credibility ratings of news sources,NewsGuard is contributing its expertise in identifying and classifying web crawlers.

* The Associated Press: A leading news agency,the AP is advocating for fair use of its content in AI training.

* Reuters: Another major news association, Reuters is actively involved in shaping the protocol’s standards.

* Technology Companies: Several tech firms are collaborating on the technical implementation of the watermarking and identification systems.

Benefits of Implementing the New Protocol

The potential benefits for online media brands are substantial:

* Protecting Intellectual Property: Safeguarding copyrighted content from unauthorized use.

* maintaining Website Performance: Reducing the strain on servers caused by excessive crawling.

* Preserving SEO Rankings: Preventing negative impacts on search engine visibility.

* Potential Revenue Streams: Exploring opportunities for licensing content to AI developers.

* Fair Compensation: Establishing a framework for publishers to be compensated for the use of their content in AI training.

Practical Tips for Publishers – What You Can Do Now

While the new protocol is being finalized, publishers can take several steps to mitigate the impact of AI scraping:

* Strengthen robots.txt: While not foolproof, ensure your robots.txt file is comprehensive and clearly defines which areas of your site should not be crawled.

* Implement Rate Limiting: Use web request firewalls (WAFs) or server-side configurations to limit the number of requests from a single IP address within a given timeframe.

*