• Publish Your article
  • Editorial Policy
  • Contact
  • Advertise
Tuesday, July 1, 2025
No Result
View All Result
UK Herald
  • Home
  • Politics
  • Business
  • Entertainment
    • All
    • Sports
    England rugby stadium Twickenham given new name after more than 100 years in shock new deal

    England rugby stadium Twickenham given new name after more than 100 years in shock new deal

    Peter Morgan dead at 65: Former Wales and Lions rugby star who became a politician passes away as club pays tribute

    Peter Morgan dead at 65: Former Wales and Lions rugby star who became a politician passes away as club pays tribute

    Horse racing tips: Unexposed Group 1 contender can stun the big guns at 14-1

    Horse racing tips: Unexposed Group 1 contender can stun the big guns at 14-1

    Woman ‘raped seven times by two French rugby stars who left her riddled with bite marks & with horror injuries’

    Woman ‘raped seven times by two French rugby stars who left her riddled with bite marks & with horror injuries’

    Horse racing tips: Gary Moore’s charge can gain revenge after falling last time out

    Horse racing tips: Gary Moore’s charge can gain revenge after falling last time out

    Ian Buckett dead at 56: Former Wales rugby star who was ‘admired and feared equally’ dies as tributes pour in

    Ian Buckett dead at 56: Former Wales rugby star who was ‘admired and feared equally’ dies as tributes pour in

    Horse racing tips: Bash the bookies with these longshots including 9-1 fancy

    Horse racing tips: Bash the bookies with these longshots including 9-1 fancy

    Shayne Philpott dead at 58 – New Zealand All Blacks rugby legend dies after suffering ‘medical event’

    Shayne Philpott dead at 58 – New Zealand All Blacks rugby legend dies after suffering ‘medical event’

    Horse racing tips: This 7-1 chance appears to have been laid out for race he won last year

    Horse racing tips: This 7-1 chance appears to have been laid out for race he won last year

  • Lifestyle
    • All
    • Fashion
    • food
    • Health
    • Travel
    Is it safe to travel to Turkey? Latest Foreign Office advice as wildfires rage in Izmir

    Is it safe to travel to Turkey? Latest Foreign Office advice as wildfires rage in Izmir

    I grew up in Yorkshire — here’s my insider’s guide to a weekend in the Dales

    I grew up in Yorkshire — here’s my insider’s guide to a weekend in the Dales

    All the food and drink banned on EasyJet, Ryanair, Jet2 and TUI flights

    All the food and drink banned on EasyJet, Ryanair, Jet2 and TUI flights

    ‘Just gorgeous’: Your favourite European islands that aren’t in Greece

    ‘Just gorgeous’: Your favourite European islands that aren’t in Greece

    ‘Charming’ city in Spain is a lesser-known gem with £40 flights and ‘hardly any tourists’

    ‘Charming’ city in Spain is a lesser-known gem with £40 flights and ‘hardly any tourists’

    Anti-tourism protesters have an important message — and it’s not for tourists

    Anti-tourism protesters have an important message — and it’s not for tourists

    ‘Ambitious’ new sleeper train will connect 100 European cities — with private rooms from £67

    ‘Ambitious’ new sleeper train will connect 100 European cities — with private rooms from £67

    Unassuming UK village is an absolute gem for foodies with ‘exceptional’ restaurants

    Unassuming UK village is an absolute gem for foodies with ‘exceptional’ restaurants

    TUI relaunches UK flights to forgotten year-round sunshine destination after 3 years

    TUI relaunches UK flights to forgotten year-round sunshine destination after 3 years

    New Alternative Treatments for Erectile Dysfunction

    New Alternative Treatments for Erectile Dysfunction

    Trending Tags

    • Golden Globes
    • Mr. Robot
    • MotoGP 2017
    • Climate Change
    • Flat Earth
  • Health
  • Opinion
  • Science
  • Tech
  • Crypto
  • Travel
  • Real Estate
  • Sports
  • More
    • Press Release
UK Herald
No Result
View All Result

UK and US publishers back move to block AI scrapers by default

by Justin Marsh
July 1, 2025
0
0
SHARES
Share on FacebookShare on TwitterReddit


ChatGPT blocked

Internet infrastructure provider Cloudflare is now blocking all AI scrapers accessing content by default in an industry first.

The move has been backed by more than a dozen major news and media publishers including the Associated Press, The Atlantic, Buzzfeed, Conde Nast, DMGT, Dotdash Meredith, Fortune, Gannett, The Independent, Sky News, Time and Ziff Davis.

Cloudflare said its default setting for new domains is now to block AI crawlers that don’t have permission or provide compensation, giving them more control.

Website owners can then decide if they want to give AI crawlers access to their content and how they can use it if they do.

This means, for example, that if a publisher did a deal with OpenAI they could specifically choose to allow access to only its GPTBot and no others. Or publishers could continue to exclude everything if they don’t feel they are getting any value back from scraping.

The AI crawlers are now able to state their purpose – for example whether they are used for training, inference or search – to help website owners decide whether to allow them to scrape.

Cloudflare co-founder and chief executive Matthew Prince said: “If the internet is going to survive the age of AI, we need to give publishers the control they deserve and build a new economic model that works for everyone – creators, consumers, tomorrow’s AI founders, and the future of the web itself.

“Original content is what makes the internet one of the greatest inventions in the last century, and it’s essential that creators continue making it. AI crawlers have been scraping content without limits.

“Our goal is to put the power back in the hands of creators, while still helping AI companies innovate. This is about safeguarding the future of a free and vibrant internet with a new model that works for everyone.”

Cloudflare is one of the world’s largest internet networks and its customers are said to account for 20% of traffic on the worldwide web.

Previously, in September last year, it had added the ability to block AI crawlers in one click which has since been adopted by more than one million customers.

Cloudflare published analysis showing which AI-only crawlers were doing the most scraping. OpenAI’s GPTBot had the biggest share at 30% in May (up from 5% in May 2024) followed by Anthropic’s ClaudeBot (21%, down from 27% last year).

Meta’s bot Meta-ExternalAgent took a 19% share of the scraping, Amazonbot was on 11% and Bytespider, linked to training models like Ernie and Tiktok-related AI, was on 7.2% (down from a much bigger 42% in 2024).

!function(){“use strict”;window.addEventListener(“message”,(function(a){if(void 0!==a.data[“datawrapper-height”]){var e=document.querySelectorAll(“iframe”);for(var t in a.data[“datawrapper-height”])for(var r,i=0;r=e[i];i++)if(r.contentWindow===a.source){var d=a.data[“datawrapper-height”][t]+”px”;r.style.height=d}}}))}();

However, Googlebot, which indexes content for Google Search, made up a 50% share of all AI and search web crawlers – up from 30% last year.

!function(){“use strict”;window.addEventListener(“message”,(function(a){if(void 0!==a.data[“datawrapper-height”]){var e=document.querySelectorAll(“iframe”);for(var t in a.data[“datawrapper-height”])for(var r,i=0;r=e[i];i++)if(r.contentWindow===a.source){var d=a.data[“datawrapper-height”][t]+”px”;r.style.height=d}}}))}();

Cloudflare is also experimenting with “pay per crawl”, currently in private beta testing, which would let website owners charge a flat, per-request price to crawlers.

If a crawler does not have a billing relationship with Cloudflare, meaning they can’t be financially charged, it would be blocked but take away the message that there could be a relationship with that website in future.

Publishers can currently set a flat price for all crawlers accessing their site but Cloudflare said this would mean they can allow certain scrapers to bypass charges, if they already have an arrangement for example.

Publishers react to AI scrapers blocking move

Rich Caccappolo, vice chairman at DMG Media which owns the Daily Mail, Metro and The i Paper, said: “DMGT welcomes Cloudflare’s initiative to help prevent the current epidemic of unauthorised scraping of websites by AI companies and their proxies, who are using copyrighted content for commercial purposes without paying for it.

“We support any innovation that creates a structured and transparent relationship between content creators and AI platforms and respects fundamental property rights.

“It will require a concerted effort by regulators, politicians, legislators, technology providers and content creators to build a new economic model for the AI era. We commend every step taken in pursuit of that goal.”

DMGT has previously financially backed Prorata.ai, which is building a model through which it attributes the source of content generated by AI chatbots to certain publishers and pays them accordingly.

The publishers speaking out in support of Cloudflare’s move are a mix of those who have done AI content deals with the likes of OpenAI and those who have not.

Conde Nast chief executive Roger Lynch said the new approach was a “game-changer for publishers and sets a new standard for how content is respected online. When AI companies can no longer take anything they want for free, it opens the door to sustainable innovation built on permission and partnership.”

Dotdash Meredith chief executive Neil Vogel said: “We have long said that AI platforms must fairly compensate publishers and creators to use our content. We can now limit access to our content to those AI partners willing to engage in fair arrangements.”

Renn Turiano, chief consumer and product officer of Gannett which owns USA Today and more than 200 US local publications, said that “blocking unauthorised scraping and the use of our original content without fair compensation is critically important” and that they believe the new tech “will help combat the theft of valuable IP”.

And Sky News executive chairman David Rhodes said: “This permission-based model will help secure the future of quality digital journalism, which is our commitment. Sky News is all about providing ‘the full story, first’– so we wanted to be among the first to join Cloudflare’s framework for setting fair terms of trade in news.

“We’ll help design the future of these services as video becomes an ever-larger part of both crawling and publishing.”

The change was also backed by tech bosses at the likes of Reddit and Pinterest. Reddit’s chief executive Steve Huffman said the “whole ecosystem of creators, platforms, web users and crawlers will be better when crawling is more transparent and controlled”.

More testimonials from news publisher leaders backing the Cloudflare deal can be found here.

The post UK and US publishers back move to block AI scrapers by default appeared first on Press Gazette.



Source link

Related Posts

Google’s site reputation abuse policy: the wrong solution to a real problem

Google’s site reputation abuse policy: the wrong solution to a real problem

by Justin Marsh
June 26, 2025
0

Google last year began cracking down on sites with strong search authority “abusing” their position by adding third-party commercial content such as an affiliate shopping arm or voucher code section. However many...

Top Invoice Finance Providers in the UK for 2025

Top Invoice Finance Providers in the UK for 2025

by Clara White
June 25, 2025
0

In today's fast-paced business environment, maintaining healthy cash flow is crucial for companies of all sizes. For many UK businesses, invoice finance has emerged as a vital solution to bridge the working...

Ian Key: North of England tabloid reporter who worked with integrity and passion

Ian Key: North of England tabloid reporter who worked with integrity and passion

by Justin Marsh
June 21, 2025
0

Ian Key – who spent many years working for the Daily Express, Daily Mail, Today and the News of the World – has died suddenly at home aged 78. He spent the majority...

News diary 16-22 June: US Tiktok sell-or-ban deadline, Chris Brown in UK court

News diary 16-22 June: US Tiktok sell-or-ban deadline, Chris Brown in UK court

by Justin Marsh
June 16, 2025
0

A look ahead at the key events leading the news agenda next week, from the team at Foresight News. Leading the week Following this week’s Spending Review, we’re on the lookout next...

Daily Star drops page three pictures of scantily dressed women in redesign

Daily Star drops page three pictures of scantily dressed women in redesign

by Justin Marsh
June 11, 2025
0

The Daily Star has ditched page three models amid a wider redesign across print and online. The revamp includes a new rhombus-shaped logo, first used on the masthead on Wednesday’s paper, and...

Telegraph withdraws banker school fees story after being deceived by source

Telegraph withdraws banker school fees story after being deceived by source

by Justin Marsh
June 6, 2025
0

The Telegraph has withdrawn a story about a rich banker hit by the cost of school fee increases after being deceived by a source. The article has been widely discussed on social...

Next Post
AXIOM-4: What will be the role of Shubhanshu Shukla in the mission? What did India send in space? What will India benefit?

AXIOM-4: What will be the role of Shubhanshu Shukla in the mission? What did India send in space? What will India benefit?

Popular News

AXIOM-4: What will be the role of Shubhanshu Shukla in the mission? What did India send in space? What will India benefit?

AXIOM-4: What will be the role of Shubhanshu Shukla in the mission? What did India send in space? What will India benefit?

July 1, 2025
UK and US publishers back move to block AI scrapers by default

UK and US publishers back move to block AI scrapers by default

July 1, 2025
Labour can’t just throw money at nuclear and hope it works

Labour can’t just throw money at nuclear and hope it works

June 30, 2025
Is it safe to travel to Turkey? Latest Foreign Office advice as wildfires rage in Izmir

Is it safe to travel to Turkey? Latest Foreign Office advice as wildfires rage in Izmir

June 30, 2025
Tesla Car Factory arrives at the owner’s house without a driver: the price of auto drive cars will be shocked

Tesla Car Factory arrives at the owner’s house without a driver: the price of auto drive cars will be shocked

June 29, 2025
Week-in-Review: Welfare rebellion has permanently damaged Starmer

Week-in-Review: Welfare rebellion has permanently damaged Starmer

June 28, 2025
I grew up in Yorkshire — here’s my insider’s guide to a weekend in the Dales

I grew up in Yorkshire — here’s my insider’s guide to a weekend in the Dales

June 27, 2025
UK Herald

All Rights Reserved © UK HERALD - The Voice of UK

Important Links

  • Publish Your article
  • Editorial Policy
  • Contact
  • Advertise

...

No Result
View All Result
  • Home
  • Politics
  • UK News
  • Business
  • Science
  • National
  • Entertainment
  • Gaming
  • Sports
  • Fashion
  • Lifestyle
  • Travel
  • Health
  • Food

All Rights Reserved © UK HERALD - The Voice of UK