• Publish Your article
  • Editorial Policy
  • Contact
  • Advertise
Tuesday, July 22, 2025
No Result
View All Result
UK Herald
  • Home
  • Politics
  • Business
  • Entertainment
    • All
    • Sports
    England rugby stadium Twickenham given new name after more than 100 years in shock new deal

    England rugby stadium Twickenham given new name after more than 100 years in shock new deal

    Peter Morgan dead at 65: Former Wales and Lions rugby star who became a politician passes away as club pays tribute

    Peter Morgan dead at 65: Former Wales and Lions rugby star who became a politician passes away as club pays tribute

    Horse racing tips: Unexposed Group 1 contender can stun the big guns at 14-1

    Horse racing tips: Unexposed Group 1 contender can stun the big guns at 14-1

    Woman ‘raped seven times by two French rugby stars who left her riddled with bite marks & with horror injuries’

    Woman ‘raped seven times by two French rugby stars who left her riddled with bite marks & with horror injuries’

    Horse racing tips: Gary Moore’s charge can gain revenge after falling last time out

    Horse racing tips: Gary Moore’s charge can gain revenge after falling last time out

    Ian Buckett dead at 56: Former Wales rugby star who was ‘admired and feared equally’ dies as tributes pour in

    Ian Buckett dead at 56: Former Wales rugby star who was ‘admired and feared equally’ dies as tributes pour in

    Horse racing tips: Bash the bookies with these longshots including 9-1 fancy

    Horse racing tips: Bash the bookies with these longshots including 9-1 fancy

    Shayne Philpott dead at 58 – New Zealand All Blacks rugby legend dies after suffering ‘medical event’

    Shayne Philpott dead at 58 – New Zealand All Blacks rugby legend dies after suffering ‘medical event’

    Horse racing tips: This 7-1 chance appears to have been laid out for race he won last year

    Horse racing tips: This 7-1 chance appears to have been laid out for race he won last year

  • Lifestyle
    • All
    • Fashion
    • food
    • Health
    • Travel
    British airport announces ‘major redesign’ including new restaurants and runway views 

    British airport announces ‘major redesign’ including new restaurants and runway views 

    ‘Like the Caribbean’: Ibiza’s quiet little sister with lunar beaches and turquoise oceans 

    ‘Like the Caribbean’: Ibiza’s quiet little sister with lunar beaches and turquoise oceans 

    This street in Europe is just 50cm wide and has traffic lights so tourists don’t get stuck

    This street in Europe is just 50cm wide and has traffic lights so tourists don’t get stuck

    My holiday breakfast buffet hack is controversial but it’s a money saver

    My holiday breakfast buffet hack is controversial but it’s a money saver

    UK company launches £5,000 sex cruise — these are the rules passengers must follow

    UK company launches £5,000 sex cruise — these are the rules passengers must follow

    Ryanair calls for ‘urgent’ reform as French air strikes stretch to four days 

    Ryanair calls for ‘urgent’ reform as French air strikes stretch to four days 

    Is it safe to travel to Crete? Latest tourist advice amid wildfire evacuations

    Is it safe to travel to Crete? Latest tourist advice amid wildfire evacuations

    Is it safe to travel to Turkey? Latest Foreign Office advice as wildfires rage in Izmir

    Is it safe to travel to Turkey? Latest Foreign Office advice as wildfires rage in Izmir

    I grew up in Yorkshire — here’s my insider’s guide to a weekend in the Dales

    I grew up in Yorkshire — here’s my insider’s guide to a weekend in the Dales

    All the food and drink banned on EasyJet, Ryanair, Jet2 and TUI flights

    All the food and drink banned on EasyJet, Ryanair, Jet2 and TUI flights

    Trending Tags

    • Golden Globes
    • Mr. Robot
    • MotoGP 2017
    • Climate Change
    • Flat Earth
  • Health
  • Opinion
  • Science
  • Tech
  • Crypto
  • Travel
  • Real Estate
  • Sports
  • More
    • Press Release
UK Herald
No Result
View All Result

UK and US publishers back move to block AI scrapers by default

by Justin Marsh
July 1, 2025
0
0
SHARES
Share on FacebookShare on TwitterReddit


ChatGPT blocked

Internet infrastructure provider Cloudflare is now blocking all AI scrapers accessing content by default in an industry first.

The move has been backed by more than a dozen major news and media publishers including the Associated Press, The Atlantic, Buzzfeed, Conde Nast, DMGT, Dotdash Meredith, Fortune, Gannett, The Independent, Sky News, Time and Ziff Davis.

Cloudflare said its default setting for new domains is now to block AI crawlers that don’t have permission or provide compensation, giving them more control.

Website owners can then decide if they want to give AI crawlers access to their content and how they can use it if they do.

This means, for example, that if a publisher did a deal with OpenAI they could specifically choose to allow access to only its GPTBot and no others. Or publishers could continue to exclude everything if they don’t feel they are getting any value back from scraping.

The AI crawlers are now able to state their purpose – for example whether they are used for training, inference or search – to help website owners decide whether to allow them to scrape.

Cloudflare co-founder and chief executive Matthew Prince said: “If the internet is going to survive the age of AI, we need to give publishers the control they deserve and build a new economic model that works for everyone – creators, consumers, tomorrow’s AI founders, and the future of the web itself.

“Original content is what makes the internet one of the greatest inventions in the last century, and it’s essential that creators continue making it. AI crawlers have been scraping content without limits.

“Our goal is to put the power back in the hands of creators, while still helping AI companies innovate. This is about safeguarding the future of a free and vibrant internet with a new model that works for everyone.”

Cloudflare is one of the world’s largest internet networks and its customers are said to account for 20% of traffic on the worldwide web.

Previously, in September last year, it had added the ability to block AI crawlers in one click which has since been adopted by more than one million customers.

Cloudflare published analysis showing which AI-only crawlers were doing the most scraping. OpenAI’s GPTBot had the biggest share at 30% in May (up from 5% in May 2024) followed by Anthropic’s ClaudeBot (21%, down from 27% last year).

Meta’s bot Meta-ExternalAgent took a 19% share of the scraping, Amazonbot was on 11% and Bytespider, linked to training models like Ernie and Tiktok-related AI, was on 7.2% (down from a much bigger 42% in 2024).

!function(){“use strict”;window.addEventListener(“message”,(function(a){if(void 0!==a.data[“datawrapper-height”]){var e=document.querySelectorAll(“iframe”);for(var t in a.data[“datawrapper-height”])for(var r,i=0;r=e[i];i++)if(r.contentWindow===a.source){var d=a.data[“datawrapper-height”][t]+”px”;r.style.height=d}}}))}();

However, Googlebot, which indexes content for Google Search, made up a 50% share of all AI and search web crawlers – up from 30% last year.

!function(){“use strict”;window.addEventListener(“message”,(function(a){if(void 0!==a.data[“datawrapper-height”]){var e=document.querySelectorAll(“iframe”);for(var t in a.data[“datawrapper-height”])for(var r,i=0;r=e[i];i++)if(r.contentWindow===a.source){var d=a.data[“datawrapper-height”][t]+”px”;r.style.height=d}}}))}();

Cloudflare is also experimenting with “pay per crawl”, currently in private beta testing, which would let website owners charge a flat, per-request price to crawlers.

If a crawler does not have a billing relationship with Cloudflare, meaning they can’t be financially charged, it would be blocked but take away the message that there could be a relationship with that website in future.

Publishers can currently set a flat price for all crawlers accessing their site but Cloudflare said this would mean they can allow certain scrapers to bypass charges, if they already have an arrangement for example.

Publishers react to AI scrapers blocking move

Rich Caccappolo, vice chairman at DMG Media which owns the Daily Mail, Metro and The i Paper, said: “DMGT welcomes Cloudflare’s initiative to help prevent the current epidemic of unauthorised scraping of websites by AI companies and their proxies, who are using copyrighted content for commercial purposes without paying for it.

“We support any innovation that creates a structured and transparent relationship between content creators and AI platforms and respects fundamental property rights.

“It will require a concerted effort by regulators, politicians, legislators, technology providers and content creators to build a new economic model for the AI era. We commend every step taken in pursuit of that goal.”

DMGT has previously financially backed Prorata.ai, which is building a model through which it attributes the source of content generated by AI chatbots to certain publishers and pays them accordingly.

The publishers speaking out in support of Cloudflare’s move are a mix of those who have done AI content deals with the likes of OpenAI and those who have not.

Conde Nast chief executive Roger Lynch said the new approach was a “game-changer for publishers and sets a new standard for how content is respected online. When AI companies can no longer take anything they want for free, it opens the door to sustainable innovation built on permission and partnership.”

Dotdash Meredith chief executive Neil Vogel said: “We have long said that AI platforms must fairly compensate publishers and creators to use our content. We can now limit access to our content to those AI partners willing to engage in fair arrangements.”

Renn Turiano, chief consumer and product officer of Gannett which owns USA Today and more than 200 US local publications, said that “blocking unauthorised scraping and the use of our original content without fair compensation is critically important” and that they believe the new tech “will help combat the theft of valuable IP”.

And Sky News executive chairman David Rhodes said: “This permission-based model will help secure the future of quality digital journalism, which is our commitment. Sky News is all about providing ‘the full story, first’– so we wanted to be among the first to join Cloudflare’s framework for setting fair terms of trade in news.

“We’ll help design the future of these services as video becomes an ever-larger part of both crawling and publishing.”

The change was also backed by tech bosses at the likes of Reddit and Pinterest. Reddit’s chief executive Steve Huffman said the “whole ecosystem of creators, platforms, web users and crawlers will be better when crawling is more transparent and controlled”.

More testimonials from news publisher leaders backing the Cloudflare deal can be found here.

The post UK and US publishers back move to block AI scrapers by default appeared first on Press Gazette.



Source link

Related Posts

PR agency sells AI tool which sends out automated expert comment to journalists

PR agency sells AI tool which sends out automated expert comment to journalists

by Justin Marsh
July 21, 2025
0

A PR agency is selling an AI tool that automatically answers pitches from journalists on services such as ResponseSource, HARO and Qwoted. The AI tool, called Synapse (not be confused with PR...

How social and video drive revenue for UK podcast network Crowd

How social and video drive revenue for UK podcast network Crowd

by Justin Marsh
July 16, 2025
0

“We don’t really call them podcasts any more,” says Mike Carr, CEO of one of Europe’s fastest-growing podcast networks. Carr co-founded Crowd in Manchester in 2020 along with three other ex-BBC Sports...

Journalist says 4,000 fake AI news websites created to game Google algorithms

Journalist says 4,000 fake AI news websites created to game Google algorithms

by Justin Marsh
July 11, 2025
0

More than 4,000 fake news websites powered by generative AI have been set up to game Google Discover and search, according to award-winning French journalist Jean-Marc Manach. The sites are largely in...

News diary 7-13 July: 7/7 bombings 20th anniversary, Netanyahu at White House, Macron state visit to UK

by Justin Marsh
July 6, 2025
0

A look ahead at the key events leading the news agenda next week, from the team at Foresight News. Leading the week Monday (July 7): St Paul’s service to mark the 20th...

Google’s site reputation abuse policy: the wrong solution to a real problem

Google’s site reputation abuse policy: the wrong solution to a real problem

by Justin Marsh
June 26, 2025
0

Google last year began cracking down on sites with strong search authority “abusing” their position by adding third-party commercial content such as an affiliate shopping arm or voucher code section. However many...

Top Invoice Finance Providers in the UK for 2025

Top Invoice Finance Providers in the UK for 2025

by Clara White
June 25, 2025
0

In today's fast-paced business environment, maintaining healthy cash flow is crucial for companies of all sizes. For many UK businesses, invoice finance has emerged as a vital solution to bridge the working...

Next Post
AXIOM-4: What will be the role of Shubhanshu Shukla in the mission? What did India send in space? What will India benefit?

AXIOM-4: What will be the role of Shubhanshu Shukla in the mission? What did India send in space? What will India benefit?

Popular News

‘Deluded’: Conservatives condemned after strategist says party has ‘done the apologies’

‘Deluded’: Conservatives condemned after strategist says party has ‘done the apologies’

July 22, 2025
British airport announces ‘major redesign’ including new restaurants and runway views 

British airport announces ‘major redesign’ including new restaurants and runway views 

July 21, 2025
PR agency sells AI tool which sends out automated expert comment to journalists

PR agency sells AI tool which sends out automated expert comment to journalists

July 21, 2025
Keir Starmer could learn to fear a left insurgency

Keir Starmer could learn to fear a left insurgency

July 20, 2025
MPs to vote on making Premier League games free-to-air

MPs to vote on making Premier League games free-to-air

July 18, 2025
Best hot hatchbacks to buy in 2025

Best hot hatchbacks to buy in 2025

July 18, 2025
‘Like the Caribbean’: Ibiza’s quiet little sister with lunar beaches and turquoise oceans 

‘Like the Caribbean’: Ibiza’s quiet little sister with lunar beaches and turquoise oceans 

July 18, 2025
UK Herald

All Rights Reserved © UK HERALD - The Voice of UK

Important Links

  • Publish Your article
  • Editorial Policy
  • Contact
  • Advertise

...

No Result
View All Result
  • Home
  • Politics
  • UK News
  • Business
  • Science
  • National
  • Entertainment
  • Gaming
  • Sports
  • Fashion
  • Lifestyle
  • Travel
  • Health
  • Food

All Rights Reserved © UK HERALD - The Voice of UK