• Publish Your article
  • Editorial Policy
  • Contact
  • Advertise
Saturday, October 25, 2025
No Result
View All Result
UK Herald
  • Home
  • Politics
  • Business
  • Entertainment
    • All
    • Sports
    England rugby stadium Twickenham given new name after more than 100 years in shock new deal

    England rugby stadium Twickenham given new name after more than 100 years in shock new deal

    Peter Morgan dead at 65: Former Wales and Lions rugby star who became a politician passes away as club pays tribute

    Peter Morgan dead at 65: Former Wales and Lions rugby star who became a politician passes away as club pays tribute

    Horse racing tips: Unexposed Group 1 contender can stun the big guns at 14-1

    Horse racing tips: Unexposed Group 1 contender can stun the big guns at 14-1

    Woman ‘raped seven times by two French rugby stars who left her riddled with bite marks & with horror injuries’

    Woman ‘raped seven times by two French rugby stars who left her riddled with bite marks & with horror injuries’

    Horse racing tips: Gary Moore’s charge can gain revenge after falling last time out

    Horse racing tips: Gary Moore’s charge can gain revenge after falling last time out

    Ian Buckett dead at 56: Former Wales rugby star who was ‘admired and feared equally’ dies as tributes pour in

    Ian Buckett dead at 56: Former Wales rugby star who was ‘admired and feared equally’ dies as tributes pour in

    Horse racing tips: Bash the bookies with these longshots including 9-1 fancy

    Horse racing tips: Bash the bookies with these longshots including 9-1 fancy

    Shayne Philpott dead at 58 – New Zealand All Blacks rugby legend dies after suffering ‘medical event’

    Shayne Philpott dead at 58 – New Zealand All Blacks rugby legend dies after suffering ‘medical event’

    Horse racing tips: This 7-1 chance appears to have been laid out for race he won last year

    Horse racing tips: This 7-1 chance appears to have been laid out for race he won last year

  • Lifestyle
    • All
    • Fashion
    • food
    • Health
    • Travel
    I visited this lesser-known European gem and spent just £300 in a week

    I visited this lesser-known European gem and spent just £300 in a week

    I stayed in Mauritius’ ‘magic place’ – this is my honest review

    I stayed in Mauritius’ ‘magic place’ – this is my honest review

    The exact dates you need to book off in 2026 to get 56 days of holiday

    The exact dates you need to book off in 2026 to get 56 days of holiday

    Italy’s ‘less glitzy’ gateway destination named one of 2026’s best places to visit

    Italy’s ‘less glitzy’ gateway destination named one of 2026’s best places to visit

    This historic Japanese city is hiking its tourist tax by 900% to stop you coming

    This historic Japanese city is hiking its tourist tax by 900% to stop you coming

    Major airline removes carry-on bag sizers from boarding gates

    Major airline removes carry-on bag sizers from boarding gates

    Qatar Airways Al Mourjan Business Lounge review: ‘It doesn’t get bigger than this…’

    Qatar Airways Al Mourjan Business Lounge review: ‘It doesn’t get bigger than this…’

    What is Bleach or Facial, Best for Skin?

    What is Bleach or Facial, Best for Skin?

    Tourists are flocking to a mind-bending Chinese megacity where Google Maps doesn’t work

    Tourists are flocking to a mind-bending Chinese megacity where Google Maps doesn’t work

    An ESTA change has just made travelling to America more expensive for Brits

    An ESTA change has just made travelling to America more expensive for Brits

    Trending Tags

    • Golden Globes
    • Mr. Robot
    • MotoGP 2017
    • Climate Change
    • Flat Earth
  • Health
  • Opinion
  • Science
  • Tech
  • Crypto
  • Travel
  • Real Estate
  • Sports
  • More
    • Press Release
UK Herald
No Result
View All Result

UK and US publishers back move to block AI scrapers by default

by Justin Marsh
July 1, 2025
0
0
SHARES
Share on FacebookShare on TwitterReddit


ChatGPT blocked

Internet infrastructure provider Cloudflare is now blocking all AI scrapers accessing content by default in an industry first.

The move has been backed by more than a dozen major news and media publishers including the Associated Press, The Atlantic, Buzzfeed, Conde Nast, DMGT, Dotdash Meredith, Fortune, Gannett, The Independent, Sky News, Time and Ziff Davis.

Cloudflare said its default setting for new domains is now to block AI crawlers that don’t have permission or provide compensation, giving them more control.

Website owners can then decide if they want to give AI crawlers access to their content and how they can use it if they do.

This means, for example, that if a publisher did a deal with OpenAI they could specifically choose to allow access to only its GPTBot and no others. Or publishers could continue to exclude everything if they don’t feel they are getting any value back from scraping.

The AI crawlers are now able to state their purpose – for example whether they are used for training, inference or search – to help website owners decide whether to allow them to scrape.

Cloudflare co-founder and chief executive Matthew Prince said: “If the internet is going to survive the age of AI, we need to give publishers the control they deserve and build a new economic model that works for everyone – creators, consumers, tomorrow’s AI founders, and the future of the web itself.

“Original content is what makes the internet one of the greatest inventions in the last century, and it’s essential that creators continue making it. AI crawlers have been scraping content without limits.

“Our goal is to put the power back in the hands of creators, while still helping AI companies innovate. This is about safeguarding the future of a free and vibrant internet with a new model that works for everyone.”

Cloudflare is one of the world’s largest internet networks and its customers are said to account for 20% of traffic on the worldwide web.

Previously, in September last year, it had added the ability to block AI crawlers in one click which has since been adopted by more than one million customers.

Cloudflare published analysis showing which AI-only crawlers were doing the most scraping. OpenAI’s GPTBot had the biggest share at 30% in May (up from 5% in May 2024) followed by Anthropic’s ClaudeBot (21%, down from 27% last year).

Meta’s bot Meta-ExternalAgent took a 19% share of the scraping, Amazonbot was on 11% and Bytespider, linked to training models like Ernie and Tiktok-related AI, was on 7.2% (down from a much bigger 42% in 2024).

!function(){“use strict”;window.addEventListener(“message”,(function(a){if(void 0!==a.data[“datawrapper-height”]){var e=document.querySelectorAll(“iframe”);for(var t in a.data[“datawrapper-height”])for(var r,i=0;r=e[i];i++)if(r.contentWindow===a.source){var d=a.data[“datawrapper-height”][t]+”px”;r.style.height=d}}}))}();

However, Googlebot, which indexes content for Google Search, made up a 50% share of all AI and search web crawlers – up from 30% last year.

!function(){“use strict”;window.addEventListener(“message”,(function(a){if(void 0!==a.data[“datawrapper-height”]){var e=document.querySelectorAll(“iframe”);for(var t in a.data[“datawrapper-height”])for(var r,i=0;r=e[i];i++)if(r.contentWindow===a.source){var d=a.data[“datawrapper-height”][t]+”px”;r.style.height=d}}}))}();

Cloudflare is also experimenting with “pay per crawl”, currently in private beta testing, which would let website owners charge a flat, per-request price to crawlers.

If a crawler does not have a billing relationship with Cloudflare, meaning they can’t be financially charged, it would be blocked but take away the message that there could be a relationship with that website in future.

Publishers can currently set a flat price for all crawlers accessing their site but Cloudflare said this would mean they can allow certain scrapers to bypass charges, if they already have an arrangement for example.

Publishers react to AI scrapers blocking move

Rich Caccappolo, vice chairman at DMG Media which owns the Daily Mail, Metro and The i Paper, said: “DMGT welcomes Cloudflare’s initiative to help prevent the current epidemic of unauthorised scraping of websites by AI companies and their proxies, who are using copyrighted content for commercial purposes without paying for it.

“We support any innovation that creates a structured and transparent relationship between content creators and AI platforms and respects fundamental property rights.

“It will require a concerted effort by regulators, politicians, legislators, technology providers and content creators to build a new economic model for the AI era. We commend every step taken in pursuit of that goal.”

DMGT has previously financially backed Prorata.ai, which is building a model through which it attributes the source of content generated by AI chatbots to certain publishers and pays them accordingly.

The publishers speaking out in support of Cloudflare’s move are a mix of those who have done AI content deals with the likes of OpenAI and those who have not.

Conde Nast chief executive Roger Lynch said the new approach was a “game-changer for publishers and sets a new standard for how content is respected online. When AI companies can no longer take anything they want for free, it opens the door to sustainable innovation built on permission and partnership.”

Dotdash Meredith chief executive Neil Vogel said: “We have long said that AI platforms must fairly compensate publishers and creators to use our content. We can now limit access to our content to those AI partners willing to engage in fair arrangements.”

Renn Turiano, chief consumer and product officer of Gannett which owns USA Today and more than 200 US local publications, said that “blocking unauthorised scraping and the use of our original content without fair compensation is critically important” and that they believe the new tech “will help combat the theft of valuable IP”.

And Sky News executive chairman David Rhodes said: “This permission-based model will help secure the future of quality digital journalism, which is our commitment. Sky News is all about providing ‘the full story, first’– so we wanted to be among the first to join Cloudflare’s framework for setting fair terms of trade in news.

“We’ll help design the future of these services as video becomes an ever-larger part of both crawling and publishing.”

The change was also backed by tech bosses at the likes of Reddit and Pinterest. Reddit’s chief executive Steve Huffman said the “whole ecosystem of creators, platforms, web users and crawlers will be better when crawling is more transparent and controlled”.

More testimonials from news publisher leaders backing the Cloudflare deal can be found here.

The post UK and US publishers back move to block AI scrapers by default appeared first on Press Gazette.



Source link

Related Posts

Are press trip freebies taxable income for freelances and influencers?

Are press trip freebies taxable income for freelances and influencers?

by Justin Marsh
October 24, 2025
0

Are press trips classed as taxable income for freelance journalists and influencers? A recent Linkedin post by a travel blogger and content creator Claire Hall suggested they are and and has been...

Ofcom orders BBC to air on TV finding that Gaza film broke broadcasting rules

Ofcom orders BBC to air on TV finding that Gaza film broke broadcasting rules

by Justin Marsh
October 19, 2025
0

The BBC’s use of the son of a Hamas official as a narrator for a Gaza documentary amounted to a “serious breach” of the Broadcasting Code, Ofcom has ruled. As a sanction...

Intelligent pricing: Lessons in precision from Aller Media Nordic

by Justin Marsh
October 14, 2025
0

For years, pricing was treated as a back-office function in many publishing organisations — necessary, but not strategic. Yet as economic pressures mount and the search for sustainable reader revenue intensifies, getting...

Daily Mail appoints ‘world’s first’ influencer correspondent

Daily Mail appoints ‘world’s first’ influencer correspondent

by Justin Marsh
October 9, 2025
0

The Daily Mail has appointed showbiz journalist Molly Clayton to become the title’s first influencer correspondent. Clayton joined the Mail as a trainee in 2020 and has been showbiz correspondent working across...

News diary 6-12 October: Labour deputy leader ballot opens, Greater Anglia trains nationalised, Nobel Prizes announced

by Justin Marsh
October 4, 2025
0

A look ahead at the key events leading the news agenda next week, from the team at Foresight News. Leading the week Monday (October 6): Shadow Chancellor Mel Stride delivers speech on...

Newsletter tips from the New York Times: Visuals, strong host, short, intimate

Newsletter tips from the New York Times: Visuals, strong host, short, intimate

by Justin Marsh
September 29, 2025
0

New York Times editorial director for newsletters Jodi Rudoren told Press Gazette what makes a good email update as the title launched a new daily edition. The World has a five-strong editorial...

Next Post
AXIOM-4: What will be the role of Shubhanshu Shukla in the mission? What did India send in space? What will India benefit?

AXIOM-4: What will be the role of Shubhanshu Shukla in the mission? What did India send in space? What will India benefit?

Popular News

Starmer is governed by polls, not principles – voters can sense it

Starmer is governed by polls, not principles – voters can sense it

October 24, 2025
Are press trip freebies taxable income for freelances and influencers?

Are press trip freebies taxable income for freelances and influencers?

October 24, 2025
Back electoral reform to embed ‘progressive majority’ and thwart Farage, Starmer urged

Back electoral reform to embed ‘progressive majority’ and thwart Farage, Starmer urged

October 22, 2025
I visited this lesser-known European gem and spent just £300 in a week

I visited this lesser-known European gem and spent just £300 in a week

October 22, 2025
Week-in-Review: Starmer and Badenoch repel their pretenders – for now

Week-in-Review: Starmer and Badenoch repel their pretenders – for now

October 20, 2025
How to pull over safely for emergency vehicles

How to pull over safely for emergency vehicles

October 19, 2025
I stayed in Mauritius’ ‘magic place’ – this is my honest review

I stayed in Mauritius’ ‘magic place’ – this is my honest review

October 19, 2025
UK Herald

All Rights Reserved © UK HERALD - The Voice of UK

Important Links

  • Publish Your article
  • Editorial Policy
  • Contact
  • Advertise

...

No Result
View All Result
  • Home
  • Politics
  • UK News
  • Business
  • Science
  • National
  • Entertainment
  • Gaming
  • Sports
  • Fashion
  • Lifestyle
  • Travel
  • Health
  • Food

All Rights Reserved © UK HERALD - The Voice of UK