• Publish Your article
  • Editorial Policy
  • Contact
  • Advertise
Saturday, October 4, 2025
No Result
View All Result
UK Herald
  • Home
  • Politics
  • Business
  • Entertainment
    • All
    • Sports
    England rugby stadium Twickenham given new name after more than 100 years in shock new deal

    England rugby stadium Twickenham given new name after more than 100 years in shock new deal

    Peter Morgan dead at 65: Former Wales and Lions rugby star who became a politician passes away as club pays tribute

    Peter Morgan dead at 65: Former Wales and Lions rugby star who became a politician passes away as club pays tribute

    Horse racing tips: Unexposed Group 1 contender can stun the big guns at 14-1

    Horse racing tips: Unexposed Group 1 contender can stun the big guns at 14-1

    Woman ‘raped seven times by two French rugby stars who left her riddled with bite marks & with horror injuries’

    Woman ‘raped seven times by two French rugby stars who left her riddled with bite marks & with horror injuries’

    Horse racing tips: Gary Moore’s charge can gain revenge after falling last time out

    Horse racing tips: Gary Moore’s charge can gain revenge after falling last time out

    Ian Buckett dead at 56: Former Wales rugby star who was ‘admired and feared equally’ dies as tributes pour in

    Ian Buckett dead at 56: Former Wales rugby star who was ‘admired and feared equally’ dies as tributes pour in

    Horse racing tips: Bash the bookies with these longshots including 9-1 fancy

    Horse racing tips: Bash the bookies with these longshots including 9-1 fancy

    Shayne Philpott dead at 58 – New Zealand All Blacks rugby legend dies after suffering ‘medical event’

    Shayne Philpott dead at 58 – New Zealand All Blacks rugby legend dies after suffering ‘medical event’

    Horse racing tips: This 7-1 chance appears to have been laid out for race he won last year

    Horse racing tips: This 7-1 chance appears to have been laid out for race he won last year

  • Lifestyle
    • All
    • Fashion
    • food
    • Health
    • Travel
    What is Bleach or Facial, Best for Skin?

    What is Bleach or Facial, Best for Skin?

    Tourists are flocking to a mind-bending Chinese megacity where Google Maps doesn’t work

    Tourists are flocking to a mind-bending Chinese megacity where Google Maps doesn’t work

    An ESTA change has just made travelling to America more expensive for Brits

    An ESTA change has just made travelling to America more expensive for Brits

    The best Maldives alternatives that are cheaper — and three are in Europe

    The best Maldives alternatives that are cheaper — and three are in Europe

    Italy strikes for Gaza: What tourists need to know amid travel disruption in Rome and Milan

    Italy strikes for Gaza: What tourists need to know amid travel disruption in Rome and Milan

    ‘Dystopian’ cruise ship divides travellers — and gives a glimpse into the future

    ‘Dystopian’ cruise ship divides travellers — and gives a glimpse into the future

    Airline launches flights from £9 to European hotspots including Majorca, Alicante and Seville

    Airline launches flights from £9 to European hotspots including Majorca, Alicante and Seville

    I feel like a stranger in the UK after 21 years abroad

    I feel like a stranger in the UK after 21 years abroad

    Is it safe to travel to Poland right now? Latest advice after Russian drone attack

    Is it safe to travel to Poland right now? Latest advice after Russian drone attack

    I stayed at the new London hotel with immunity-boosting IV drips and a free photobooth

    I stayed at the new London hotel with immunity-boosting IV drips and a free photobooth

    Trending Tags

    • Golden Globes
    • Mr. Robot
    • MotoGP 2017
    • Climate Change
    • Flat Earth
  • Health
  • Opinion
  • Science
  • Tech
  • Crypto
  • Travel
  • Real Estate
  • Sports
  • More
    • Press Release
UK Herald
No Result
View All Result

UK and US publishers back move to block AI scrapers by default

by Justin Marsh
July 1, 2025
0
0
SHARES
Share on FacebookShare on TwitterReddit


ChatGPT blocked

Internet infrastructure provider Cloudflare is now blocking all AI scrapers accessing content by default in an industry first.

The move has been backed by more than a dozen major news and media publishers including the Associated Press, The Atlantic, Buzzfeed, Conde Nast, DMGT, Dotdash Meredith, Fortune, Gannett, The Independent, Sky News, Time and Ziff Davis.

Cloudflare said its default setting for new domains is now to block AI crawlers that don’t have permission or provide compensation, giving them more control.

Website owners can then decide if they want to give AI crawlers access to their content and how they can use it if they do.

This means, for example, that if a publisher did a deal with OpenAI they could specifically choose to allow access to only its GPTBot and no others. Or publishers could continue to exclude everything if they don’t feel they are getting any value back from scraping.

The AI crawlers are now able to state their purpose – for example whether they are used for training, inference or search – to help website owners decide whether to allow them to scrape.

Cloudflare co-founder and chief executive Matthew Prince said: “If the internet is going to survive the age of AI, we need to give publishers the control they deserve and build a new economic model that works for everyone – creators, consumers, tomorrow’s AI founders, and the future of the web itself.

“Original content is what makes the internet one of the greatest inventions in the last century, and it’s essential that creators continue making it. AI crawlers have been scraping content without limits.

“Our goal is to put the power back in the hands of creators, while still helping AI companies innovate. This is about safeguarding the future of a free and vibrant internet with a new model that works for everyone.”

Cloudflare is one of the world’s largest internet networks and its customers are said to account for 20% of traffic on the worldwide web.

Previously, in September last year, it had added the ability to block AI crawlers in one click which has since been adopted by more than one million customers.

Cloudflare published analysis showing which AI-only crawlers were doing the most scraping. OpenAI’s GPTBot had the biggest share at 30% in May (up from 5% in May 2024) followed by Anthropic’s ClaudeBot (21%, down from 27% last year).

Meta’s bot Meta-ExternalAgent took a 19% share of the scraping, Amazonbot was on 11% and Bytespider, linked to training models like Ernie and Tiktok-related AI, was on 7.2% (down from a much bigger 42% in 2024).

!function(){“use strict”;window.addEventListener(“message”,(function(a){if(void 0!==a.data[“datawrapper-height”]){var e=document.querySelectorAll(“iframe”);for(var t in a.data[“datawrapper-height”])for(var r,i=0;r=e[i];i++)if(r.contentWindow===a.source){var d=a.data[“datawrapper-height”][t]+”px”;r.style.height=d}}}))}();

However, Googlebot, which indexes content for Google Search, made up a 50% share of all AI and search web crawlers – up from 30% last year.

!function(){“use strict”;window.addEventListener(“message”,(function(a){if(void 0!==a.data[“datawrapper-height”]){var e=document.querySelectorAll(“iframe”);for(var t in a.data[“datawrapper-height”])for(var r,i=0;r=e[i];i++)if(r.contentWindow===a.source){var d=a.data[“datawrapper-height”][t]+”px”;r.style.height=d}}}))}();

Cloudflare is also experimenting with “pay per crawl”, currently in private beta testing, which would let website owners charge a flat, per-request price to crawlers.

If a crawler does not have a billing relationship with Cloudflare, meaning they can’t be financially charged, it would be blocked but take away the message that there could be a relationship with that website in future.

Publishers can currently set a flat price for all crawlers accessing their site but Cloudflare said this would mean they can allow certain scrapers to bypass charges, if they already have an arrangement for example.

Publishers react to AI scrapers blocking move

Rich Caccappolo, vice chairman at DMG Media which owns the Daily Mail, Metro and The i Paper, said: “DMGT welcomes Cloudflare’s initiative to help prevent the current epidemic of unauthorised scraping of websites by AI companies and their proxies, who are using copyrighted content for commercial purposes without paying for it.

“We support any innovation that creates a structured and transparent relationship between content creators and AI platforms and respects fundamental property rights.

“It will require a concerted effort by regulators, politicians, legislators, technology providers and content creators to build a new economic model for the AI era. We commend every step taken in pursuit of that goal.”

DMGT has previously financially backed Prorata.ai, which is building a model through which it attributes the source of content generated by AI chatbots to certain publishers and pays them accordingly.

The publishers speaking out in support of Cloudflare’s move are a mix of those who have done AI content deals with the likes of OpenAI and those who have not.

Conde Nast chief executive Roger Lynch said the new approach was a “game-changer for publishers and sets a new standard for how content is respected online. When AI companies can no longer take anything they want for free, it opens the door to sustainable innovation built on permission and partnership.”

Dotdash Meredith chief executive Neil Vogel said: “We have long said that AI platforms must fairly compensate publishers and creators to use our content. We can now limit access to our content to those AI partners willing to engage in fair arrangements.”

Renn Turiano, chief consumer and product officer of Gannett which owns USA Today and more than 200 US local publications, said that “blocking unauthorised scraping and the use of our original content without fair compensation is critically important” and that they believe the new tech “will help combat the theft of valuable IP”.

And Sky News executive chairman David Rhodes said: “This permission-based model will help secure the future of quality digital journalism, which is our commitment. Sky News is all about providing ‘the full story, first’– so we wanted to be among the first to join Cloudflare’s framework for setting fair terms of trade in news.

“We’ll help design the future of these services as video becomes an ever-larger part of both crawling and publishing.”

The change was also backed by tech bosses at the likes of Reddit and Pinterest. Reddit’s chief executive Steve Huffman said the “whole ecosystem of creators, platforms, web users and crawlers will be better when crawling is more transparent and controlled”.

More testimonials from news publisher leaders backing the Cloudflare deal can be found here.

The post UK and US publishers back move to block AI scrapers by default appeared first on Press Gazette.



Source link

Related Posts

Newsletter tips from the New York Times: Visuals, strong host, short, intimate

Newsletter tips from the New York Times: Visuals, strong host, short, intimate

by Justin Marsh
September 29, 2025
0

New York Times editorial director for newsletters Jodi Rudoren told Press Gazette what makes a good email update as the title launched a new daily edition. The World has a five-strong editorial...

Hearst UK losses grow as search changes hit digital ad revenue

Hearst UK losses grow as search changes hit digital ad revenue

by Justin Marsh
September 24, 2025
0

A second year of traffic turbulence led to continued revenue decline at Hearst UK in 2024, according to newly-filed accounts. Total revenue at National Magazine Company Ltd (the trading name of Hearst...

Journalist going to trial after highlighting porn shared by ex-MP on X

Journalist going to trial after highlighting porn shared by ex-MP on X

by Justin Marsh
September 19, 2025
0

Retired Brighton-based journalist Greg Hadfield will go to trial over drawing attention to an obscene X message posted by the account of former Labour MP Ivor Caplin. Hadfield applied for the court...

News diary 15-21 September: Trump visits UK, interest rate decision, sentencing of Constance Marten and Mark Gordon

News diary 15-21 September: Trump visits UK, interest rate decision, sentencing of Constance Marten and Mark Gordon

by Justin Marsh
September 14, 2025
0

The week ahead looks to be dominated by the visit of Donald Trump for the UK, as the US President arrives Tuesday and takes part in a State Banquet with King Charles...

Lifestyle brand Refinery29 closes UK office and lays off journalists

Lifestyle brand Refinery29 closes UK office and lays off journalists

by Justin Marsh
September 9, 2025
0

US-based digital media company Refinery29 has closed its UK operations and made an unknown number of company-wide cuts. The site’s output includes content on lifestyle, fashion, wellness, relationships and money. Its sub-brand...

Daily Mail names first president for North America to lead revenue growth

Daily Mail names first president for North America to lead revenue growth

by Justin Marsh
September 4, 2025
0

The Daily Mail has created the role of president for North America to lead its revenue growth in the world’s largest media market. The brand has appointed Greta Lawn, who is joining...

Next Post
AXIOM-4: What will be the role of Shubhanshu Shukla in the mission? What did India send in space? What will India benefit?

AXIOM-4: What will be the role of Shubhanshu Shukla in the mission? What did India send in space? What will India benefit?

Popular News

What is Bleach or Facial, Best for Skin?

What is Bleach or Facial, Best for Skin?

October 3, 2025
Animal health ‘fundamental’ to rebuilding trust with rural communities, says Labour MP

Animal health ‘fundamental’ to rebuilding trust with rural communities, says Labour MP

October 2, 2025
Tourists are flocking to a mind-bending Chinese megacity where Google Maps doesn’t work

Tourists are flocking to a mind-bending Chinese megacity where Google Maps doesn’t work

October 1, 2025
Labour branded ‘nasty party’ after minister calls Ed Davey ‘fat bloke in wetsuit’

Labour branded ‘nasty party’ after minister calls Ed Davey ‘fat bloke in wetsuit’

September 30, 2025
Newsletter tips from the New York Times: Visuals, strong host, short, intimate

Newsletter tips from the New York Times: Visuals, strong host, short, intimate

September 29, 2025
Renewing the message: how can Labour fix its broken comms?

Renewing the message: how can Labour fix its broken comms?

September 28, 2025
An ESTA change has just made travelling to America more expensive for Brits

An ESTA change has just made travelling to America more expensive for Brits

September 28, 2025
UK Herald

All Rights Reserved © UK HERALD - The Voice of UK

Important Links

  • Publish Your article
  • Editorial Policy
  • Contact
  • Advertise

...

No Result
View All Result
  • Home
  • Politics
  • UK News
  • Business
  • Science
  • National
  • Entertainment
  • Gaming
  • Sports
  • Fashion
  • Lifestyle
  • Travel
  • Health
  • Food

All Rights Reserved © UK HERALD - The Voice of UK