Pulling my site from Google over AI training

ADDED 4 October 2023:

Google has announced a new token you can block to exclude your website from training Bard and Vertex AI: Google-Extended. To block your site from being used to train Google’s AI products, you should include this code in your robots.txt file:

# Google AI
User-agent: Google-Extended
Disallow: /

As a standalone token, that means that we don’t need to block Google from indexing our websites to block them from using our content to train their AI products.

⭐ ADDED 11 December 2023:

Except!!!! Google-Extended applies to their products but not their generative search results. So if you don’t want your content to appear in generative search results, you still need to block Googlebot.

ORIGINAL ARTICLE (published 11 July 2023):

After thinking about it for a couple days, I’ve decided to de-index my website from Google. It’s reversible — I’m sure Google will happily reindex it if I let them — so I’m just going ahead and doing it for now. I’m not down with Google swallowing everything posted on the internet to train their generative AI models. I was pushed over the edge by posts from Jeremy Keith and Vasilis van Gemert, thanks y’all.

I don’t have Google Search Console set up for this website so I don’t know how much search traffic I get. My other blog, Cascadia Inspired, got about 200 hits in the past three months. I’m not going to cry over that — they’re mostly going to one 2015 article anyway (and probably not that helpful of a post, to my eye. Around New Year’s every year I usually get an influx of people to my ten-year-old guide to doing a creative annual review. Sorry folks, I’m sure someone else has written something better by now.) 😉

I’m going to start by pulling my websites out of Google search, then work on adding my sites to directories. Maybe I’ll even join a webring 💍✨

Adding a noindex meta tag to my WordPress header

Because my website has already been indexed by Google, I need to allow the Google bot to re-crawl the pages and see the new “noindex” instruction. So in the future I’ll also block the Googlebot crawler, but not just yet 😉

I added this code to the functions.php file of my child theme:

add_action( 'wp_head', function() {
global $page;
echo '<meta name="Googlebot" content="noindex, nofollow, noimageindex">';
});

I figured out how to adapt this from WPExplorer. This random wordpress plugin help forum suggested another version, I don’t know which is better 🤷‍♀️

I’m not 100% on whether the noimageindex is actually helpful for Googlebot since that’s their text bot, but can’t hurt right? (Tell me if it hurts lol.) Yoast says there’s a better way to block image indexing but I’m scared of touching the .htaccess file and definitely nothing with my server 😂 (I’m on shared hosting anyway, so I think the edits I can make are limited?)

Blocking bots that collect training data for AIs (and more)

In addition, I created a robots.txt file to tell “law abiding” bots what they’re not allowed to look at. I ought to have done this before but kind of assumed it came with my WordPress install 😅 (Nope.)

AI user agents to block

There’s so many now, just copy from my robots file tbh.

ADDED 4 October 23: To block training of Google’s Bard, I blocked Google-Extended.

I specifically want to deter my website being used for training LLMs, so I blocked Common Crawl.

To block OpenAI, I blocked both user agents ChatGPT-User and GPTBot. (Added GPTBot 10 August 23)

ADDED 4 October 23: Per Neil Clarke’s article, I have also blocked Omgilibot, Omgili, and FacebookBot. (Via Jeremy Keith)

ADDED 14 February 2024: I also blocked user agents used in AI training sets: anthropic-ai, Bytespider, FacebookBot, and PerplexityBot (source)

ADDED 16 April 2024: prompted by Ethan Marcotte, I blocked several more known and suspected user agents used in AI training: Claude-Web, ClaudeBot, cohere-ai, Diffbot, YouBot, ChatGPT

Added 17 June 2024: I’ve now blocked Apple’s AI training bot Applebot-Extended (thanks for the heads-up James!) Does anyone else feel like this is getting ridiculous?

I also blocked Amazonbot and applebot to block Siri and Alexa’s “smart answers.” I believe this also excludes me from Apple search.

I’ve also now blocked Googlebot and bingbot in protest of their generative AI search results — I’ve had the code up for my pages to be deindexed by Google for over six months and I’m done waiting anymore.

Dark Visitors apparently has a WordPress plugin to update your robots.txt whenever a new agent comes out, but for now I’m stickin’ with manual. I am also still wary of modifying my .htaccess file and breaking something, so it’s just my robots.txt making my stance clear — I can’t control whether companies have any sort of ethics and comply, unfortunately.

Other user agents

Searching on DuckDuckGo, I found an older article from a theme maker with specific advice for WordPress robots.txt. From there I jumped to Jeff Star’s recommendations from 2020.

I also appreciate fellow opinionated individuals on the internet so I followed some other blocks from Rohan Kumar. I would happily take more opinionated suggestions of junk bots to block if anyone else has opinions or can point me to a list somewhere 😉

Note: this article generated a lot of interest! See a Hacker News discussion.

Syndicated to IndieWeb News