Permission

@adactio

What about Bing? I’ve found Google Bard to be largely useless and irrelevant. I’ve found Bing’s approach of being able to “search the web” (Microsoft’s technology - Prometheus - can call on the Bing search index) to be the most useful of all AI chat tools, specifically because it provides links to sources. You can also provide it with links/sites to work with; e.g. “based on information at https://adactio.com/ …”.

I find it very hard to trust LLMs that do not also cite sources.

Adactio: Jeremy Keith

@adactio An excellent suggestion.

We can rescind our invitation to Google.

– Jeremy Keith

I’ve been thinking about this quite a bit lately and I’m happy to see I’m not the only one. It’s hard to imagine Google’s value add for me and my website.

Bonus: fewer requests a day to my server—by thousands.

Published Tuesday, Jul 11 at 13:58

ADDED 4 October 2023:

Google has announced a new token you can block to exclude your website from training Bard and Vertex AI: Google-Extended. To block your site from being used to train Google’s AI products, you should include this code in your robots.txt file:

# Google AI User-agent: Google-Extended Disallow: /

As a standalone token, that means that we don’t need to block Google from indexing our websites to block them from using our content to train their AI products.

⭐ ADDED 11 December 2023:

Except!!!! Google-Extended applies to their products but not their generative search results. So if you don’t want your content to appear in generative search results, you still need to block Googlebot.

ORIGINAL ARTICLE (published 11 July 2023):

After thinking about it for a couple days, I’ve decided to de-index my website from Google. It’s reversible — I’m sure Google will happily reindex it if I let them — so I’m just going ahead and doing it for now. I’m not down with Google swallowing everything posted on the internet to train their generative AI models. I was pushed over the edge by posts from Jeremy Keith and Vasilis van Gemert, thanks y’all.

I don’t have Google Search Console set up for this website so I don’t know how much search traffic I get. My other blog, Cascadia Inspired, got about 200 hits in the past three months. I’m not going to cry over that — they’re mostly going to one 2015 article anyway (and probably not that helpful of a post, to my eye. Around New Year’s every year I usually get an influx of people to my ten-year-old guide to doing a creative annual review. Sorry folks, I’m sure someone else has written something better by now.) 😉

I’m going to start by pulling my websites out of Google search, then work on adding my sites to directories. Maybe I’ll even join a webring 💍✨

Adding a noindex meta tag to my WordPress header

Because my website has already been indexed by Google, I need to allow the Google bot to re-crawl the pages and see the new “noindex” instruction. So in the future I’ll also block the Googlebot crawler, but not just yet 😉

I added this code to the functions.php file of my child theme:

add_action( 'wp_head', function() { global $page; echo '<meta name="Googlebot" content="noindex, nofollow, noimageindex">'; });

I figured out how to adapt this from WPExplorer. This random wordpress plugin help forum suggested another version, I don’t know which is better 🤷‍♀️

I’m not 100% on whether the noimageindex is actually helpful for Googlebot since that’s their text bot, but can’t hurt right? (Tell me if it hurts lol.) Yoast says there’s a better way to block image indexing but I’m scared of touching the .htaccess file and definitely nothing with my server 😂 (I’m on shared hosting anyway, so I think the edits I can make are limited?)

Blocking bots that collect training data for AIs (and more)

In addition, I created a robots.txt file to tell “law abiding” bots what they’re not allowed to look at. I ought to have done this before but kind of assumed it came with my WordPress install 😅 (Nope.)

AI user agents to block

There’s so many now, just copy from my robots file tbh.

ADDED 4 October 23: To block training of Google’s Bard, I blocked Google-Extended.

I specifically want to deter my website being used for training LLMs, so I blocked Common Crawl.

To block OpenAI, I blocked both user agents ChatGPT-User and GPTBot. (Added GPTBot 10 August 23)

ADDED 4 October 23: Per Neil Clarke’s article, I have also blocked Omgilibot, Omgili, and FacebookBot. (Via Jeremy Keith)

ADDED 14 February 2024: I also blocked user agents used in AI training sets: anthropic-ai, Bytespider, FacebookBot, and PerplexityBot (source)

ADDED 16 April 2024: prompted by Ethan Marcotte, I blocked several more known and suspected user agents used in AI training: Claude-Web, ClaudeBot, cohere-ai, Diffbot, YouBot, ChatGPT

Added 17 June 2024: I’ve now blocked Apple’s AI training bot Applebot-Extended (thanks for the heads-up James!) Does anyone else feel like this is getting ridiculous?

I also blocked Amazonbot and applebot to block Siri and Alexa’s “smart answers.” I believe this also excludes me from Apple search.

I’ve also now blocked Googlebot and bingbot in protest of their generative AI search results — I’ve had the code up for my pages to be deindexed by Google for over six months and I’m done waiting anymore.

Dark Visitors apparently has a WordPress plugin to update your robots.txt whenever a new agent comes out, but for now I’m stickin’ with manual. I am also still wary of modifying my .htaccess file and breaking something, so it’s just my robots.txt making my stance clear — I can’t control whether companies have any sort of ethics and comply, unfortunately.

Other user agents

Searching on DuckDuckGo, I found an older article from a theme maker with specific advice for WordPress robots.txt. From there I jumped to Jeff Star’s recommendations from 2020.

I also appreciate fellow opinionated individuals on the internet so I followed some other blocks from Rohan Kumar. I would happily take more opinionated suggestions of junk bots to block if anyone else has opinions or can point me to a list somewhere 😉

Note: this article generated a lot of interest! See a Hacker News discussion.

Syndicated to IndieWeb News

@adactio thanks for writing about this and the pointers to the conversation that has already been happening.

I wonder if formulating this as a well-known URI might be easier to layer in than further encumbering robots.txt? @mnot has there been talk of this already?

Also re: buy in from Google, there does seem to be some interest from them? https://blog.google/technology/ai/ai-web-publisher-controls-sign-up/

A principled approach to evolving choice and control for web content

@anniegreens Oh. What an idea. Like, it’s a strike. Google blackout.

@anniegreens That’s VERY true. Google has indeed broken the social contract.

If I stopped Google’s bots (or any search engine’s bots for that matter) from crawling my site, I would notice zero change. I’ve never been found or benefited from search engines.

Every business contact has been made through recommendation, social media presence or by my own ‘reaching out’.

@sarajw @anniegreens People find my site by me saying “hey, look… here’s my site, let’s talk!” and then adding the URL lol.

@simoncox Sometimes.

@simoncox just products of a broken system.

@johnmu

😢

@simoncox That’s a very nuanced question. I would say we probably inadvertently shot ourselves in the foot. Even those of us who have held firmly to non-search promotion strategies still rely on search engines for exposure.

@michael_martinez Yep and we have helped build this ecosystem.

@simoncox Dont hate the player, hate the game.

Also, if we didn’t help decent brands improve their sites then just imagine the shite that Google would be ranking top

@Jamesboswell Exactly - we do a great job under difficult circumstances, most of us, but we may get painted with the tar brush.

@adactio Same question to @chriscoyier. Does the inclusion of links in an answer, as Bing AI does, restore the social contract?

I find this conundrum really interesting; I’m ambivalent and conflicted on it.

What’s happening with LLMs seems to be a version of the original vision of the Semantic Web, as envisioned by Tim Berners-Lee.

https://en.wikipedia.org/wiki/Semantic_Web

And are we ourselves not spitters of word smoothies based on our own unique experience of the world?

Semantic Web - Wikipedia

@adactio @chriscoyier

Perhaps Sir Tim’s Solid technology is the answer?

https://techcrunch.com/2020/11/08/tim-berners-lees-startup-inrupt-releases-solid-privacy-platform-for-enterprises/

TechCrunch is part of the Yahoo family of brands

@charlesroper @adactio @chriscoyier It’s one thing we looked at - data pods as way to allow people to store personalisation, so we don’t have to - then along with our Knowledge Graph and LLM provide real recommendations, not ones based on weak signals. Didn’t get to finish it in this hackathon PoC, but it would be something we might add later - also allow “bring your identity” and sharing.

@tanepiper @adactio @chriscoyier Wow - that sounds fascinating and pretty radical. Have you written about it at all? Would love to learn more.

@charlesroper @adactio @chriscoyier The video of the demo should be up soon from OntoCommons. Haven’t really touched upon it enough yet to write about it, but it’s only part of the puzzle of being non-creepy with recommendations.

@charlesroper @adactio I think it goes a long way. I think it’s telling that sources weren’t included right away, when it’s possible to do so.

See https://www.phind.com/ and type in a coding question, and see below how it credits where it got the information.

Phind: AI search engine

@chriscoyier @adactio I’d not seen Phind before - interesting. Quite similar to Bing AI.

What I found working in biodiversity informatics for over a decade is that people are generally much more comfortable sharing their data - often a lifetime of dedicated, expert work - if a) they’re credited and b) it’s used for good causes and not for profit.

It was one of the more challenging aspects of the work. Remarkable parallels with LLMs consuming data scraped from the web….

@chriscoyier @adactio Credit and the capability to withdraw permission are very important.

@adactio DeviantArt proposed (and obeys) “noai” and “noimageai” meta tags: https://www.aimeecozza.com/noai-noimageai-meta-tag-how-to-install/

Would be nice to see other crawlers follow suit.

What’s DeviantArt’s new “noai” meta tag and how to install it

(As usual, this roundup is a mix of French and English. Comme d’habitude, ce récap est un mélange d’anglais et de français.)

Toggle

• Quelques nouvelles perso
• Reading
  • Books
  • Articles académiques
  • Articles non-académiques
• Watching
  • Movies
  • TV Shows
  • Video essays
  • Other
• Listening
  • Podcasts
  • Music
• Going
  • Publications similaires :

Quelques nouvelles perso

Je commence à être vraiment confortablement installé à Grenoble. Le fait d’avoir mes potes à 10min de vélo, c’est vraiment extraordinaire, et quelque chose que j’avais oublié à Paris : pas besoin de s’organiser plusieurs jours à l’avance, un « je vais au insérer bar dans une heure si tu veux » ça couvre les bases et ça facilite tellement les interactions !

J’ai aussi bien profité de mon mois de juillet pour lire et me balader en France, comme vous pourrez le constater dans ce récap.

Reading Books

In English:

• We Deserve Monuments, by Jas Hammonds, had me sobbing during most of the second half of the book.
• The graphic novel Shubeik Lubeik by Deena Mohamed was beautiful and moving, and I highly recommend it. It was also nice to read something in translation from Arabic – I haven’t done much of that yet and the writing style is really special!
• All my Rage by Sabaa Tahir follows two Pakistani teenagers in an US high school, with their own issues and trauma and, well, all their rage.
• Spinning by Tillie Walden, which was recommended by the hosts of the (French) RomComment podcast, is Walden’s recollections of being a tween mid-level figure skater who moves to Texas, trains every morning at 4am, finds out she’s gay, and is understandably a bit overwhelmed.
• In Wild and Crooked, by Leah Thomas, two kids become unlikely friends. There’s the criminal-in-training daughter of a guy who murdered someone, and a disabled kid who finds solace in tabletop role playing. Oh, and whose dad got murdered by the first one’s. When that comes to light, oddly enough, it doesn’t go down well with the town and their respective families.
• I read the Scholomance trilogy pretty much in one sitting (aside from this pesky « I need sleep » little thing).
• Finally, a bit more nonfiction: I love Daniel M. Lavery’s blog The Chatner, so I couldn’t miss the essay collection Something that may shock and discredit you, about being trans, or being religious, or maybe Greek mythology or reality TV, I’m not quite sure.

En français :

• Abobo Marley de Yaya Diomandé parle d’un jeune homme ivoirien qui rêve de venir en Europe et illustre à merveille l’adage «un tiens vaut mieux que deux tu l’auras» en détruisant méthodiquement chacune de ses chances de réussir au pays.
• Grenoble Calling: une histoire orale du punk dans une ville de province, de Nicolas Bonanni et Margaux Capelier, était une lecture très appropriée pour cet emménagement et super intéressante. J’en ai résumé quelques points pertinents sur la nouvelle page Wikipédia Punk à Grenoble mais je recommande vraiment le bouquin, et en particulier le CD qui va avec.
• Je suis une fille sans histoire, d’Alice Zeniter, est un petit message d’amour à la rédaction d’histoires, aux femmes, et à la langue française. Il est drôle, il est émouvant, il ne révolutionne rien mais rappelle des messages importants, je l’ai beaucoup aimé. Ça tombe d’ailleurs bien, puisque j’ai toujours L’art de perdre quelque part dans mes cartons (la bibliothèque n’a toujours pas été livrée, non).

Articles académiques

• « L’écriture inclusive, je ne connais pas très bien… mais je déteste ! »

Articles non-académiques

• ‘I no longer know how to think about borsch’
• Why fake kidnapping stories are everywhere
• Giving Google permission to index and use our content

Watching Movies

In chronological order:

• But I’m a cheerleader was very cringe and perfect for a too-hot afternoon where I didn’t know what to do. It’s not a good movie. I didn’t want a good movie.
• Nimona, however, absolutely was a good movie, and you should watch it.
• Birds of Prey was fun, and sometimes fun is all I want.
• Barbie was also fun, but this time I have to be honest – fun was not all I wanted. I was happy to see something pink and happy at the theater for once and that made it all worth it, but honestly, not a masterpiece…
• unlike Oppenheimer, which solidifies my transition into a very manly man, I guess. That one was absolutely incredible.

TV Shows

My partner told me about Severance, which had vaguely been on my radar since it came out. I took it as a sign to finally watch it and it was GREAT. Amazing. I need season two now.

I also watched Good Omens season 2. It was okay – I have to be honest here, I liked season 1 but I just liked it. Season 2 was the same: it was nice and I’m immediately going to forget about it.

Video essays

• The weird rise of fake podcasts
• Gen Z’s Obsession with Youth & Beauty is VERY Disturbing
• The Playlistification of Music

Other

My friend Sébastien made yet another amazing art project and you should watch his wonderful video, which includes the demo and how he made it.

Listening Podcasts

In English:

• « Liberal Fascism » by If books could kill
• « Children of men, the only modern movie » by Kino Lefter
• « The golden age of internet piracy » by Srsly Wrong
• « Bonnie and Clyde (and Blanche and Buck) » with Jamie Loftus by You’re Wrong About
• « Chat control? The Stasi look like amateurs compared to this shite » by Vulnerable by Design
• Hal Schrieve on Gender Reveal (including lots of talks about trans literature and what «t4t» and «chaser» really mean)
• « The country of the blind » by 99% invisible

En français :

• « Peter Norman, justice pour le troisième homme de Mexico » by Les grands récits
• « Enfances et colonisation : être enfant en situation coloniale, ça veut dire quoi ? » par Des Colonisations
• « Que reste-t-il de la NUPES après le combat des retraites ? » par Penser les luttes
• « Histoire de l’anarchisme » par Paroles d’histoire (table ronde)
• Laurent Chalumeau chez Bookmakers, longue interview émaillée d’anecdotes qui m’a donné envie de lire ses romans (vous me direz, c’est le but du podcast)

Music

• Found CDs at the yearly library sale, which I haven’t listened to yet. I’m really, really glad to have found Joey Bada$$’s eponymous (?) album, which I used to listen to on repeat.
• Grenoble Calling, le CD de punk dans le livre mentionné plus haut.

Going

I spent a long weekend in Narbonne (south-west France) for a Wikimedia weekend, which was really sweet. Narbonne looked wonderful and has loads of adorable little corners.

I also spent one day in Lyon on a whim and rediscovered some of my favourite places there.

Publications similaires :

1. Gender Reveal épisode 116 : River Butcher sur la transmasculinité
2. L’obsession fasciste pour l’Antiquité gréco-romaine
3. Comprendre et éviter l’épuisement militant
4. La conquête du pain, Pierre Kropotkine

I’m trying a new format, which is sharing 3-5 links (with context) on one given topic. I’m starting with what I believe to be the easiest one, since I already have a giant collection about it: moving towards a homemade web.

This is inspired by Tracy Durnell’s « article pairings ».

Toggle

• The problem
• The wall of text
• The solution
• The radical

The problem

The Internet isn’t meant to be so small is a good explanation of the current Internet situation.

the internet wasn’t supposed to be like this. It wasn’t supposed to be six boring men with too much money creating spaces that no one likes but everyone is forced to use because those men have driven every other form of online existence into the ground.

The wall of text

Rediscovering the small web is full of hope and goes into detail about why we got here, what can be done, what can’t, and what else to think about.

The solution

The Handcrafted Artisanal Web

Nadreck shares a few personal rules, including posting at least once a week on your website, visiting other people’s websites, and promoting other people’s work on your own website and on your social media if you use social media.

The radical

In his blog post Permission, Jeremy encourages us to completely leave Google. After all, the service is getting worse and worse and we don’t benefit from them anymore, especially with the new Gemini updates; why should we keep giving them value by donating our creations to their index?