Should You Block AI Bots from Crawling Your Website?

I’ve been thinking a lot about Squarespace’s new feature that lets you block AI bots from your site (Yoast recently added this as well). On the surface, it seems like a good idea to protect your content, right? But let’s pause and look at the bigger picture, especially through the lens of AI’s growing role in connecting people with online content.

The Photo Stealers Facebook Page had a post that saw a significant number of shares and reactions.

History Lesson

Rewind to 15 years ago. Some website owners decided to block Google from indexing their sites, fearing content theft. This choice, though well-intentioned, made them virtually invisible to a vast audience. Today, we face a similar crossroads with AI bots like ChatGPT.

Here’s why this matters: AI, especially tools like ChatGPT, are becoming key players in how people find and interact with websites. They’re not just scanning the web; they’re actively assisting users in discovering and engaging with content in new and meaningful ways.

The first Wedding Booked via ChatGPT?

I have a personal story that really brings this home. Recently, I landed a wedding photography gig thanks to AI.

A couple planning their wedding used ChatGPT to find experienced wedding photographers in Portland, Oregon.

Guess what? My name and website popped up in their search. This would never have happened if I had blocked AI bots from my site.

Due to my efforts to make my website appear in ChatGPT results, I helped my brand get found. It would have been significantly less likely to happen, if I had blocked my site from being crawled by ChatGPT, Google Bard, and other similar AI bots.

Rethink the popular narrative and gut reaction

Blocking AI bots on your website today is a lot like those who chose to block Google in the past. It’s turning away from a rapidly growing way that people discover and interact with online content. The digital world is evolving, and AI is a significant part of this transformation. We might not like it, we might not agree with it, but we’re not going to be able to resist the progress.

My recommendation – Do not block AI bots from Crawling Your Websites

So, before you decide to use that AI-blocking feature, remember that doing so might keep you out of the reach of potential opportunities. The key to thriving online is staying visible and adaptable in our constantly evolving digital landscape.

Similar Posts

Leave a Reply

Your email address will not be published. Required fields are marked *

3 Comments

  1. Re: Blocking AI from your website. From your experience AI was a positive = A new client. The fear of AI is now in full bloom. Frankly, I have not made up my mind yet one way or the other. Let’s all just see how this plays out. That’s what our small family wedding business will do.

    Best Wishes

  2. Thank you for this information. My initial thought was to not allow it, however, since the integration of AI into search it seemed beneficial. I’ve found myself using GPT to search as well. As a photographer I’m all for the traffic. Cool that you got booked from ChatGPT.

  3. You may be right if you’re selling a service like wedding photography, but as a writer I tend to agree with Morten Rand-Hendriksen:

    »AI companies are not search engine companies. Search engines provide links to your content. That can be a net benefit to you. AI companies train their models on your content. Then they replicate your content. That can be a net loss to you. «
    https://front-end.social/@mor10/112021863794913553