Should You Block AI Bots from Crawling Your Website?

I’ve been thinking a lot about Squarespace’s new feature that lets you block AI bots from your site (Yoast recently added this as well). On the surface, it seems like a good idea to protect your content, right? But let’s pause and look at the bigger picture, especially through the lens of AI’s growing role in connecting people with online content.

History Lesson

Rewind to 15 years ago. Some website owners decided to block Google from indexing their sites, fearing content theft. This choice, though well-intentioned, made them virtually invisible to a vast audience. Today, we face a similar crossroads with AI bots like ChatGPT.

Here’s why this matters: AI, especially tools like ChatGPT, are becoming key players in how people find and interact with websites. They’re not just scanning the web; they’re actively assisting users in discovering and engaging with content in new and meaningful ways.

The first Wedding Booked via ChatGPT?

I have a personal story that really brings this home. Recently, I landed a wedding photography gig thanks to AI.

A couple planning their wedding used ChatGPT to find experienced wedding photographers in Portland, Oregon.

Guess what? My name and website popped up in their search. This would never have happened if I had blocked AI bots from my site.

Due to my efforts to make my website appear in ChatGPT results, I helped my brand get found. It would have been significantly less likely to happen, if I had blocked my site from being crawled by ChatGPT, Google Bard, and other similar AI bots.

Rethink the popular narrative and gut reaction

Blocking AI bots on your website today is a lot like those who chose to block Google in the past. It’s turning away from a rapidly growing way that people discover and interact with online content. The digital world is evolving, and AI is a significant part of this transformation. We might not like it, we might not agree with it, but we’re not going to be able to resist the progress.

My recommendation – Do not block AI bots from Crawling Your Websites

So, before you decide to use that AI-blocking feature, remember that doing so might keep you out of the reach of potential opportunities. The key to thriving online is staying visible and adaptable in our constantly evolving digital landscape.