Should a small business block AI crawlers like GPTBot?

For almost every local business in Scotland the answer is no. Block the crawlers, the software AI companies send out to read websites, and you make it harder for ChatGPT, Gemini, Perplexity and Claude to name you when somebody asks them for a joiner in Dalkeith. The copy on a small business website is worth far less than the recommendation is, and blocking is the one decision here that actively costs you customers.

There is a real case for blocking, and it is covered below. It just does not apply to a plumber, a garage, a salon or a solicitor.

What is the difference between a training bot and a search bot?

This is the distinction that makes the whole decision simpler, and most advice skips it.

A training crawler collects pages to teach future versions of an AI. Your page goes into a huge pile of training material. Nothing links back to you, and nothing happens this quarter.

A search crawler builds the library of pages the assistant answers from today. If it cannot read your page, you are not in the answer.

A user-action fetcher opens your page there and then because a person asked a question that needs it. Blocking that one means an assistant literally cannot open your website on a customer's behalf.

They are separate bots with separate names, so you can allow the second and third and refuse the first. That is the approach generally recommended for small businesses as of September 2026: allow search and user-action crawlers, then decide about training on its own merits (a 2026 guide to AI crawlers).

Which crawlers are we actually talking about?

One point about all of it: robots.txt, the small file on your website that tells these crawlers what they may read, is voluntary. It is a note on the door, not a lock. A well-behaved bot from a company with a reputation to protect obeys it. A scraper (software that copies websites wholesale) that is already ignoring your terms will ignore this too. If something genuinely must not be public, the answer is a login, not a line in a file.

Does blocking Google-Extended remove me from AI Overviews?

No, because AI Overviews, the AI summaries Google shows above its ordinary results, are not built from it, and getting this wrong is the most expensive mistake in the whole topic.

Google-Extended is a name you can use in robots.txt to say whether Google may use your pages to train Gemini, its AI, and to let Gemini look things up. It has nothing to do with where you rank in Google Search. AI Overviews are built from the pages Googlebot, Google's ordinary crawler, reads: the same visits that put you in the normal results.

So blocking Google-Extended does not take you out of AI Overviews. And blocking Googlebot would take you out of Google entirely, which is not a trade any local business should make. People do this by accident, usually by pasting a "block all AI" rule from a forum post written by somebody whose business model is completely unlike theirs.

When is blocking genuinely the right call?

There are real cases, and pretending otherwise would be dishonest.

You sell the words or the pictures themselves. A photographer's portfolio, a writer's archive, a paid course, a research report behind a signup. The content is the product. Training on it is uncompensated use of your stock.

You publish a large collection of original information. Prices, listings, a directory you built by hand over years. Somebody else could profit from copying it.

A client contract or a licence requires it. Then it is not your decision.

Even in those cases, the usual answer is to block the training bots and allow the search ones. You still want to be found. You just do not want your life's work in a training pile you were not asked about.

What is not a good reason: a general unease about AI. That feeling is reasonable and it will not protect anything. The practical effect of blocking on a five-page business website is that fewer customers find you, and the AI gets trained on your competitor instead.

How do you allow or block them in robots.txt?

robots.txt is a plain text file at the top level of your site that crawlers read before anything else. Lines are grouped by the name each crawler goes by (its user agent), and the most specific matching group usually wins, which is why naming a crawler explicitly is worth doing even when a catch-all rule already allows everything.

User-agent: OAI-SearchBot
Allow: /

User-agent: ChatGPT-User
Allow: /

User-agent: GPTBot
Disallow: /

User-agent: CCBot
Disallow: /

That is the middle way: searchable, quotable, not trained on. To allow everything instead, use Allow: / for each named crawler, or simply leave the catch-all rule allowing everything and add nothing.

Two practical warnings. First, a Disallow line with no path does nothing. The syntax is Disallow: / to refuse everything. Second, check the file after any site rebuild. A move to a new website platform quietly rewriting robots.txt to Disallow: / is a genuinely common way for a small business to disappear from search for a fortnight.

What does this site do?

weblothian.co.uk allows all of them. Every crawler in the table above is named explicitly in this site's robots.txt with Allow: /, alongside the catch-all rule, and nothing limits how much of a page an AI service may quote.

That is a deliberate choice rather than a default. This site's content exists to explain what the business does and what it costs. An assistant that reads all of it and answers a stranger's question accurately has done the job the pages were written for. The only parts refused are the behind-the-scenes addresses the site's own forms talk to, which hold nothing useful to quote, and the admin panel, which shows "page not found" to everyone anyway.

The commercial case is getting easier to make. Visitors sent to US retail sites by AI assistants rose 393% year on year in the first quarter of 2026, and they went on to buy around 42% more often than visitors from ordinary search (Limy's 2026 guide). Fewer visitors, better qualified, because the assistant has already answered the browsing questions and sends the person who is ready to buy.

So what should you actually do this week?

Open yourdomain.co.uk/robots.txt in a browser. Most owners have never looked at it.

If it says Disallow: / under any crawler name you recognise from the table, find out who put it there and why. If the file does not exist at all, that is fine, because no file means everything is allowed, though naming the bots explicitly is tidier and clearer to anyone who reads it later.

Then leave it alone and go and answer your reviews, which will do far more for whether an assistant recommends you than any line in that file. If you would rather somebody checked yours and explained what it currently says, that is a ten-minute job inside the consultation (opens in a new tab). Keeping that file, and the rest of the site, right over time is part of management and maintenance. If the worry underneath the question is security rather than crawlers, Cyber Essentials in 2026 is the more useful read.