Decide what you want your public website to do before switching AI bots on or off. A business that wants customers to discover its service pages may want search access, while making a separate choice about model training. Confidential information belongs behind appropriate access controls, whichever crawler policy you choose.
“Allow AI” and “block AI” are too broad to describe that decision. Different platforms use different crawlers and controls. Before paying someone to change them, ask which use they are changing, which pages it affects and what evidence will confirm the result.
Put the business decision first: what information should be public, where should customers find it, and which other uses do you want to permit? Then ask your website provider to map that decision to the actual controls.
Start With the Value of the Public Page
For a local trade or professional service, a useful public page explains the job, the area covered and how to enquire. Making that information available to relevant search systems can support discovery. It does not mean an AI platform will recommend the business or send a customer.
A publisher, paid research service or business selling access to a valuable database has a different decision to make. Its commercial model may depend on visits, subscriptions or licensed access. Do not assume that an access policy chosen for a plumber is suitable for that business.
Separate the information you want customers to find from material they should only receive privately. Customer documents, private quotes, internal records and account pages should not become public to satisfy an AI visibility proposal.
Make a short list of the important public pages and their purpose. That gives an agency something specific to assess, rather than permission to open the whole website.
Separate Search, Training and User Requests
A search crawler collects information that can help a platform return relevant results. A training crawler collects material that may be used to develop a model. A user-triggered request is another activity: an assistant may visit a page while responding to a person.
Those descriptions help frame the question, but they are not universal platform settings. Check the operator's current documentation rather than sorting every bot by its name alone.
OpenAI documents independent controls for OAI-SearchBot, used for ChatGPT search, and GPTBot, used for potential model-training material. A website can allow the search crawler while disallowing the training crawler. It also distinguishes ChatGPT-User, used for certain user-initiated actions; robots.txt rules may not apply to those requests. See OpenAI's crawler documentation.
This distinction matters when an agency says you must allow training to appear in ChatGPT search. Ask it to identify the actual requirement and supporting documentation. Search access and training preferences are separate decisions in OpenAI's documented controls.
For the wider content and visibility work, our ChatGPT business guide explains how to establish a baseline and inspect the sources in answers. Here, the job is to agree the access policy before changing it.
Treat Google's Controls Separately
Google's naming can be confusing. Google-Extended is a robots.txt product token controlling specified training and grounding uses in Gemini products. It is not a separate HTTP crawler user agent. Google says it does not affect inclusion or ranking in Google Search. See Google's crawler documentation.
Google's AI Overviews and AI Mode sit within Search. A page must be indexed and eligible for a Search snippet to qualify as a supporting link; meeting those conditions does not guarantee inclusion. Google also identifies crawling access through robots.txt and hosting or CDN infrastructure as a relevant foundation. See Google's AI features guidance.
Do not treat a Google-Extended preference as the setting that decides whether your page appears in an AI Overview. Equally, do not assume that allowing it guarantees a Gemini recommendation. Ask which product a proposed setting applies to and what trade-off you are accepting.
Our Google AI search guide covers the wider eligibility and content work. An access decision should fit that work without becoming a promise of placement.
Choose a Policy That Fits the Business
Use these examples to frame a discussion with your website provider. They are starting points for a decision, not universal settings or legal conclusions.
| Business Situation | Decision to Make | What to Ask the Provider |
|---|---|---|
| A service business wants its public service pages discovered. | Which search access supports that goal, and what training preference does the owner want? | Name the platform, crawler, affected pages and separate controls. |
| A business sells subscriptions or access to original research. | What stays public, what requires a subscription, and which automated uses fit the business model? | Assess those boundaries before proposing an allow or block rule. |
| A website holds customer or internal documents. | Who should be authorised to access each document? | Confirm the access controls; do not use a crawler preference as the privacy mechanism. |
A mixed business may need more than one answer. Its public service pages can have a different purpose from its customer portal. Keep the scope attached to the pages and uses involved.
Ask for Evidence Before Approving a Change
Ask the agency or website provider to answer five questions in plain English:
- What is the current situation? Which page and documented crawler are being checked, and what rule or response was found?
- What decision needs my approval? Is this about search access, training use, user requests or access to private information?
- What will change? Identify the affected rule, page scope and person responsible for implementation. Include the way to restore the previous configuration.
- What else could it affect? Consider ordinary search, other automated services and the site's security controls before changing a broad rule.
- How will you verify it? Agree on the evidence that will show whether the intended access changed and whether the important public pages still work.
A page opening in your browser does not demonstrate that a particular crawler can fetch it. A request made with a crawler's name is also not, by itself, proof that the real operator made that request. Ask the technical person to distinguish their test from verified crawler activity and to record any remaining uncertainty.
Avoid approving “we'll unblock all bots” as a substitute for a defined task. The provider should explain the relevant restriction and its proposed change. Settings in a host, security service or plugin may need review alongside robots.txt; the exact checks depend on your setup.
A Worked Example: Public Services, Private Quotes
Consider an engineering consultancy with public service pages and a customer area containing project quotes. The owner wants prospective customers to understand its services in AI search, but prefers not to allow model-training use of its public material.
A useful brief names the service pages, records that training preference and keeps customer documents private. The website provider then checks which documented controls can express those choices for the selected platforms. OpenAI's separate search and training controls provide one concrete distinction to examine. Google's controls need their own assessment.
If an access check finds no relevant restriction, the provider should say so. The next useful work may be clearer service information or source accuracy, rather than a configuration change. If a restriction is found, record the affected pages, approved change and follow-up result.
This is a planning example. It does not establish that this business has appeared in an answer, that a bot has visited it or that a crawler rule can protect its private documents.
Review Access and Outcomes as Different Results
Keep the owner's decision, the implemented configuration and the verification record together. Revisit them when platforms change their guidance or when you change hosting, security tools or the purpose of the website.
Then assess search visibility separately. Record the platform, customer question, date and sources returned. A successful access check is useful technical evidence; a mention, citation, referral visit and qualified enquiry are further results to measure. None follows automatically from allowing a crawler.
Australian Web Experts' AI SEO service includes reviewing important pages, relevant technical settings and business information within an agreed scope. If an access policy is holding up your next decision, get in touch with your website, the pages you want customers to find and the uses you want to discuss.