How ChatGPT search works and why Bing matters more than it seems
ChatGPT searches the web with its own crawler plus search providers, among which OpenAI names Microsoft. How it works, what your site needs to show up, and why Bing deserves an afternoon of your time.
By Javier Tenjo, Full stack developer
On this page
When people talk about rankings, nearly everyone thinks of Google. That is reasonable: it is where most searches happen. But when someone asks ChatGPT a question and it looks things up on the web, the route is different. OpenAI uses its own crawler and external search providers, and the one it names in its documentation is Microsoft, in other words the search engine many sites have not looked at in years: Bing.
This does not mean changing priorities. It means there is an afternoon of work, nearly always pending, that pays off more today than it used to.
How does ChatGPT search the web?
ChatGPT decides on its own when a question needs current information, and you can also ask it to search. From there, as OpenAI explains in its Help Center, the process goes like this:
- It rewrites the question. It does not search for the sentence as you typed it. It turns it into one or more targeted queries, much like the ones an experienced searcher would type.
- It queries its search providers. It sends those queries, reviews the top results and, if needed, sends more specific ones.
- It reads and writes. From the passages it found it composes the answer and adds citations linking to the source pages.
Two practical consequences follow. A user’s long question ends up as several short searches, so familiar keywords still count. And your page is not competing for a position, but to be one of the few results the assistant opens and cites.
What does Bing have to do with it?
OpenAI does not have a complete search engine of its own. It combines two things:
- Its own crawler,
OAI-SearchBot, which crawls sites so they can be shown in search answers. - External search providers. In its documentation, OpenAI points to the privacy policies of two of them: Microsoft and Shopify. Microsoft means Bing; Shopify supplies product results.
OpenAI does not publish how much of each answer comes from its crawler and how much from Bing, and that share may change. What can be said is that Bing is one of the ways in, and a site Bing does not know has one way fewer.
Bing is more than bing.com
Even without ChatGPT, Bing’s index reaches further than its market share suggests:
- Copilot, Microsoft’s assistant built into Windows, Edge and Office, answers with Bing results.
- Other search engines such as DuckDuckGo and Yahoo rely largely on Bing’s results.
- Windows and Edge ship with it as the default, and many people in corporate environments never change it.
Added up, that is an audience few sites measure and for which hardly anyone has set anything up.
What does your site need to show up?
1. Let OAI-SearchBot in
This is the condition OpenAI sets: for a site to be eligible for search answers, it has to allow the OAI-SearchBot crawler in its robots.txt.
User-agent: OAI-SearchBot
Allow: /
Three details from the official documentation:
- It is independent of
GPTBot, the crawler that collects content for model training. You can block training and allow search. We explain it in robots.txt and AI crawlers. - A
robots.txtchange takes around 24 hours to be reflected in ChatGPT search. robots.txtalone is not enough. OpenAI also asks that your host or CDN does not block its crawler’s IP addresses. Some providers and firewalls hold back AI bots by default, so that setting is worth checking.
You can check what your site allows today with the robots.txt tester.
2. Make sure you are in Bing
There is a way to find out in ten seconds, with no account or tools. Go to bing.com and search for your domain with the site: operator in front:
site:yourdomain.com
Bing answers with only the pages from that site it has in its index. What to look for:
- If there are no results, Bing does not know your site, or has it blocked.
- If far fewer pages show up than you have, there are parts of the site it is not finding.
- If a specific page is missing, you can check it with its full address:
site:yourdomain.com/services/.
The result count it shows is approximate, so treat it as a first signal rather than an inventory. The exact figure is in the dashboard covered next.
Bing Webmaster Tools is Bing’s equivalent of Search Console, and it is free. Setup is short:
- Go to bing.com/webmasters with a Microsoft account.
- Add your site. If it is already verified in Google Search Console, you can import it from there and skip verification.
- Submit your sitemap. If you are unsure what it should contain, see what an XML sitemap is.
- Use URL inspection to confirm your main pages are indexed, and fix whatever the dashboard flags as an error.
Surprises are common: sites that rank well on Google with half their pages missing from Bing, because nobody ever sent it a sitemap.
3. Announce changes with IndexNow
IndexNow is a protocol for notifying search engines the moment a page is created, changed or removed, without waiting for the crawler to come back. Bing is one of its creators, and Microsoft explicitly recommends it to keep information fresh “across search and AI experiences”.
For an assistant that answers with what it finds today, having your new price or retired service known in minutes rather than weeks makes a difference. The details are in what IndexNow is.
4. If you have a physical location, complete Bing Places
This is Bing’s business listing, the equivalent of Google’s. Microsoft recommends keeping address, hours and contact details current there, and it also lets you import the listing you already have on Google.
5. Write so you can be cited
Microsoft’s recommendations for appearing in AI answers match long-standing good practice:
- Clear headings, tables and FAQ sections, so information can be extracted accurately.
- Depth on the topics you know well, rather than many shallow pages.
- Claims backed with examples, data and sources.
- Content kept up to date.
How do you know it is working?
In Bing Webmaster Tools. Alongside the classic search reports, it includes a report called AI Performance, which shows how often your site was cited in Copilot answers and Bing’s AI summaries, which pages were cited and for which queries. It does not cover ChatGPT, but it is one of the few official sources of data on assistant citations.
In your analytics. Visits arriving from ChatGPT show up with chatgpt.com as the source, and those from Copilot and Bing with their own. A segment to track them month by month is worth creating.
By asking. In ChatGPT, with search turned on, ask the questions a customer would ask and look at which sources it cites. If competitors appear and you do not, look at which of their pages it chose: they tend to be the ones that answer that question directly.
What about Google?
It remains the priority. None of the above competes with the work on Google: a well-resolved robots.txt, a correct sitemap and clear content serve both sides equally. The difference is that on Google nearly every site has done that homework and on Bing hardly any has.
OpenAI states that placement in its results is not guaranteed and depends on several relevance and reliability factors. Being in Bing and allowing its crawler does not secure a citation, but without those two conditions the chances drop considerably.
To see how this fits into a broader strategy, continue with what GEO is and why it does not replace SEO.
Sources: searching the web with ChatGPT and OpenAI’s crawlers, from OpenAI’s documentation, and the AI Performance announcement on the Bing blog.
Try it
- robots.txt testerWhich URLs each crawler may fetch, AI bots included, under the actual rules in your file.Open the tool, free
- IndexNow submitterTell Bing, Yandex and friends about new or changed pages, without waiting for the crawl.Open the tool, free
- Sitemap checkerHow many URLs it declares, whether they are well formed, and a sample of which ones actually respond.Open the tool, free