Is ChatGPT a Black Box? How We Know What AI Says About Your Business

Here's the question we hear on almost every first call. "How do you know what ChatGPT is saying about me? Isn't it a black box?" It's a fair question, and the honest answer is that two different things are hiding inside it. One of them is a black box. The other is about as transparent as a printed receipt.

The black box is the why. Nobody, including the companies that build these models, can fully explain why a model chose one business over another in a given answer. In 2025, Anthropic's CEO Dario Amodei wrote in an essay on interpretability that when a generative AI system does something, "we have no idea, at a specific or precise level" why it makes the choices it does. That's the company behind Claude, describing its own product. So when a vendor tells you they know exactly why Gemini skipped you last Tuesday, be skeptical.

The what, though, isn't hidden at all. Every answer an assistant gives is a block of text plus a list of the web pages it looked at to write it. You can read it, save it, date it, and compare it with next week's. That's the whole basis of AI visibility tracking, and it's what this post walks through. We'll show how our system asks the question, what it keeps, how it reads the reply, and where its view stops short of what a customer sees on their own phone.

The app on your phone isn't the only way in. Each AI company sells a developer channel to its models, built so that other software can ask a question and get the answer back as data instead of as pixels on a screen. That's how Lighthouse Local's AI visibility tracker reaches four of the five we track, with web search switched on, the way a person hunting for a local business would use them.

The reply comes back with receipts. OpenAI's Responses API reference defines a URL citation as a citation for a web resource used to generate a model response, and each one carries that page's address and title. Google's Grounding with Google Search for Gemini returns each cited URL along with the exact span of the answer it supports. Anthropic's web search tool for Claude attaches the URL, the page title, and the quoted text for each source. Perplexity's Sonar and Agent API returns the URLs of the sources used to generate the response. Four companies, four documents, the same promise. Ask through the front door and you get the answer and its sources as structured data, not a screenshot.

Google AI is the fifth, and it works differently, which is worth being precise about. No developer channel hands out AI Overviews, so our system runs the query on a live Google results page for the business's location and reads the AI Overview block exactly as it appears there. Of the five we track, it's the closest to what a real searcher sees.

Here's what lands in our database when the tracker asks one question. The business is Bluewater Pool Care in Sarasota, Florida, an example we'll carry through the rest of this post. The owner typed the question the way a customer would, and because the prompt is marked local, the system folds the city and state into what it sends, so the assistant isn't left guessing which Sarasota.

platform:        ChatGPT
asked:           2026-09-08, 06:14 UTC
question typed:  who is a reliable pool cleaning service in Sarasota
question sent:   who is a reliable pool cleaning service in Sarasota, Florida
web search:      on
answer:          kept word for word (214 words)
                 "For weekly pool cleaning in Sarasota, three companies come
                 up often. Suncoast Pool Pros has over 300 reviews and offers
                 same-day service. Bluewater Pool Care, at bluewaterpoolcare.com,
                 is praised for reliable weekly visits and clear pricing. Gulf
                 Gate Pools is a smaller option for the south side..."
sources:         1. yelp.com/search?find_desc=pool+cleaning&find_loc=Sarasota
                 2. google.com/maps/place/Suncoast+Pool+Pros
                 3. bluewaterpoolcare.com/weekly-service
raw reply:       kept, so the answer can be re-read later without asking again
verdict:         Mentioned + Cited, named 2nd of 3 businesses

Three things to notice. The answer is stored word for word, not summarized, so you can read the exact sentence a customer would have read. The sources are stored in the order the assistant listed them, so you can see which page of yours, if any, did the work. And the raw reply from the provider is kept as well, so when we improve how answers are read, every old answer gets re-read without a single new request. That's also why the dashboard can show you what an assistant said in March, not only what it says today.

Storing text is easy. Deciding whether the text mentions you is where the real work sits, because assistants rarely write a business name the way it appears on your invoice. They drop the LLC, add the city, type your domain instead of your name, or spell it the way Google Maps does. So the tracker keeps a list of names for each business, not one. It starts with the name you typed, adds the name on your Google Business Profile the first time it finds your listing, and lets you add an "also known as" for anything else, a former name, an abbreviation, the way locals say it. A competitor gets the same treatment on its own, with its name pulled from its Google listing and from its own homepage.

Every answer is then read against that list. A match on any of the names counts as a mention, and so does your exact domain typed in the visible text, because "bluewaterpoolcare.com handles weekly service" is a recommendation even without the name. Position is recorded too. If you're the second business named out of three, that's stored, since being named first and being named last are different outcomes for the customer reading.

Being cited is scored separately from being mentioned, and the difference matters more than it sounds. Cited means one of your pages appears in the sources list. Mentioned means your name appears in the answer text. An assistant can pull facts from your site and never say your name, or name you without linking anywhere. So the dashboard labels every answer mentioned, not mentioned, or competitor won, with cited tracked as its own flag, and that same reading pass ranks every competitor you track in that answer. On the answers page the matched names are highlighted in the text itself, so what the system counted and what you can see are the same thing. Which of your pages get cited, and why, is its own subject, and our guide to how AI sources local business facts covers it.

Ask ChatGPT the same question twice and don't expect the same answer. In 2026, SparkToro had 600 volunteers run twelve prompts a combined 2,961 times through three assistants, ChatGPT among them, and found less than a 1 in 100 chance that ChatGPT or Google's AI would give the same list of brands in any two responses. The same list in the same order came up about one time in a thousand.

That's not a bug someone forgot to fix. In 2025, researchers at Thinking Machines Lab showed that even with the randomness setting turned all the way down, the way these models are served in batches means the same question can produce different words. Variation is built into the plumbing.

So a single answer, ours or anyone's, is a coin flip weighted by how visible you really are, and the tracker treats it that way. Each prompt is asked at most once per platform per day, on a schedule, with weekly runs through your full prompt pool plus daily checks on whichever prompts you've flagged as priority, and every answer is kept. What you read on the dashboard is the rate, how often you were named across all of those answers, and the trend of that rate over weeks, which is the number that holds still long enough to act on. The math behind why a rate needs a few hundred answers before a change is real is worked out in our guide to tracking your brand's visibility in AI search.

Now the limits, because a window is only useful when you know where its frame is.

The developer channel has no memory of your customer. When a real person asks ChatGPT for a pool service, the answer can be shaped by their past chats, their saved preferences, and their account. Google is going further still. In 2026 it announced Personal Intelligence in AI Mode, which can draw on a user's Gmail and Photos to make search feel, in Google's words, uniquely yours. Our tracker asks as a stranger with no history. That's a limitation and a feature at once. You'll never see the version of the answer that was bent by one person's inbox, but you do see the baseline the assistant starts from for everyone.

The model behind the developer channel may not be the exact one a logged-in user gets that day, and the companies change their consumer defaults without notice. Location handling differs by platform as well. Some accept a city as a setting, others only understand the city when it's in the question, which is why the tracker writes it into the question for local prompts.

What this adds up to is a representative sample, not a mirror of any one screen. It has the same standing a poll does. No poll tells you what your neighbor will say, and a good one still tells you where you stand. When you want to see one specific screen, open the app and ask, and our guide to how to tell whether AI recommends your business shows how to read what comes back.

You don't have to take any of this on faith. Open ChatGPT, start a new chat, and paste the prompt below with your own details filled in. Then close that chat, open a fresh one, and paste it again. Do it a third time.

I'm looking for a [service] in [city, state]. Recommend three businesses and say why you picked each one. Then list the web pages you used, with a link for each. Don't ask me any follow-up questions.

You'll have three answers. Count how many name you, note where you sit in each list, and look at which pages show up in the sources. That's a hand-run version of what the tracker does, three answers instead of a few hundred, one assistant instead of ChatGPT, Gemini, Claude, Perplexity, and Google AI, and nothing saved for next month. If your name showed up in none of the three, the fix usually starts on your own site, and our free AI visibility audit returns a ranked list of what to change. It doesn't look at what the assistants have said about you. It looks at your site. Reading the answers is the tracker's job.

So, is ChatGPT a black box? Its reasoning is, and anyone who claims otherwise is selling something. Its answers aren't. They can be requested through a door each company built for this, stored word for word with their sources, read against every name you go by, and stacked up until the noise settles into a rate. That's what our AI visibility tracker does on a schedule, and you can read the full process on our how-it-works page. What Lighthouse Local for Business adds on top is the team that fixes whatever the answers surface, with every step run past you first. You can't see why an assistant made its choice. You can see every choice it made, and that's the part you can change.