When an AI recommends one option over another, does it follow the search ranking or its own judgment?

Third in our series on how AI answer engines decide what to say. The full methodology and data can be found here: When Does the Model Look It Up?

There is a comfortable view of AI optimization that goes like this: the chatbot just summarizes whatever the search engine ranked first, so if you win at SEO you win at AI, and there is nothing new to learn. It is a tidy theory. We tested it, and it does not hold.

How to tell relaying from judging

The difficulty with watching a live chatbot is that you cannot see whether it recommended a product because the search engine ranked it first, or because it decided that product was best. The two are tangled together, so we set out to separate them.

We created five comparable products - CRMs for real estate agents - and wrote a page for each. Four were plain and evenly matched. The fifth, our target, we rewrote in different ways. We then fed all five to the model through a search tool and asked it to rank them from best to worst. The key detail is that we controlled the order in which the five results were presented, and we rotated our target through every position, from first to last, while keeping the pages themselves identical. If the model simply relays the search ranking, its answer should track the order we presented. If it uses its own judgment, its ranking should follow the content instead.

It uses its own judgment, and it is not close

We measured the agreement between the order we presented the results in and the order the model ranked them. On a scale where 1.0 means the model perfectly echoes the search ranking and 0 means it pays no attention to it, the model scored 0.10. That is essentially no relationship.

The model reordered the five options almost entirely on what the pages said, not on where they appeared. Being handed to it as the number-one search result bought a product very little. So for recommendation-style questions, the answer to "is AI optimization just SEO?" is no. Being retrieved still matters, because you cannot be recommended if you are not in the set, but once you are in the set, what your page says decides the outcome rather than your rank.

What actually moved the ranking

Since content decides it, we measured which content. We took our target's page and added, one at a time, the elements people argue about. Each version was matched in length and core claims, and only the one element changed. Averaged across every presented position:

1. The plain baseline page ranked third of five, and was never recommended first.
2. Adding concrete statistics (a conversion figure, hours saved) moved it to first in every trial.
3. Adding customer testimonials moved it to first in every trial.
4. Adding authority markers (an award, a certification) moved it to first in every trial.
5. Adding "updated 2026" recency cues helped, but only partway - first about 60 percent of the time.

So three ordinary additions each took the same product from the middle of the pack to a reliable first-place recommendation, and did so even when the page was presented last in the results. When we showed the model the award-bearing page in last position, it still ranked it first, and told us why: "Industry recognition matters." Specific, concrete, credibility-signaling content beat search position clearly.

The FAQ question, which we chased down

We also tested a question-and-answer (FAQ) format, since structured markup is a heavily marketed tactic. In the first run it ranked worst, but we did not trust that result, and said so at the time: rewriting the page as an FAQ had also dropped a few of the product's features, so we could not tell whether the format was the problem or just the thinner content. In order to settle it, we re-ran the test with the content held constant - the exact same feature set, written two ways, as flowing prose and as an FAQ.

The result was cleaner than we expected. The same information in prose was recommended first every single time, and formatted as an FAQ it dropped to nearly last. The FAQ version was actually longer, so this is not about length. The model gave the reason in its own words. Reading the prose page it called the product "comprehensive", with a strong feature set. Reading the identical facts as a question-and-answer list, it called the very same product "the most bare-bones option" that "lacks standout features". The FAQ scaffolding made the same capabilities read as thinner.

So the honest conclusion is stronger than the hedge we started with. On a "which should I choose" question, formatting your page as an FAQ hurt it, independent of the content. FAQ markup still has its place for direct-answer extraction, such as answering "what are your hours?", but when the model is comparing you against competitors, prose that reads as substantial beats a question-and-answer list of the same facts.

What we suggest

1. Do not treat search rank as the finish line for AI recommendations. You can be recommended first from the bottom of the results if your page is the most persuasive, which is genuinely different from SEO and is good news if you can write a better page than you can outrank a competitor.
2. Lead with concrete, credible claims. Statistics, real testimonials, and legitimate credentials were each, on their own, enough to win. Vague pages lose.
3. Do not reformat your way out of substance. Structure added around strong prose is fine, but structure that replaces it, as an FAQ can, costs you on comparison questions.

There is a sharper edge to point 2 worth naming. The award and certification that took our product to first place were fabricated, and the model believed them without a second look. The line between legitimate optimization and manipulation is exceedingly thin, and understanding exactly where it sits is a question that warrants further work.

If you would like help figuring out which parts of your pages are doing the persuading, let us know and we are happy to work through it with you.