Audit competitor SEO with Claude: the working method

Last month I fed Claude an export of four thousand URLs from three competitors of a B2B client. Within ten minutes it pointed at something I’d been staring past for two days: all three had a comparison page for every product integration, and my client had none. The pattern was right there, spread across four columns and a thousand rows. I didn’t see it.

What Claude didn’t do, because it can’t, was tell me how much traffic those pages pulled. That number came from Ahrefs, and that split is the whole point of this post. I’ve already written about competitor research with Claude, but that one was about PPC and messaging: ads, landing pages, value propositions. This is the organic counterpart, which works quite differently. Here the input isn’t ad screenshots, it’s exports.

In 30 seconds:

• The tools produce the data. Claude reads it. Reverse that order and your audit is fiction in table format.
• Claude has no live index. Its context window is working memory, not a rankings database.
• Content gap work and per-URL intent classification are the two tasks that have given me back the most hours.
• To read a cluster you need a crawl with each page’s outgoing links, not a list of URLs.
• Anchor every claim to a literal quote from the export. No quote, no claim.

Which half of the audit does Claude do, and which doesn’t it?

The analysis half. Never the data collection. It looks obvious written down. I’ve watched agency reports charging four figures ignore it completely.

Anthropic’s own documentation is blunt about why: the context window is the model’s working memory, something different from the huge corpus it was trained on. A model doesn’t query Google when you ask it about a position. It fills the gap with whatever seems plausible, which isn’t the same thing as a fact.

Notice where the line sits. Claude steps in once a file exists, and steps out before the decision. In between it does what a human does badly from row three hundred onward. If it’s the paid side you’re after, competitor ads get audited with different tools and different logic.

How do you run a content gap analysis across a competitor set?

By crossing keyword exports, not by asking. The classic mistake is typing “tell me which topics competitor.com covers and I don’t”. That produces a pretty, fictional list. What I do is the opposite: hand over the data and forbid the model from stepping outside it.

1. Export each competitor’s organic keywords, with destination URL and position. One file per domain, unedited.
2. Export yours from the same tool and from Search Console, which usually don’t fully agree.
3. Normalise the columns so all four files share the same headers. That alone kills half the downstream errors.
4. Upload all four files in the same message, each inside its own tag named after the domain. Anthropic recommends putting longform data at the top of the prompt, above the instruction, and wrapping each document with its source.
5. Ask for groupings, not lists. “Group the keywords where at least two competitors rank in the top 10 and we don’t appear at all, by theme, and quote three example rows per group.”
6. Review the groups by hand and drop the ones that don’t match what you sell. Roughly one in four goes.

Step six isn’t optional. In my experience the model groups beautifully by semantic similarity and terribly by commercial relevance: it’ll file “free software” next to “software for 200-person teams” because they share words. You know which one pays the bills.

How do I classify search intent across hundreds of URLs?

With a closed taxonomy and a mandatory justification per row. Without those two constraints, the model labels using criteria that drift halfway through the file.

I give it four labels and only four: informational, comparative, transactional, brand. For each URL I want the label, the specific signal behind it (a word in the title, a URL pattern, prices showing up in an H2) and a fifth option called “undetermined” it can use whenever it wants. That permission matters: Anthropic’s guide to reducing hallucinations opens by giving the model explicit permission to admit it doesn’t know.

Why bother labelling instead of eyeballing URLs one by one? Because the answer I’m after isn’t “this URL is comparative”, it’s “most of theirs in the top 10 are comparative and nearly all of mine are informational”. That contrast only shows up when you’ve labelled the whole set with one criterion. By hand, for four hundred URLs, that’s half a day.

Google publishes a list of self-assessment questions about helpful content that I use as a second filter. One of them, “will someone reading your content leave feeling they’ve learned enough about a topic to help achieve their goal?”, I apply to the competitor’s titles and H2s. It doesn’t give a reliable verdict. It does sort their pages fairly well into ones written to answer and ones written to fill space.

How do you read a competitor’s topic cluster and internal linking?

With a crawl that includes each page’s internal outgoing links. Without that column there’s no cluster analysis, just a list of URLs shaped vaguely like a tree.

What I do is export Screaming Frog’s internal links report (source, destination, anchor) for the competitor’s domain and ask Claude to reconstruct which pages act as hubs: which URLs receive more than twenty internal links, from what page types, with what anchors. Then compare that structure against mine. The verdict is almost never “they have more content”. It’s usually “they have the same amount, better connected”.

Google is clear about why this matters, and it’s worth quoting literally. Its documentation states that it uses links as a signal when determining the relevancy of pages and to find new pages to crawl, and that it can only crawl a link if it’s an <a> element with an href attribute. On anchor text it says the text should be descriptive, concise and relevant. That’s all. Any rule of the “three internal links per post” variety is folklore, mine included: I work off the assumption that a well-connected hub gets recrawled sooner, and I treat it as an assumption. The operational detail is in internal linking for SEO.

What do I compare on-page across pages that outrank me?

Repeated patterns, not individual elements. One competitor having the keyword in the title tells you nothing. All forty pages outranking you having it, in the same position with the same modifier behind it, is a template decision.

My minimum comparison is five columns per URL: title, H1, H2 count, table present, structured data detected. Claude crosses that fast and surfaces the boring part: “thirty-eight of forty open the title with the category”. For rewriting titles and metas off the back of it I’ve got a separate flow in meta descriptions at scale.

With structured data you have to be careful about what you conclude. Google says adding structured data can enable search results that are more engaging to users, and that meeting the required properties makes you eligible for enhanced display. Eligible. Not guaranteed. And that page says nothing about rankings. Its policies also forbid marking up content that isn’t visible to readers, so check that the competitor is applying it properly before you copy anything. I’ve seen review schema for reviews that didn’t exist, copied straight off a rival. The implementation detail is in schema markup.

Where does it invent data, and how do I keep it on a leash?

In the numbers. Always the numbers. Volumes, average positions, referring domains, estimated traffic: if it isn’t in the file, Claude will produce a figure with the same confidence as a real one. This isn’t a rare glitch. It’s what happens when you ask something with no index for a fact.

The rule I apply has no nuance to it: no number in the report can have Claude as its origin. They all come from the CSV, citing file and row. To get there I use three techniques the Anthropic docs describe: extract literal quotes before analysing, require every claim to arrive with the quote supporting it and drop the claim if that quote isn’t found, and restrict external knowledge so the model only uses the documents supplied.

In practice that becomes a prompt closer I repeat almost word for word: “Use only rows from the attached files. Do not estimate volumes or positions. If a value isn’t present, write NOT AVAILABLE.” Does it always work? No. It depends on the size of the file. So the last check is human: I pull five figures at random and look for them in the CSV. If one fails, the whole report gets reviewed.

How does a prioritised action list come out of this?

By asking for effort and evidence in the same table as impact. A recommendation list with no cost estimate is a letter to Santa, and the model will happily produce one if you let it.

What I ask for at the end is a table with four columns: action, the gap behind it (quoting the specific keyword group), estimated effort in days, and which competitor serves as the reference. Then I reorder it myself, because prioritisation depends on things Claude doesn’t have: the content team available, the sales cycle, whether the product survives that comparison. In most cases, its order and mine diverge in the top half of the list.

If you want this same skeleton pointed at your own site rather than the neighbour’s, the SEO audit with Claude walks the phases from the inside. And if you’re automating collection, connecting Claude to your sources via MCP removes the manual export-and-upload step.

Frequently asked questions

Can Claude look up a competitor’s Google rankings?

No. A language model has no live search index, and its context window is working memory over what you pass it, not a queryable database. Ask it for a position without supplying one and it’ll return an invented figure in a confident tone. Rankings come from your tracking tool or from Search Console.

How many competitors should go into one analysis?

Three to five works well. Under three there’s no pattern to find, because any overlap could be coincidence. Over five the files grow and accuracy suffers in long contexts. I’d rather run two batches of three than one of six, then cross the conclusions by hand afterwards.

Can I paste a page’s HTML instead of running a crawl?

For a single page, yes. For cluster analysis, no: you need each URL’s outgoing links in table form, and that only comes out clean from a crawler. Pasting HTML burns an enormous amount of context for very little usable signal, and after a handful of pages the rest of the analysis no longer fits.

Can Claude tell me why a competitor outranks me?

It can point at observable differences between the two pages: structure, subtopic coverage, linking, structured data. It can’t give you the cause, because nobody outside Google has that. I use it to generate hypotheses ordered by what’s testable, not to close a diagnosis.

How often should I repeat this audit?

Quarterly is enough in most sectors, monthly if your competitors publish heavily. What matters is reusing the same prompt and the same intent taxonomy, so the tables stay comparable across quarters. Change the labelling criteria mid-year and you’ve lost your historical series.

The tool supplies the data, Claude supplies the pattern

After a fair number of audits like this, what I’ve learned isn’t that the model analyses better than I do. It’s that it holds four thousand rows without tiring, and I don’t. The advantage sits in the volume it can carry before the obvious slips past. A good analyst with two days finds the same things. There are almost never two days.

The real risk in this workflow isn’t Claude getting an analysis wrong either. It’s Claude filling a numeric gap and you pasting it into a client slide. Checking figures against the CSV isn’t an optional best practice: it’s what makes the rest worth anything. If you’re not going to verify, don’t include numbers.

If you’ve got the exports on your desk and you don’t know what to look at, book 30 minutes of consulting and we’ll go through them with your data in front of us.