Scout7 logo

Scout7

Back

research_paper

AI Crawler Optimization: 4 of 32 Pricing Pages Showed Bots No Price. None Used a Login.

September 24, 2026 · 12 min read · Scout7

We fetched 32 SaaS pricing pages like an AI agent: 28 showed a price. The 4 misses each failed a different way. Here are the 4 checks to run on yours.

On 24 September 2026 we fetched 32 B2B SaaS pricing pages the way many AI agents do: a GPTBot user-agent, no JavaScript, no cookies. 28 of them showed a real price in the plain HTML. The other 4 showed the bot no price at all. Not one of those 4 used a login.

Each of the 4 failed in a different way. A 403 sent only to the bot. A redirect chain the bot could not finish. A price that lived only in structured data. A price that appeared only after JavaScript ran. All four look fine when you open the page in Chrome. So you would never notice. This is AI crawler optimization at its most basic: can the bot read the number?

Key takeaways:

  • 28 of 32 pricing pages (88%) gave an AI agent a readable price
  • 4 of 32 failed, each for a different reason, and none of them used a login
  • robots.txt was not the problem: 0 of 32 blocked OAI-SearchBot or PerplexityBot
  • You can check your own pricing page in 5 minutes with four terminal commands

Can AI agents read my pricing page? Probably, if it is server-rendered. The risk is a quiet failure you only see from a terminal.

What did we test on 32 SaaS pricing pages?

We fetched 32 B2B SaaS pricing pages with a small Python script. No JavaScript ran, no cookies were kept, and each request carried the GPTBot user-agent string. We also fetched each page with a normal Chrome user-agent, to compare.

Why no JavaScript? Because that is how most AI crawlers read the web. Vercel's crawler study found that "none of the major AI crawlers currently render JavaScript" (Vercel, Dec 2024). That study is older than a year, but we found nothing newer that says otherwise.

  • Work tools: Notion, Linear, Asana, monday.com, ClickUp, Airtable, Slack, Zoom, Miro, Calendly, Loom
  • Developer tools: Supabase, Vercel, Netlify, GitHub, GitLab, Postman, Sentry, Datadog, Retool
  • Sales and support: Intercom, Zendesk, Freshdesk, Pipedrive, Attio
  • Design, docs and security: Figma, Webflow, Typeform, DocuSign, Dropbox, 1Password, Jira
  • How: Python urllib from one machine in India, one fetch per page per user-agent
  • What we read: the visible text of the raw HTML (scripts and styles stripped), plus each site's robots.txt

We left out content-marketing tools on purpose, since that is our own category.

We counted a page as readable when its plain HTML text held a currency amount like $10 or ₹545. By that test, 28 of 32 pages (87.5%) passed. For every page we logged the status codes, the prices each fetch saw, and each robots.txt rule.

# Basic non-rendering fetch simulation
curl -s -A "GPTBot/1.2" https://example.com/pricing | grep -o '\$[0-9][0-9.,]*'

That one-liner is the rough idea. Our script went one step further and stripped scripts and styles first. That matters: 1Password's raw HTML contains "$0 - $250,000" as a label in a sales form, which a plain grep would count as a price.

How many pricing pages could an AI agent actually read?

Out of 32 tested platforms, exactly 28 returned explicit numerical prices on the very first terminal request. Well-engineered platforms like Notion ($10, $20), Asana ($10.99, $24.99), and Supabase ($25) delivered complete plan details within plain static HTML.

We are not the first to test this. Two other 2026 studies looked at wider samples and found lower pass rates:

  • Growth Unhinged and Profound (September 2026) found that "57 of the 77 public pricing pages were AI-ready" (Growth Unhinged)
  • Siteline (June 2026) ran a Claude agent that does not render JavaScript across 100 B2B products and 534 runs. "65% of plans surfaced pricing directly" (Siteline)

Our sample was 32 large, well-funded brands. They mostly get this right. So the lesson is not "AI can't read pricing". The lesson is in the exceptions.

One more thing surprised us. For 2 of the 32 pages, the page a bot got was different from the page a browser got. Pipedrive sent the browser a normal page and sent the GPTBot user-agent a 403. Zoom did the opposite: our browser fetch from India showed no prices, but the GPTBot fetch showed ten. What you see in your own browser is not proof of what an agent sees.

Why did the other 4 pricing pages show no price?

None of the four used a login, a sales call or a gated form. Each one hit a different technical side-effect. Here is what our fetch got on 24 September 2026:

Page What the GPTBot fetch got Cause
Pipedrive HTTP 403, no page The server turned away the bot user-agent. robots.txt allows GPTBot. A browser user-agent got a 200 and prices like $14 and $24.
DocuSign A 307 with an empty body /products-and-pricing redirects to ecom.docusign.com, then through a chain of 307s on an A/B-test path. A fetcher with no cookies never reached a page.
Jira A page, but no price in the text The price sits only in JSON-LD structured data: 7.91 and 14.54 USD per user per month. The visible price appears after JavaScript runs.
1Password A page, but no plan price anywhere No plan price in the raw HTML at all. Prices load with JavaScript.

Jira's case is the interesting one. The price is in the page, just not where a text reader looks:

"priceSpecification":{"@type":"UnitPriceSpecification","price":"7.91","priceCurrency":"USD","billingDuration":"P1M"}

Some AI fetchers may read that. Many readers of the page text will not. A Hacker News user asked the same question last year: "Since many AI crawlers do not render JavaScript, what will happen to your Structured Data written in JSON-LD and implemented using Google Tag Manager without a server-side rendering strategy?" (Hacker News).

A fair caveat on Pipedrive. We sent a GPTBot user-agent string from our own machine. The real GPTBot comes from OpenAI's IP ranges, and a CDN may treat it differently. What we can say is this: a request that says "I am GPTBot" got a 403.

Is robots.txt blocking AI agents from your pricing page?

In our sample, rarely. Here is how many of the 32 robots.txt files blocked each bot from the pricing page:

Bot What it is Pages that block it
GPTBot OpenAI's training crawler 2 of 32 (Figma, Loom)
ClaudeBot Anthropic's training crawler 1 of 32 (Figma)
OAI-SearchBot OpenAI's search bot 0 of 32
PerplexityBot Perplexity's search bot 0 of 32

Figma shows the pattern clearly. Its robots.txt blocks GPTBot and ClaudeBot from the whole site. But a separate rule lets OAI-SearchBot, PerplexityBot, Claude-SearchBot and Claude-User read /pricing/. That is a clear choice: no training, yes to answers.

The bigger risk sits outside robots.txt. Pipedrive's robots.txt allowed GPTBot, and the server still said 403. Your CDN or firewall can overrule the file. That is worth checking now, because defaults are moving. From 15 September 2026, Cloudflare says that for new domains, "Training and Agent will be blocked by default on the pages that display ads, while Search will remain allowed by default" (Cloudflare).

If robots.txt says yes and your firewall says no, the bot gets no.

# Selective crawler governance example
User-agent: GPTBot
Disallow: /pricing

User-agent: OAI-SearchBot
Allow: /pricing

If you do block training bots, do it on purpose, and keep the answer bots in.

Why does a bot-readable price matter to B2B buyers?

Because buyers now ask an AI before they ask you. Three 2026 surveys point the same way:

  • 82% of buyers "sourced software recommendations from an AI chatbot in the last 24 months" (G2 2026 Buyer Behavior Report, 1,038 B2B software decision-makers, G2 via PR Newswire)
  • Transparent pricing "has been buyers' #1 wish-list item for vendors for four years running" (TrustRadius 2026, 1,862 buyers, TrustRadius / HG Insights)
  • 67% of B2B buyers favor a rep-free buying experience (Gartner, 646 buyers, as reported by Digital Commerce 360)

Buyers say it plainly too. "I get put off by "Contact Sales" pricing," one wrote on Hacker News this month (Hacker News).

And the AI shortlist is short. I asked ChatGPT "best project management tools" in 20 fresh chats. The same 6 tools came back in all 20. Only 11 names showed up in the whole category (our 20-run test). If a bot can't read your price, it has plenty of other names to give instead.

So how do you check what a bot sees on your own pricing page?

AI crawler optimization in 5 minutes: how do you check your pricing page?

Four terminal commands. No tools to buy. Run them from a terminal, not a browser, because your browser runs JavaScript and carries cookies. A bot does neither.

Step 1: Check the status a bot gets

Send a request with the GPTBot user-agent and print only the status code.

curl -s -o /dev/null -w "HTTP Status: %{http_code}\n" \
  -A "Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; GPTBot/1.2; +https://openai.com/gptbot)" \
  https://yoursite.com/pricing

You want 200. A 403 means a bot rule at your CDN or firewall is turning the request away, even if robots.txt allows it. That was Pipedrive's result.

Step 2: Look for your price in the raw HTML

Fetch the page the same way and search it for a dollar amount.

curl -sL -A "Mozilla/5.0 (compatible; GPTBot/1.2; +https://openai.com/gptbot)" \
  https://yoursite.com/pricing | grep -o '\$[0-9][0-9.,]*' | head -n 10

You want your real plan prices. Empty output means the price only appears after JavaScript (1Password) or only in structured data (Jira). The fix is to server-render the plan cards so the number is in the HTML text. This grep also catches any $ inside scripts or forms, so check that the numbers it prints are your plan prices. 1Password's page would print "$0" and "$250,000" from a revenue drop-down, not a plan price.

Step 3: Follow the redirects with no cookies

List every hop the pricing URL takes.

curl -sIL https://yoursite.com/pricing | grep -iE '^HTTP|^location'

You want a short chain that ends in a 200. A chain of 307s that never lands, as with DocuSign, means a fetcher without cookies gets nothing. A/B-test and geo redirects are the usual suspects.

Step 4: Read robots.txt for the AI bots

Print the rules for the training bots and the answer bots.

curl -s https://yoursite.com/robots.txt | grep -iA 2 -E 'GPTBot|OAI-SearchBot|PerplexityBot'

Decide on purpose. Blocking training bots is a fair choice. Blocking OAI-SearchBot or PerplexityBot from /pricing means those answer engines can't read your price. Also add ClaudeBot and Claude-SearchBot to the grep if Claude matters to your buyers.

  • Status: 200 for the bot user-agent
  • Price: your real prices in the raw HTML text
  • Redirects: a short chain that ends in 200
  • robots.txt: answer bots allowed on /pricing

What can't this test tell you?

A lot. Here are the limits, said plainly:

  • One day, one place. We fetched each page once per user-agent on 24 September 2026, from India. Pages change and run A/B tests.
  • A user-agent string is not the real bot. Real crawlers come from their own IP ranges, and a CDN may treat them differently.
  • 32 big brands, not small SaaS. Smaller sites may do better or worse. This test does not measure them.
  • Readable is not the same as quoted right. We checked whether a price is in the HTML, not whether an AI answer repeats it correctly. That is a different test.
  • Some fetchers may read JSON-LD. Jira's price could still reach an agent that parses structured data.

For more on seeing your site the way a crawler does, read 5 minutes to see your site like GPTBot and our take on blocking AI bots in robots.txt.

Have you checked if ChatGPT knows your price? Run the four checks, then ask it.

Frequently asked questions

Can ChatGPT read my pricing if it loads with JavaScript?

Often not. Vercel's crawler study found that none of the major AI crawlers rendered JavaScript. In our test, 1Password's plan prices were missing from the raw HTML because they load with JavaScript. Put the price in the server-rendered HTML to be safe.

Does a JSON-LD price count if the page text has none?

It depends on the fetcher. Jira's price was only in JSON-LD structured data in our test. An agent that parses structured data may find it. An agent that reads the page text will not. Having the price in both places removes the doubt.

Why does my site return 403 to GPTBot?

Usually a bot rule at your CDN or firewall, not your robots.txt. Pipedrive's robots.txt allowed GPTBot, yet a GPTBot user-agent got a 403. Check your bot settings, and remember that the real GPTBot comes from OpenAI's IP ranges.

Does robots.txt stop AI answer engines from seeing my pricing?

Not in our sample. 0 of 32 blocked OAI-SearchBot or PerplexityBot from the pricing page. 2 of 32 blocked OpenAI's training crawler, GPTBot.

Scout7 checks whether AI answers mention you, then writes and publishes the content that helps you show up.

Sources