How to A/B test a landing page

You want to know if a new headline or offer will get more people to sign up, book or buy, so you plan to A/B test your landing page. At 300 visits a month, a normal A/B test can take many months to tell you anything, and most founders stop early and keep a "winner" that was luck. Here is how to A/B test a landing page the honest way: count what your traffic can tell you first, then test where you have the numbers.

Do these three things

About 1 hour to set up today, then the 3 hours a day of sending you already do. A page test runs 2 to 4 whole weeks; a message test takes about 12 days.

Show the three actions in fullHide the full list

1. 1Today: put your visits and signups into a free calculator that says how many visits an A/B test needs. If a page test would take more than 4 weeks of your traffic, test your new headline in your messages instead.
2. 2Every day: send 100 people who fit to the test, half to version A and half to version B, and do not look at the result until each version reaches its planned number.
3. 3Every Sunday: count visits, signups and replies for each version in one sheet. When both reach the planned number, check the result once, keep the winner and start the next test.

Get help doing these

ChatGPT or Claude reads this guide and works through it with you.

Do this with ChatGPT(opens in a new tab)Do this with Claude(opens in a new tab)

1. Work out how many visits your A/B test needs before you start

An A/B test shows two versions of your page to different visitors and counts which one gets more conversions. Before you build anything, work out how many visits each version needs. At 300 visits a month, the answer decides whether you test the page at all.

Open your site stats, for example Google Analytics, Shopify Analytics or your site builder's stats. Write down two numbers for the last 4 weeks: visits to the page, and how many of those visitors took your main action. Divide to get actions per 100 visits. If you do not count the action yet, set up conversion tracking first. A test with no reliable count is a guess.

If you run a SaaS: the action is a trial or account started. Count people who used the main feature as well. That is your activation rate, and a page that brings more signups but fewer active users has not won.

If you run a B2B company or an agency: the action is a call booked or a quote asked for, from your enquiry form.

If you run a D2C store: the action is an order. With very few orders, also count "add to cart", which happens more often. Carts that are not paid for are a separate problem, with their own fixes for abandoned carts.

Then open Evan Miller's free sample size calculator (evanmiller.org/ab-testing/sample-size.html). It uses the usual settings: 95% confidence and 80% power. Fill in two boxes:

• Baseline conversion rate: your actions per 100 visits, for example 3%.
• Minimum Detectable Effect: the smallest jump worth finding. At your traffic, ask for a big one. Choose "Absolute" and type the same number as your baseline. That means "find it if version B doubles the result".

The calculator shows "Sample size: N per variation". Multiply it by 2 for both versions, then divide by your visits a week. That is how many weeks the test needs.

Here is what that means at small traffic, with the calculator's own formula:

Then use this rule:

• 4 weeks or less: test the page itself. Go to step 2, then steps 4 to 6.
• More than 4 weeks: do not split your page traffic yet. Do step 2, then test the change in your messages in step 3. Keep counting the page with the "change one thing, count the next 300 visits" rule from writing a landing page that converts.

Why 4 weeks: a test should run in whole weeks, but the longer it runs, the more visitors come back on a new device or with cleared cookies and see both versions. Safari deletes cookies set by scripts after 7 days without a visit, and most testing tools rely on such cookies. Google also asks that tests run only as long as needed.

If you had fewer than 5 actions in the last 4 weeks, the calculator has nothing to work with. Skip the page test and go to step 3.

If you send 100 people who fit to the page every day, your visits a week go up, and the weeks go down. Count that traffic once you have it, not before.

• Tool: your site stats and Evan Miller's sample size calculator. Both free.
• Time: 20 minutes.
• You will have: one line in a sheet: "Needs [N] visits per version, [W] weeks at my traffic. Route: page test or message test."
• It worked if: you have a number of weeks, and you chose the route by the 4-week rule.
• Common mistake: starting the test first and "seeing what happens". After a month of 300 visits, a test with 9 signups on one side and 6 on the other tells you nothing, but it feels like an answer.

2. Pick one big change and write down what you expect

Test one big change to what the page promises, offers or proves, not a button colour or a single word. At small traffic, only a big difference can show up, so give the test something big to find.

Find the change in what visitors already show you:

1. Watch 10 recordings of visits in Microsoft Clarity, which is free with no traffic limit. Write down where each person stopped: the headline, the price, a worry, the form. The landing page guide shows how to set it up.
2. Run a five-second test with 10 people who fit your buyer. Show the page for 5 seconds, hide it, and ask: "What does it do, and who is it for?" Write down the answers word for word.
3. Pick the one part most people got wrong or stopped at, often the first screen or the button. That part is your test.

Good things to test, biggest first:

• The promise: the headline and the line under it, written for your ideal customer. Use your buyers' own words from customer interviews or your value proposition.
• The offer: a free trial against a demo, a sample pack against an order, a free audit against a call. For a SaaS, choosing between a free trial and a free plan is its own decision; make it before you test the offer.
• The proof: real testimonials and numbers near the button, against none.
• The price shown: on a pricing page, after you set the price itself. Test prices with care; showing two prices to people at the same time can upset buyers who compare.

Ask ChatGPT or Claude for 5 versions of the part you chose, using the words from the recordings and the five-second test. Pick the one most different from what you have now. Then write one line:

Version A (now): [the current headline, offer or proof, word for word]
Version B (new): [the new one, word for word]
If we change [the part] to version B, then more people will [the one action], because [what visitors or buyers told us].
We count: [the one action] per 100 visits (or replies per 100 messages).
Visits or messages needed per version: [the number from step 1].

Change only that part. If B has a new headline, a new photo and a shorter form, and it wins, you will not know which change did it.

• Tool: Microsoft Clarity (free), ChatGPT or Claude, a Google Doc.
• Time: about 1 hour.
• You will have: version A, version B and the test plan, written down.
• It worked if: you show version B to 10 people for 5 seconds, and at least 7 can say what it offers and who it is for.
• Common mistake: testing a button colour because it is easy. At 300 visits a month it can never show a difference, and you lose a month.

3. If your page is too small, A/B test the promise in your messages

When your page cannot finish a test in 4 weeks, test the promise where you already have volume: your 100 personal messages a day. Send version A's promise in half of them and version B's in the other half, each with a link to its own page. You compare replies in days instead of months.

This tests the promise, not the whole page. It answers the question most founders really have: "Which headline makes the right people want it?" When the new promise wins, put it on the page.

How to run it:

1. Make a copy of your page with only version B changed. Give it its own address, for example /start-b. In your page builder, set its canonical URL to the original page and keep it out of your menu and sitemap.
2. Tag each page link with UTM parameters, for example utm_content=a and utm_content=b, so visits and signups show up by version.
3. Write the opening of your message twice: version A's promise and version B's promise. Keep everything else the same.
4. Send your 100 personal messages a day from your lead list, alternating: A, B, A, B. Log the version next to each name.
5. Count replies, visits and signups per version. Work out the number first: put your replies per 100 messages into the same calculator, with the same settings as step 1. At about 3 replies per 100, finding a doubling needs 575 messages per version: 1,150 in all, about 12 days at 100 a day. Do not compare before then.

AI drafts each message from your template and one fact about the person; you add one true line about them. That is about a minute a message, so 100 messages take about 2 hours. The follow-ups stay on the same version as the first message.

Keep the volume safe. If you email more than about 50 strangers a day, send from a warmed second domain so your main email is not hurt. LinkedIn limits how many connection invitations you can send each week, so on LinkedIn message people you are already connected to and send the rest by email.

If you run a D2C store: you do not message shoppers one by one. Test the two promises as two ads with the same picture instead, using the method in testing Facebook ad creatives, each ad linking to its own page. Compare clicks per 100 views and add-to-carts. Keep the spend to a small, fixed budget. Also try the two promises in your 2 posts a day, alternating A and B, as extra evidence.

Version A
Subject: [the problem, in 3–5 words]
Hi [name], [one true line about them]. [Version A's promise, in one line.] Here is how it works: [page A link with utm_content=a]. Would it help with [the problem]?Version B
Subject: [the same subject]
Hi [name], [one true line about them]. [Version B's promise, in one line.] Here is how it works: [page B link with utm_content=b]. Would it help with [the problem]?

• Tool: Gmail or your email, LinkedIn, UTM tags, a Google Sheet, ChatGPT or Claude.
• Time: inside the 3 hours a day of sending you already do, plus 30 minutes to make page B.
• You will have: replies, visits and signups for each version, from people who fit.
• It worked if: 50 of each version went out every day, every reply is logged by version, and after 1,150 messages you have a result to check in step 6.
• Common mistake: sending version A on Monday and version B on Tuesday. Monday's people are not Tuesday's people. Alternate one message at a time.

4. Set up the split with a free tool

When your page can finish a test in 4 weeks, split the visitors on the page itself: half see version A, half see version B, at random, and each person keeps seeing the same version. Use the tool your platform already has.

If you run a Shopify store: use Shopify's own test, called Rollouts. Its A/B tests, called Experiments, need the Grow plan or higher ($79 a month billed yearly, or $105 monthly, checked 11 October 2026). In your Shopify admin go to Markets, then Rollouts, and click Create rollout. Name it and choose Experiment as the type. In Changes, click Add changes, then Edit theme to change your current theme in the theme editor, or pick another theme under Replace the current theme. Click Apply. New experiments split visitors 50% control and 50% treatment. Set the start under Experiment runs from, then click Set end date and choose the date from step 5. Rollouts cannot change Liquid template code or vintage themes. For a product page, use step 2's change on the product description or the trust lines near the price.

If your site is not on Shopify: use PostHog. Its free plan includes 1 million feature flag requests a month, and experiments are billed with feature flags, with no credit card needed. Add the PostHog snippet to your site, and add disable_web_experiments: false to its settings, or no-code tests will not run. Open the PostHog toolbar on your page, click the Experiment tab, then New experiment. Name it, select the test version, click Add element, choose the headline, type version B's text and click Save experiment. In PostHog, add your one action as the metric and launch. PostHog warns that no-code changes can fail on sites built as single-page apps; if your site is one, ask your developer to set the test up in code.

If you run Google search ads: the split above covers ad visitors too. One setting can spoil it. When AI Max is on, "Final URL expansion" is on by default and can send clicks to other pages on your site. Turn it off: Campaigns, Settings, click the campaign, open AI Max, and under Asset optimization uncheck Final URL expansion, then Save. To test two separate landing pages on ad traffic alone, use Campaigns, Experiments, Ad variations, choose Update URLs, and set the experiment split. Google splits people by cookie, so each person sees one version. More on the first ads in Google Ads for a SaaS.

Follow Google's rules for tests, so your page does not drop out of search:

• Show Google the same pages people see. Never send Google's crawler only to version A.
• If version B has its own address, add a canonical tag on it pointing to version A. Google prefers this to a noindex tag.
• If you redirect people to version B, use a temporary 302 redirect, not a permanent 301.
• End the test and remove version B as soon as you have the answer.

Then check it works. Open the page in 10 new private windows and confirm you see both versions. Do the action once on each and check that it is counted for the right version. Open both versions on your phone: B must load as fast as A, because a slow version loses for the wrong reason. Google counts a page as fast when its main content shows within 2.5 seconds. If either is slower than that, speed it up first.

• Tool: Shopify Rollouts (Grow plan), or PostHog (free), plus Google Ads if you run search ads.
• Time: about 1 hour.
• You will have: one live test, split 50/50, with an end date and the action counted for each version.
• It worked if: 10 private-window visits showed both versions, and a test action was counted in each.
• Common mistake: making version B a copy at a new address with no canonical tag, so Google sees two near-identical pages and may show the test page in search.

5. Run the test for whole weeks and do not stop early

Run the test until each version reaches the number of visits from step 1, in whole weeks: at least 2 weeks and at most 4. Do not look at the result before then, and do not change either page while it runs.

Every time you look and stop when one version is ahead, you raise the chance of a false winner. Evan Miller, who wrote the calculator in step 1, shows that checking ten times can turn a result you think is 1% likely to be luck into one that is about 5% likely. Decide the number first, then wait.

Every day, send 100 people who fit to the page:

• your 100 personal messages, with the page link, logged in your sheet;
• 2 posts a day on each channel you use, for example founder posts on LinkedIn;
• 20 helpful answers a day across 5 communities where your buyers talk, with the link only where the rules allow it and someone asked.

That is the volume that makes 4 weeks possible. AI drafts the messages and the posts; you add the true line about each person and write the community answers yourself. It is the same 3 hours a day as in the landing page guide, and it brings traffic to a new site for free.

While it runs:

• Do not edit either version, the price or the offer. If something is broken, fix it on both and restart the count.

• Do not start an email blast or a sale in the middle. A burst of friends or bargain hunters changes who arrives. If it happens, note the dates.

• Do not pause ads for one version only.

• Tool: your sending routine, and the end date set in your testing tool.

• Time: about 3 hours a day, every day, which you already spend sending.

• You will have: a test that reached its planned number in whole weeks, with nothing changed on the way.

• It worked if: 100 people who fit got the link every day, and nobody opened the results to decide early.

• Common mistake: stopping on day 4 because version B has 4 signups and A has 1. With numbers that small, the next 4 days can flip it.

6. Read the result once, keep the winner and start the next test

Every Sunday, write each version's counts in one sheet, to see how close each one is to its planned number. When both reach it, check the result once with a free significance test. Keep the winner, or keep the simpler version if there is no clear winner, and start the next test.

The Sunday count is for the totals only. Do not compare the two versions until both reach the planned number.

Test | Version | Started | Visits or messages | Actions (signups, calls, orders) | Replies | Actions per 100 | Notes (dates of sales, blasts, fixes)

When both versions reach the planned number, open Evan Miller's free chi-squared test (evanmiller.org/ab-testing/chi-squared.html). For each version, type the actions in "# successes" and the visits or messages in "# trials". Keep the confidence level at 95%. Read the "Verdict":

• Yes: the difference is very unlikely to be luck. Keep the version with more actions per 100. Put it on the page for everyone, end the test, and delete or redirect version B.
• No: you cannot tell them apart at this traffic. Keep the version you like better, usually the simpler or clearer one. Next time, test something bigger.
• The winner brought more signups but they do not use the product or pay: it promised something the product does not give. Keep the old version and look at why people do not buy.

If you run a SaaS: before you call a winner, check the trials it brought in their first 14 days: did they use the main feature, and did they pay? Trials that turn into paying customers are what matter.

If you run a B2B company or an agency: count which booked calls turned into real deals, using where each deal came from. A version that brings more calls from people who will never buy has not won. Ask about it on the discovery call.

Then write the next test plan from step 2, and start it on Monday. One test at a time keeps the counts clean. Once a month, look at all the tests together in your monthly marketing review. If the page still converts poorly after two clear tests, the problem is likely the whole page, not one part. Work through the fixes for a SaaS site with traffic but no signups, or a store with traffic but no sales.

• Tool: a Google Sheet and Evan Miller's chi-squared test. Both free.
• Time: 15 minutes every Sunday, and 30 minutes when a test ends.
• You will have: a log of every test with its numbers and its verdict, and the winning version live for everyone.
• It worked if: every test in the log reached its planned number before you checked, and has a "Yes" or "No" written next to it.
• Common mistake: calling 6 signups against 4 a win. Type them in: at 500 visits each, the verdict is "No".

Your landing page test checklist

0 of 11 done

Work through this list with an AI assistant

It drafts the work and keeps track of what is done.

Do this with ChatGPT(opens in a new tab)Do this with Claude(opens in a new tab)

Step 1 · Work out how many visits your A/B test needs

• Count the page's visits and main actions for the last 4 weeks.
• Put the numbers into the sample size calculator and work out the weeks the test needs.

Step 2 · Pick one big change and write down what you expect

• Watch 10 recordings and run a five-second test with 10 people who fit.
• Write version B and the one-line test plan.

Step 3 · Test the promise in your messages

• Make page B and tag both page links.
• Every daySend 100 personal messages, alternating version A and version B.

Step 4 · Set up the split with a free tool

• Set up the test in Shopify Rollouts or PostHog, and check both versions show and count.
• Follow Google's testing rules, and turn off Final URL expansion if you run search ads.

Step 5 · Run the test for whole weeks

• Every daySend 100 people who fit to the test, and leave both versions and the results alone.

Step 6 · Read the result once and keep the winner

• Every SundayFill in one row per version in the test sheet.
• After the planned numberRun the chi-squared check once, keep the winner, and start the next test.

Copy listDownloadReset