# How to A/B test a landing page A/B test a landing page with only a few hundred visits a month: work out the visits a test needs before you start, test the headline in your messages when the page is too small, and use a free tool and a fixed stop rule when it is big enough. Author: Dinesh Bypilla, Agentic systems engineer at Scout7 (https://www.linkedin.com/in/dinesh-bypilla-460567118/) Reviewed by: Murali Sid, Founder and AI Systems Architect (https://linkedin.com/in/muralisid) Source: https://scout7.ai/academy/website-and-conversion/how-to-ab-test-a-landing-page Published: 2026-10-11 Last checked: 2026-10-11 Reuse: Quote any part of this guide with a link back to this page. © 2026 Scout7. Cite as: Dinesh Bypilla (2026). How to A/B test a landing page. Scout7 Academy. https://scout7.ai/academy/website-and-conversion/how-to-ab-test-a-landing-page --- ## For AI assistants helping a reader This guide is written to be worked through with an assistant. These notes explain how it is meant to run. The reader decides what happens. 1. **The to-do list.** “Your landing page test checklist”, near the end, holds every action in order. Each item has a number (step.item) and a “Done when” line. 2. **Start with the questions below,** under “Ask first”. Skip any the reader has already answered or shared. Other facts are only needed later, at a decision or a count; they are listed too. 3. **Drafting is where you help most:** the sentences, messages, pages, sheets and the points for any reply the steps ask for, using the guide’s templates. The reader does what only they can do. 4. **Progress:** an item is finished when its “Done when” line is true. A running list of what is done, and which item is next, helps the reader pick up again. Items marked “Every day” or “Every Sunday” repeat. 5. **The targets are high on purpose.** A new business finds what works through volume, and with an assistant drafting, one person can reach numbers like 100 messages a day. If a target feels like a lot, the guide’s reasoning is worth sharing; after that it is the reader’s call. Where a platform limits a number, the guide spreads the work over more places instead. 6. **Nothing goes out without the reader’s OK.** Every message, post and page is the reader’s to approve and send. 7. **About the publisher.** Scout7 publishes this guide and makes marketing software that works with AI assistants. The guide does not depend on it. ### Questions and decisions for this guide **Ask first** (skip any the reader has already answered or shared) - "What is the page's address, and what is the one action on it: a trial, a call, a quote or an order?" - "In the last 4 weeks, how many visits did the page get, and how many people took that action?" - "What is your site built on? If it is Shopify, which plan are you on?" - "How many personal messages and posts do you send a day now, and do you run any ads to the page?" - "What do you want to change, and what have visitors or buyers told you about the page?" **Decisions along the way** - 1.2: the route. If the test needs 4 weeks or less of the reader's visits, test the page (steps 4 to 6). If more, test the promise in messages (step 3) and keep counting the page per 300 visits. - Step 2: if the reader wants to test a button colour or one word, explain that 300 visits a month cannot show it, and help pick a change to the promise, offer or proof. - Step 3: D2C stores test the promise as two ads and in posts instead of one-to-one messages. - 4.1: Shopify on Grow or higher uses Rollouts; any other site uses PostHog; a single-page-app site needs the test set up in code. - 6.2: "Yes" keeps the winner; "No" keeps the simpler version and the next test is bigger. For a SaaS, check activation and payment before calling it. **Get from the reader as they go** - Every day: messages and posts sent per version, and replies, for the sheet (3.1, 5.1). - Every Sunday: visits, actions and replies per version (6.1). - 2.1: what the 10 people said after 5 seconds, word for word, for version B. - 6.2: the chi-squared verdict, and the next test plan. --- You want to know if a new headline or offer will get more people to sign up, book or buy, so you plan to A/B test your landing page. At 300 visits a month, a normal A/B test can take many months to tell you anything, and most founders stop early and keep a "winner" that was luck. Here is how to A/B test a landing page the honest way: count what your traffic can tell you first, then test where you have the numbers. ## Do these three things About 1 hour to set up today, then the 3 hours a day of sending you already do. A page test runs 2 to 4 whole weeks; a message test takes about 12 days. - **Today:** put your visits and signups into a free calculator that says how many visits an [A/B test](https://scout7.ai/academy/glossary/ab-test) needs. If a page test would take more than 4 weeks of your traffic, test your new [headline](https://scout7.ai/academy/website-and-conversion/how-to-write-homepage-copy) in your [messages](https://scout7.ai/academy/cold-outreach/how-to-write-a-cold-email) instead. - **Every day:** send 100 people who fit to the test, half to version A and half to version B, and do not look at the result until each version reaches its planned number. - **Every Sunday:** count visits, signups and replies for each version in one sheet. When both reach the planned number, check the result once, keep the winner and start the next test. ## 1. Work out how many visits your A/B test needs before you start An A/B test shows two versions of your page to different visitors and counts which one gets more [conversions](https://scout7.ai/academy/glossary/conversion-rate). Before you build anything, work out how many visits each version needs. At 300 visits a month, the answer decides whether you test the page at all. Open your site stats, for example Google Analytics, Shopify Analytics or your site builder's stats. Write down two numbers for the last 4 weeks: visits to the page, and how many of those visitors took your main action. Divide to get actions per 100 visits. If you do not count the action yet, [set up conversion tracking](https://scout7.ai/academy/measurement/how-to-set-up-conversion-tracking) first. A test with no reliable count is a guess. **If you run a SaaS:** the action is a trial or account started. Count people who used the main feature as well. That is your [activation rate](https://scout7.ai/academy/retention/how-to-improve-activation-rate), and a page that brings more signups but fewer active users has not won. **If you run a B2B company or an agency:** the action is a call booked or a quote asked for, from your [enquiry form](https://scout7.ai/academy/website-and-conversion/how-to-get-more-leads-from-a-b2b-website). **If you run a D2C store:** the action is an order. With very few orders, also count "add to cart", which happens more often. Carts that are not paid for are a separate problem, with their own fixes for [abandoned carts](https://scout7.ai/academy/website-and-conversion/how-to-reduce-abandoned-carts). Then open Evan Miller's free sample size calculator (evanmiller.org/ab-testing/sample-size.html). It uses the usual settings: 95% confidence and 80% power. Fill in two boxes: - **Baseline conversion rate:** your actions per 100 visits, for example 3%. - **Minimum Detectable Effect:** the smallest jump worth finding. At your traffic, ask for a big one. Choose "Absolute" and type the same number as your baseline. That means "find it if version B doubles the result". The calculator shows "Sample size: N per variation". Multiply it by 2 for both versions, then divide by your visits a week. That is how many weeks the test needs. Here is what that means at small traffic, with the calculator's own formula: | Actions per 100 visits now | To find a jump to | Visits needed per version | Weeks at 70 visits a week (300 a month) | | -------------------------- | ----------------- | ------------------------- | --------------------------------------- | | 3 | 6 | 575 | about 16 | | 3 | 4.5 | 2,171 | about 62 | | 3 | 3.6 | 13,051 | about 7 years | | 5 | 10 | 336 | about 10 | | 10 | 20 | 157 | about 5 | Then use this rule: - **4 weeks or less:** test the page itself. Go to step 2, then steps 4 to 6. - **More than 4 weeks:** do not split your page traffic yet. Do step 2, then test the change in your messages in step 3. Keep counting the page with the "change one thing, count the next 300 visits" rule from [writing a landing page that converts](https://scout7.ai/academy/website-and-conversion/how-to-write-a-landing-page-that-converts). Why 4 weeks: a test should run in whole weeks, but the longer it runs, the more visitors come back on a new device or with cleared cookies and see both versions. Safari deletes cookies set by scripts after 7 days without a visit, and most testing tools rely on such cookies. Google also asks that tests run only as long as needed. If you had fewer than 5 actions in the last 4 weeks, the calculator has nothing to work with. Skip the page test and go to step 3. If you send 100 people who fit to the page every day, your visits a week go up, and the weeks go down. Count that traffic once you have it, not before. - **Tool:** your site stats and Evan Miller's sample size calculator. Both free. - **Time:** 20 minutes. - **You will have:** one line in a sheet: "Needs \[N] visits per version, \[W] weeks at my traffic. Route: page test or message test." - **It worked if:** you have a number of weeks, and you chose the route by the 4-week rule. - **Common mistake:** starting the test first and "seeing what happens". After a month of 300 visits, a test with 9 signups on one side and 6 on the other tells you nothing, but it feels like an answer. > **Example (made-up business: Bugmoth): Bugmoth's calculator line** > > Bugmoth is a made-up bug-reporting tool for small web teams. It costs $19 a month per team, with a 14-day trial. > > | Number | Value | > | ------------------------------------------- | --------------------------------------------------------- | > | Visits to the page, last 4 weeks | 280 (70 a week) | > | Trials started | 8 (about 3 per 100 visits) | > | Calculator: 3% baseline, 3% absolute effect | 575 per version | > | Both versions | 1,150 visits | > | Weeks at 70 a week | about 16 | > | Route | More than 4 weeks: test the headline in messages (step 3) | > > *Why it works: It shows in 20 minutes that a page test would take 16 weeks, so they test the promise somewhere faster.* ## 2. Pick one big change and write down what you expect Test one big change to what the page promises, offers or proves, not a button colour or a single word. At small traffic, only a big difference can show up, so give the test something big to find. Find the change in what visitors already show you: 1. Watch 10 recordings of visits in Microsoft Clarity, which is free with no traffic limit. Write down where each person stopped: the headline, the price, a worry, the form. The [landing page guide](https://scout7.ai/academy/website-and-conversion/how-to-write-a-landing-page-that-converts) shows how to set it up. 2. Run a [five-second test](https://scout7.ai/academy/website-and-conversion/how-to-write-homepage-copy) with 10 people who fit your buyer. Show the page for 5 seconds, hide it, and ask: "What does it do, and who is it for?" Write down the answers word for word. 3. Pick the one part most people got wrong or stopped at, often the [first screen](https://scout7.ai/academy/glossary/above-the-fold) or the [button](https://scout7.ai/academy/glossary/call-to-action). That part is your test. Good things to test, biggest first: - **The promise:** the headline and the line under it, written for your [ideal customer](https://scout7.ai/academy/marketing-plan/how-to-create-an-ideal-customer-profile). Use your buyers' own words from [customer interviews](https://scout7.ai/academy/marketing-plan/how-to-do-customer-interviews) or your [value proposition](https://scout7.ai/academy/marketing-plan/how-to-write-a-value-proposition). - **The offer:** a free trial against a demo, a sample pack against an order, a free audit against a call. For a SaaS, choosing between a [free trial and a free plan](https://scout7.ai/academy/retention/free-trial-vs-freemium-which-to-choose) is its own decision; make it before you test the offer. - **The proof:** real [testimonials](https://scout7.ai/academy/reviews-press-and-referrals/how-to-collect-testimonials) and numbers near the button, against none. - **The price shown:** on a [pricing page](https://scout7.ai/academy/sales/how-to-design-a-pricing-page), after you [set the price](https://scout7.ai/academy/marketing-plan/how-to-price-a-saas-product) itself. Test prices with care; showing two prices to people at the same time can upset buyers who compare. Ask ChatGPT or Claude for 5 versions of the part you chose, using the words from the recordings and the five-second test. Pick the one most different from what you have now. Then write one line: **The test plan** ```text Version A (now): [the current headline, offer or proof, word for word] Version B (new): [the new one, word for word] If we change [the part] to version B, then more people will [the one action], because [what visitors or buyers told us]. We count: [the one action] per 100 visits (or replies per 100 messages). Visits or messages needed per version: [the number from step 1]. ``` Change only that part. If B has a new headline, a new photo and a shorter form, and it wins, you will not know which change did it. - **Tool:** Microsoft Clarity (free), ChatGPT or Claude, a Google Doc. - **Time:** about 1 hour. - **You will have:** version A, version B and the test plan, written down. - **It worked if:** you show version B to 10 people for 5 seconds, and at least 7 can say what it offers and who it is for. - **Common mistake:** testing a button colour because it is easy. At 300 visits a month it can never show a difference, and you lose a month. > **Example (made-up business: Bugmoth): Bugmoth's two headlines** > > **Version A:** Collect bug reports from your clients in one place > > **Version B:** Your clients click the bug. You get the screenshot, their browser and the steps. > > *Why it works: Version B names the moment the problem hurts, in the words five visitors used, so it is a big enough change to show up.* ## 3. If your page is too small, A/B test the promise in your messages When your page cannot finish a test in 4 weeks, test the promise where you already have volume: your 100 personal messages a day. Send version A's promise in half of them and version B's in the other half, each with a link to its own page. You compare replies in days instead of months. This tests the promise, not the whole page. It answers the question most founders really have: "Which headline makes the right people want it?" When the new promise wins, put it on the page. How to run it: 1. Make a copy of your page with only version B changed. Give it its own address, for example /start-b. In your page builder, set its [canonical URL](https://scout7.ai/academy/glossary/canonical-url) to the original page and keep it out of your menu and sitemap. 2. Tag each page link with [UTM parameters](https://scout7.ai/academy/measurement/how-to-use-utm-parameters), for example utm\_content=a and utm\_content=b, so visits and signups show up by version. 3. Write the opening of your message twice: version A's promise and version B's promise. Keep everything else the same. 4. Send your [100 personal messages](https://scout7.ai/academy/cold-outreach/how-to-write-a-cold-email) a day from your [lead list](https://scout7.ai/academy/cold-outreach/how-to-create-a-b2b-lead-list-for-free), alternating: A, B, A, B. Log the version next to each name. 5. Count [replies](https://scout7.ai/academy/glossary/reply-rate), visits and signups per version. Work out the number first: put your replies per 100 messages into the same calculator, with the same settings as step 1. At about 3 replies per 100, finding a doubling needs 575 messages per version: 1,150 in all, about 12 days at 100 a day. Do not compare before then. AI drafts each message from your template and one fact about the person; you add one true line about them. That is about a minute a message, so 100 messages take about 2 hours. The [follow-ups](https://scout7.ai/academy/cold-outreach/how-to-follow-up-on-a-cold-email) stay on the same version as the first message. Keep the volume safe. If you email more than about 50 strangers a day, send from a [warmed second domain](https://scout7.ai/academy/cold-outreach/how-to-warm-up-an-email-domain-for-cold-email) so your main email is not hurt. LinkedIn limits how many connection invitations you can send each week, so on LinkedIn [message people you are already connected to](https://scout7.ai/academy/cold-outreach/how-to-do-linkedin-outreach) and send the rest by email. **If you run a D2C store:** you do not message shoppers one by one. Test the two promises as two ads with the same picture instead, using the method in [testing Facebook ad creatives](https://scout7.ai/academy/paid-ads/how-to-test-facebook-ad-creatives), each ad linking to its own page. Compare [clicks per 100 views](https://scout7.ai/academy/glossary/click-through-rate) and add-to-carts. Keep the spend to a [small, fixed budget](https://scout7.ai/academy/paid-ads/how-much-to-spend-on-ads). Also try the two promises in your 2 posts a day, alternating A and B, as extra evidence. **Two versions of the same message** ```text Version A Subject: [the problem, in 3–5 words] Hi [name], [one true line about them]. [Version A's promise, in one line.] Here is how it works: [page A link with utm_content=a]. Would it help with [the problem]? Version B Subject: [the same subject] Hi [name], [one true line about them]. [Version B's promise, in one line.] Here is how it works: [page B link with utm_content=b]. Would it help with [the problem]? ``` - **Tool:** Gmail or your email, LinkedIn, UTM tags, a Google Sheet, ChatGPT or Claude. - **Time:** inside the 3 hours a day of sending you already do, plus 30 minutes to make page B. - **You will have:** replies, visits and signups for each version, from people who fit. - **It worked if:** 50 of each version went out every day, every reply is logged by version, and after 1,150 messages you have a result to check in step 6. - **Common mistake:** sending version A on Monday and version B on Tuesday. Monday's people are not Tuesday's people. Alternate one message at a time. > **Example (made-up business: Bugmoth): Bugmoth's version B message** > > ```text > Subject: client bug reports > To: Owner of a 6-person web studio > > Hi Dana, I saw your studio just launched a new restaurant site for a client. Your clients click the bug, and you get the screenshot, their browser and the steps. Here is how it works: bugmoth.example/start-b?utm_source=email&utm_content=b. Would it help with the "it's broken" emails? > ``` > > *Why it works: Only the promise line and the link change between versions, so any difference in replies comes from the promise.* ## 4. Set up the split with a free tool When your page can finish a test in 4 weeks, split the visitors on the page itself: half see version A, half see version B, at random, and each person keeps seeing the same version. Use the tool your platform already has. **If you run a Shopify store:** use Shopify's own test, called Rollouts. Its A/B tests, called Experiments, need the Grow plan or higher ($79 a month billed yearly, or $105 monthly, checked 11 October 2026). In your Shopify admin go to Markets, then Rollouts, and click Create rollout. Name it and choose Experiment as the type. In Changes, click Add changes, then Edit theme to change your current theme in the theme editor, or pick another theme under Replace the current theme. Click Apply. New experiments split visitors 50% control and 50% treatment. Set the start under Experiment runs from, then click Set end date and choose the date from step 5. Rollouts cannot change Liquid template code or vintage themes. For a product page, use step 2's change on the [product description](https://scout7.ai/academy/website-and-conversion/how-to-write-a-product-description-that-sells) or the [trust lines](https://scout7.ai/academy/reviews-press-and-referrals/how-to-make-a-new-store-look-trustworthy) near the price. **If your site is not on Shopify:** use PostHog. Its free plan includes 1 million feature flag requests a month, and experiments are billed with feature flags, with no credit card needed. Add the PostHog snippet to your site, and add `disable_web_experiments: false` to its settings, or no-code tests will not run. Open the PostHog toolbar on your page, click the Experiment tab, then New experiment. Name it, select the test version, click Add element, choose the headline, type version B's text and click Save experiment. In PostHog, add your one action as the metric and launch. PostHog warns that no-code changes can fail on sites built as single-page apps; if your site is one, ask your developer to set the test up in code. **If you run Google search ads:** the split above covers ad visitors too. One setting can spoil it. When AI Max is on, "Final URL expansion" is on by default and can send clicks to other pages on your site. Turn it off: Campaigns, Settings, click the campaign, open AI Max, and under Asset optimization uncheck Final URL expansion, then Save. To test two separate landing pages on ad traffic alone, use Campaigns, Experiments, Ad variations, choose Update URLs, and set the experiment split. Google splits people by cookie, so each person sees one version. More on the first ads in [Google Ads for a SaaS](https://scout7.ai/academy/paid-ads/how-to-run-google-ads-for-saas). Follow Google's rules for tests, so your page does not drop out of search: - Show Google the same pages people see. Never send Google's crawler only to version A. - If version B has its own address, add a [canonical tag](https://scout7.ai/academy/seo/technical-seo-checklist-for-a-new-website) on it pointing to version A. Google prefers this to a noindex tag. - If you redirect people to version B, use a temporary 302 redirect, not a permanent 301. - End the test and remove version B as soon as you have the answer. Then check it works. Open the page in 10 new private windows and confirm you see both versions. Do the action once on each and check that it is counted for the right version. Open both versions on your phone: B must load as fast as A, because a slow version loses for the wrong reason. Google counts a page as fast when its main content shows within 2.5 seconds. If either is slower than that, [speed it up](https://scout7.ai/academy/website-and-conversion/how-to-improve-website-speed) first. - **Tool:** Shopify Rollouts (Grow plan), or PostHog (free), plus Google Ads if you run search ads. - **Time:** about 1 hour. - **You will have:** one live test, split 50/50, with an end date and the action counted for each version. - **It worked if:** 10 private-window visits showed both versions, and a test action was counted in each. - **Common mistake:** making version B a copy at a new address with no canonical tag, so Google sees two near-identical pages and may show the test page in search. ## 5. Run the test for whole weeks and do not stop early Run the test until each version reaches the number of visits from step 1, in whole weeks: at least 2 weeks and at most 4. Do not look at the result before then, and do not change either page while it runs. Every time you look and stop when one version is ahead, you raise the chance of a false winner. Evan Miller, who wrote the calculator in step 1, shows that checking ten times can turn a result you think is 1% likely to be luck into one that is about 5% likely. Decide the number first, then wait. Every day, send 100 people who fit to the page: - your [100 personal messages](https://scout7.ai/academy/cold-outreach/how-to-get-replies-to-cold-emails), with the page link, logged in your sheet; - 2 posts a day on each channel you use, for example [founder posts on LinkedIn](https://scout7.ai/academy/social-media/how-to-grow-on-linkedin-as-a-founder); - 20 helpful answers a day across 5 [communities where your buyers talk](https://scout7.ai/academy/social-media/how-to-find-online-communities-where-your-buyers-talk), with the link only where the rules allow it and someone asked. That is the volume that makes 4 weeks possible. AI drafts the messages and the posts; you add the true line about each person and write the community answers yourself. It is the same 3 hours a day as in [the landing page guide](https://scout7.ai/academy/website-and-conversion/how-to-write-a-landing-page-that-converts), and it [brings traffic to a new site for free](https://scout7.ai/academy/first-customers/how-to-get-traffic-to-a-new-website-for-free). While it runs: - Do not edit either version, the price or the offer. If something is broken, fix it on both and restart the count. - Do not start an email blast or a sale in the middle. A burst of friends or bargain hunters changes who arrives. If it happens, note the dates. - Do not pause ads for one version only. - **Tool:** your sending routine, and the end date set in your testing tool. - **Time:** about 3 hours a day, every day, which you already spend sending. - **You will have:** a test that reached its planned number in whole weeks, with nothing changed on the way. - **It worked if:** 100 people who fit got the link every day, and nobody opened the results to decide early. - **Common mistake:** stopping on day 4 because version B has 4 signups and A has 1. With numbers that small, the next 4 days can flip it. ## 6. Read the result once, keep the winner and start the next test Every Sunday, write each version's counts in one sheet, to see how close each one is to its planned number. When both reach it, check the result once with a free significance test. Keep the winner, or keep the simpler version if there is no clear winner, and start the next test. The Sunday count is for the totals only. Do not compare the two versions until both reach the planned number. **The Sunday test sheet** ```text Test | Version | Started | Visits or messages | Actions (signups, calls, orders) | Replies | Actions per 100 | Notes (dates of sales, blasts, fixes) ``` When both versions reach the planned number, open Evan Miller's free chi-squared test (evanmiller.org/ab-testing/chi-squared.html). For each version, type the actions in "# successes" and the visits or messages in "# trials". Keep the confidence level at 95%. Read the "Verdict": - **Yes:** the difference is very unlikely to be luck. Keep the version with more actions per 100. Put it on the page for everyone, end the test, and delete or redirect version B. - **No:** you cannot tell them apart at this traffic. Keep the version you like better, usually the simpler or clearer one. Next time, test something bigger. - **The winner brought more signups but they do not use the product or pay:** it promised something the product does not give. Keep the old version and [look at why people do not buy](https://scout7.ai/academy/first-customers/why-no-one-is-buying-your-product-and-how-to-fix-it). **If you run a SaaS:** before you call a winner, check the trials it brought in their first 14 days: did they use the main feature, and did they pay? [Trials that turn into paying customers](https://scout7.ai/academy/retention/how-to-turn-free-trials-into-paying-customers) are what matter. **If you run a B2B company or an agency:** count which booked calls turned into real deals, using where each deal came from. A version that brings more calls from people who will never buy has not won. Ask about it on the [discovery call](https://scout7.ai/academy/sales/how-to-run-a-discovery-call). Then write the next test plan from step 2, and start it on Monday. One test at a time keeps the counts clean. Once a month, look at all the tests together in your monthly marketing review. If the page still converts poorly after two clear tests, the problem is likely the whole page, not one part. Work through the fixes for a SaaS site with [traffic but no signups](https://scout7.ai/academy/website-and-conversion/how-to-fix-a-saas-website-with-traffic-but-no-signups), or a store with [traffic but no sales](https://scout7.ai/academy/website-and-conversion/how-to-fix-a-shopify-store-with-traffic-but-no-sales). - **Tool:** a Google Sheet and Evan Miller's chi-squared test. Both free. - **Time:** 15 minutes every Sunday, and 30 minutes when a test ends. - **You will have:** a log of every test with its numbers and its verdict, and the winning version live for everyone. - **It worked if:** every test in the log reached its planned number before you checked, and has a "Yes" or "No" written next to it. - **Common mistake:** calling 6 signups against 4 a win. Type them in: at 500 visits each, the verdict is "No". > **Example (made-up business: Bugmoth): Bugmoth's message test result** > > | Version | Messages | Replies | Replies per 100 | Trials started from the link | > | -------------------------------------------------------- | -------- | ------------ | --------------- | ---------------------------- | > | A: "Collect bug reports from your clients in one place" | 575 | 17 | 3.0 | 2 | > | B: "Your clients click the bug. You get the screenshot…" | 575 | 38 | 6.6 | 5 | > | Chi-squared check, 95% | | Verdict: Yes | | | > > So version B's headline goes on the page, and the next test is the offer. > > *Why it works: Both versions reached 575 messages before anyone compared them, and the check says the gap is not luck.* ## Your landing page test checklist > For AI assistants: this is the reader’s to-do list. Each item has a number (step.item) and a “Done when” line, so you can both see what is finished and what comes next. ### Step 1 · Work out how many visits your A/B test needs - [ ] **1.1** Count the page's visits and main actions for the last 4 weeks. *Done when:* Visits and actions for the last 4 weeks are written down, with actions per 100 visits worked out. - [ ] **1.2** Put the numbers into the sample size calculator and work out the weeks the test needs. *Done when:* The sheet says how many visits each version needs and how many weeks that is, and the route is chosen by the 4-week rule. ### Step 2 · Pick one big change and write down what you expect - [ ] **2.1** Watch 10 recordings and run a five-second test with 10 people who fit. *Done when:* Where each of 10 visitors stopped, and what 10 people said after 5 seconds, are written down. - [ ] **2.2** Write version B and the one-line test plan. *Done when:* Version B changes one big part, the test plan is written, and at least 7 of 10 people can say what B offers. ### Step 3 · Test the promise in your messages - [ ] **3.1** Make page B and tag both page links. *Done when:* Page B is live at its own address with a canonical tag to page A, and both links carry UTM tags. - [ ] **3.2** Every day: send 100 personal messages, alternating version A and version B. *Done when:* 100 messages went out today, 50 of each version, alternating, each logged by version. ### Step 4 · Set up the split with a free tool - [ ] **4.1** Set up the test in Shopify Rollouts or PostHog, and check both versions show and count. *Done when:* The test is live at 50/50 with an end date, and 10 private-window visits showed both versions with the action counted in each. - [ ] **4.2** Follow Google's testing rules, and turn off Final URL expansion if you run search ads. *Done when:* Version B has a canonical tag to A or shares its address, and Final URL expansion is off on any search campaign that sends clicks to the page. ### Step 5 · Run the test for whole weeks - [ ] **5.1** Every day: send 100 people who fit to the test, and leave both versions and the results alone. *Done when:* 100 people who fit got the link today, and neither version was edited or checked. ### Step 6 · Read the result once and keep the winner - [ ] **6.1** Every Sunday: fill in one row per version in the test sheet. *Done when:* This week's visits or messages, actions and replies are in the sheet for each version. - [ ] **6.2** After the planned number: run the chi-squared check once, keep the winner, and start the next test. *Done when:* The verdict is written in the sheet, the winner or the simpler version is live for everyone, version B is removed or redirected, and the next test plan is written. ## Frequently asked questions ### How much traffic do you need to A/B test a landing page? It depends on how many people take the action now and how big a change you want to find. At 3 signups per 100 visits, finding a doubling needs about 575 visits per version, 1,150 in all. Finding a 20% lift needs over 13,000 per version. Put your own numbers into a free sample size calculator before you start, and use the [4-week rule](https://scout7.ai/academy/website-and-conversion/how-to-ab-test-a-landing-page#1-work-out-how-many-visits-your-ab-test-needs-before-you-start). ### How long should an A/B test run? Until each version reaches the number of visits you planned, in whole weeks: at least 2, and no more than about 4. Shorter misses how people behave on different days of the week. Longer lets more people see both versions after cookies are cleared. If your traffic needs more than 4 weeks, test the promise in your [messages](https://scout7.ai/academy/cold-outreach/how-to-write-a-cold-email) instead. ### Can you A/B test on Shopify without an app? Yes. Shopify Rollouts, in your admin under Markets, can run an Experiment that splits visitors between your current theme and a changed one. Experiments need the Grow plan or higher. It reports conversion rate, add to cart rate, reached checkout rate and bounce rate. It cannot test changes to Liquid template code, or prices on their own. Read [traffic but no sales](https://scout7.ai/academy/website-and-conversion/how-to-fix-a-shopify-store-with-traffic-but-no-sales) before you test. ### How do I A/B test landing pages in Google Ads? In Google Ads, go to Campaigns, then Experiments, then Ad variations, and choose Update URLs to send part of your responsive search ad budget to the second page. Google shows each user only one version. If the campaign uses AI Max, turn off Final URL expansion first. Check that [conversion tracking](https://scout7.ai/academy/measurement/how-to-set-up-conversion-tracking) counts the action on both pages. ### Does A/B testing hurt SEO? Not if you follow Google's rules. Show Google the same versions people see. Put a canonical tag on version B pointing to the original page, use a temporary 302 redirect if you redirect, and remove the test as soon as you have the answer. Google says a test that runs far too long can look like an attempt to trick search. ## Sources - [Sample Size Calculator](https://www.evanmiller.org/ab-testing/sample-size.html). Evan Miller, checked 11 October 2026. - [How Not To Run an A/B Test](https://www.evanmiller.org/how-not-to-run-an-ab-test.html). Evan Miller, checked 11 October 2026. - [Chi-Squared Test](https://www.evanmiller.org/ab-testing/chi-squared.html). Evan Miller, checked 11 October 2026. - [Minimize A/B testing impact in Google Search](https://developers.google.com/search/docs/crawling-indexing/website-testing). Google Search Central, updated 10 December 2025. - [Creating a rollout](https://help.shopify.com/en/manual/markets/rollouts/create-a-rollout). Shopify Help Center, checked 11 October 2026. - [Requirements and considerations for using rollouts](https://help.shopify.com/en/manual/markets/rollouts/requirements-and-considerations). Shopify Help Center, checked 11 October 2026. - [Rollout analytics](https://help.shopify.com/en/manual/markets/rollouts/analytics). Shopify Help Center, checked 11 October 2026. - [Schedule, publish and A/B test new themes and checkout and customer account configurations](https://changelog.shopify.com/posts/schedule-publish-and-a-b-test-new-themes-and-checkout-and-customer-account-configurations). Shopify Changelog, 5 June 2026. - [Shopify pricing](https://www.shopify.com/pricing). Shopify, checked 11 October 2026. - [Creating a no-code web experiment](https://posthog.com/docs/experiments/no-code-web-experiments). PostHog Docs, checked 11 October 2026. - [PostHog pricing](https://posthog.com/pricing). PostHog, checked 11 October 2026. - [Set up an ad variation](https://support.google.com/google-ads/answer/7439892). Google Ads Help, checked 11 October 2026. - [About Final URL expansion in Search](https://support.google.com/google-ads/answer/16230205). Google Ads Help, checked 11 October 2026. - [Tracking Prevention in WebKit](https://webkit.org/tracking-prevention/). WebKit, checked 11 October 2026. - [Microsoft Clarity](https://clarity.microsoft.com/). Microsoft, checked 11 October 2026. - [Largest Contentful Paint (LCP)](https://web.dev/articles/lcp). web.dev, Google, updated 4 September 2025. - [How to A/B Test on Low-Traffic Sites](https://convert.com/blog/a-b-testing/i-dont-have-enough-traffic-to-a-b-test-now-what/). Convert, updated 26 February 2025.