Landing Page A/B Tests: More Sales Can Mean Less Profit
A landing page can generate more orders and leave less money. Before I call it a winner, I want the contribution per visitor assigned to each version, the cost of the traffic and enough observation time for the orders to mature.
If your ecommerce brand spends at least $15,000 a month on Meta, a page test should answer a business decision: should this version receive more of the traffic? A better checkout percentage is useful evidence, but it is not the whole answer when discounts, basket mix or refunds change.
Write one test question before building
Choose one substantive hypothesis. For example: showing the delivered price and shipping terms earlier will help the right buyers decide. Define the main outcome, the assignment unit, the observation window and the cost you are willing to incur to learn.
Keep the offer and page change distinguishable. If the new page also cuts the price, adds a bundle and changes the audience, you are testing that combined experience. You cannot later attribute the entire difference to the headline.
Fix broken checkout steps first. The Shopify product-page diagnosis covers that work. A controlled test is for choosing between functioning alternatives, not for leaving a known defect in place just to collect a result.
Assign traffic consistently
Use a random assignment mechanism that keeps the same visitor in the same experience where practical. Record assignment before the page outcome you intend to measure. If you count only people whose new page successfully loads, you can hide a loading failure by removing it from the denominator.
A same-time split helps prevent a payday, promotion or weekday mix from becoming the explanation for the result. Two separately optimized campaigns are not automatically a randomized page test. Their traffic can differ. Record the assignment mechanism instead of assuming equal budgets create equal visitors.
NIST's experimental-design guidance explains how controlled factors and randomization help manage other influences. Applied here, decide in advance whether device, market or another important factor needs balanced assignment and separate reporting. Do not create dozens of tiny segments after seeing which one flatters the result.
Calculate what each assigned visitor leaves
This is a hypothetical illustration, not a statistically established result. Each page receives 1,000 assigned visitors at $1,000 of allocated media cost. Product costs, fulfillment, payment costs and an explicitly estimated return allowance are already included in the per-order contribution below. Fixed overhead and management costs are not.
| Metric | Page A | Page B |
|---|---|---|
| Assigned visitors | 1,000 | 1,000 |
| Orders | 40 | 50 |
| Order conversion rate | 4% | 5% |
| Contribution per order before media | $35 | $24 |
| Total contribution before media | $1,400 | $1,200 |
| Contribution per assigned visitor | $1.40 | $1.20 |
| Contribution after allocated media | $400 | $200 |
Page B converts 25% more often on a relative basis, yet produces $200 less contribution after the same media cost. The arithmetic is why I want the economic outcome beside the conversion rate. It does not establish that Page A will always win.
Use actual visitor-level order values and costs for the final analysis. The average contribution inputs here simplify the explanation. If one unusually large order changes the answer, inspect the distribution and uncertainty. Do not multiply a tiny early advantage across next year's entire budget.
Choose the decision rule before the first result
Write down the smallest improvement worth deploying and the uncertainty you can tolerate. Estimate the sample needed for that decision, then ask whether the account can supply it within a stable business window. NIST's sample-size guidance for proportions illustrates why effect size, error tolerance and power matter. Its proportion formula is not a ready-made sample calculation for revenue or contribution.
For a contribution outcome, use a method that accounts for its variability and your actual randomization unit. A visitor can place multiple orders; those orders should not become unrelated experimental participants. If you cannot support a precise comparison, call the result directional and state what additional evidence would change the decision.
Define emergency stops separately. A checkout failure, wrong price or broken event can justify stopping immediately. That is an operational incident, not an early statistical winner. Record it and repair the test before interpreting its commercial outcome.
Wait for the costs that change the answer
Give both groups the same conversion and return observation age. Show immature orders and estimated return allowances explicitly. If the new page changes expectations, its early conversion gain could arrive with a different refund pattern. Use the return-adjusted cohort worksheet to inspect that possibility.
My BFCM account case documents $544,397.42 in Meta spend for one US jewelry brand. The test above is a separate calculation for choosing a page; it is not a claimed result from that client.
If your page decisions are disconnected from the ad account's economics, look at my Meta ads audit and bring the account to me. We will define what the next test needs to prove before paying for the traffic.