Work in progress!
Please check back later for the comprehensive guide.
Always preview your test
A broken test completely invalidates the results, losing time and money.
Many of the issues outlined can be mitigated by previewing your test.
The in-app preview exists for the sake of editing and may not reflect the real website.
Please preview with the “Live preview” button.
Avoid ambiguous, dynamic elements or pages you plan to modify
Elements that move, dynamically update, or change structure have a higher risk of breaking tests (for example: dropdowns, carousels, lazy-loaded content, dynamic list of most recent or top products). Elements that are difficult to uniquely identify, such as buttons repeated multiple times on a page, also carry increased risk. If you choose to test these elements, use preview mode and thoroughly validate the experiment across screen sizes. Even with validation, these tests may stop working if the page structure changes. Any manual changes to a page or its layout can invalidate active tests. After making changes, always verify that the experiment is still functioning as intended. If targeting becomes unreliable, the test should be ended.
Prefer later funnel steps
It takes time to download A/B test data to your website. It occurs on the first page a user visits, and then is saved for the session. This means the second page onwards that a user visits within a session will have A/B test data instantly.
Avoid changing elements that affect more than UI/UX
Elements that convey pricing, legal disclosures, or user consent should not be included in experiments.
Editing form controls (for example, text inputs) is also discouraged due to their functional and behavioral implications.
The system applies best-effort safeguards to restrict experiments on these classes of elements, but ultimate responsibility remains with the experiment owner.
Never create tests with premium or non-public content
All content saved to an A/B test variation is public. Only the test participants will see the test apply on the page, but all images, text or customizations written to a test will be publicly accessible.
Take special care when editing HTML directly
This action fully replaces the selected element. Existing behavior, event listeners, and integrations will be lost. JavaScript and inline styles are not supported due to security concerns. Only a limited set of HTML tags and attributes are allowed. Large or complex markup may impact layout, performance, analytics, or tracking. Avoid using this on critical UI, forms, or navigation. Classes are safe and may be used assuming they already exist in the stylesheets of the page. Use only if you understand the impact and have thoroughly tested with the live preview.
Take note of confidence and uplift measurements
Real improvements take lots of traffic and conversions to prove. If applying a winning variant on a test with low confidence, we recommend continuing to watch conversion rates.
Reach out to support if you have any questions, concerns or suggestions.
A/B testing is a complex process. We’d be more than happy to take feedback or clarify things.
