As of February 2021, Amazon now lets you run A/B tests on some elements of your listing without any third-party tool.
Overview About Amazon A/B Testing
1. Where to access the tool?
Go to ‘Mange Experiments’ under the ‘Brands’ drop-down menu.
2. What can you test?
- Title
- A+ Content
- Main image
3. Duration of the test
You can choose from default periods of from 4 weeks to 10 weeks.
4. Eligibility
- Brand registered
- High-traffic ASINs that meet a minimum traffic threshold in recent weeks
5. You can test on two versions
6. Allocating testing traffic
Currently, there is no option for you to allocate a certain percentage of the ASIN’s traffic to each tested version.
7. Tested versions are shown to unique customers
Once a customer (using the same account) sees a version, he/she will see that version consistently throughout the test period, regardless of device type or other factors.
Visits to your page where a customer cannot be identified are not included in the sample size.
8. Renew the expired experiments
Currently, there is no option to renew the completed experiments so that you can save time from creating a new experiment from scratch.
9 Takeaways From My Two Experiments
- I tested: Title. Duration: 4 weeks
- Goal of my experiment was to identify which Title version would yield better CTR (Click Through Rate) for the listing.
1. What key data can you get from the Experiment result?
Conversion rate. No CTR (Click-Through-Rate data).
Is it a shortcoming of the feature? Yes. The first and foremost role of the Title is to motivate searchers to click on your listing (CTR), then for Ranking and Conversion rate.
Once the shopper clicks on a listing, what convince her to buy the product are photos, videos, copy, and reviews.
👉 So what I wanted to see, the difference in CTR, wasn’t shown in the Experiment result.
2. Does Title have a strong correlation with Conversion Rate?
As you can see in the Experiment result, the difference in Conversion rates of the two title versions is not significant. This is mainly because what decide conversion rate are not actually the Title but the elements mentioned above.
3. Another Experiment
In this Experiment, though the difference in Conversion rate is bigger, it’s not significant enough.
* Note that the actual Unit Session Percentage (Conversion rates) of these two ASINs are way higher than what shown in the Experiment result.
4. CTR is not shown in the experiment result
Though CTR isn’t shown in the results of the experiment, knowing which title generates a higher conversion rate is still helpful in deciding which one to choose; provided that the difference in conversion rate is significant enough.
5. Extend the Experiment period
If your data is not significant, you can extend the experiment period to improve the data confidence.
6. Short vs. Long Title
In both Experiments, the shorter title that is benefit/key feature focused outperformed the long and keyword-heavy title.
7. Does Estimate matter?
One-Year Impact Estimate shows the estimate of incremental sales in a year period based on the incremental sales that occur during the experiment, assuming the same impact persists for one year and does not take into account variable factors such as seasonality and price.
The Most Likely scenario has 50% probability of happening. And the Best Case is not likely to happen.
👉 You shouldn’t be bothered by this estimate.
8. How to test A+ content?
If you test A+ content, it’s better to test one element at a time and make sure that the two versions are different enough. Because if you test multiple elements of the A+ content, you will have a hard time identifying which element brought you higher performance in the test.
9. Which element is the most suitable for the test?
If Conversion rate is the only data that we can draw from the experiment result, I think A+ content will be the most suitable test object.
For testing Title and Main image, I will want to see CTR data in the test result.
Let me know what you think?
In 2022, Amazon released the coolest free SEO tool ever. Check it out.