App icons, aside from your preview images and name, can be the deciding factor when a user opts to download your app or not. Despite taking up only a few hundred pixels, your icon holds immense power.
Last time, we covered app icon design fundamentals. Hopefully by now, you should have three to five eye-catching app icon options to choose from.
Now’s the hard part: you can only choose one. The icon you pick is going to be what you use for your app’s marketing, brand identity and more. Your choice will have far-reaching consequences, so it’s important to get it right.
Don’t just go with the icon your friends and family vote for. Pick the winner based on hard data. That’s where A/B testing comes in.
A/B Testing Your Way to a Winner
Android: Use Google Play’s New A/B Testing Feature
Android developers have it easy when it comes to A/B testing, thanks to Google Experiments. Announced at Google I/O 2015, developers can find a new Experiment option in their console that lets them test various attributes — including their icon.
Since you’re looking for a general winner, make a Global experiment. Later on, you can do more complex (and regionally optimized) tests, but at this stage launching your app with the best icon is first priority.
Name your experiment, select your audience mix, tick “Hi-res icon” as the attribute to test, and you’re on your way to seeing which icon resonates most with users.
Load all your icon options as variants, ensuring they’re within the dimensions allowed by Google. Make sure they’re all uploaded with identical graphical fidelity (e.g., if one uploads fuzzy because it’s smaller, that can affect your results). Now you’re ready to perform the experiment.
Add variants for each icon you made. Users in different test groups will see different icons, but that version is preserved while your experiment is active so you don’t need to worry about users getting confused if they revisit your app page.
After you feel you’ve collected enough data points, a winner should emerge. Congratulations, you’ve tested your way to your best app icon.
Don’t have an Android app? With a little more effort and creativity, you can execute a test for iOS (and other app stores) too. Read on to see how.
iOS & Other App Stores: Paid A/B Testing on Facebook
Facebook’s a great platform for testing app icons. Its large user base and granular targeting options, plus the fact it’s one of the top-ranking services on both web and mobile, makes it ideal for A/B testing.
a. Make Sure it’s a CPM Campaign
When you make your test campaign, ensure it’s set to Cost Per Impression (CPM). That means you’ll pay the same amount for a set number of impressions (people seeing your banner ad in their feeds). Choosing Cost Per Click (CPC) can skew results because ads that receive more clicks can appear more frequently in users’ feeds.
b. Make Simple Mobile Banner Ads
Within the campaign, you should be testing multiple app icon concepts with banner ads. It’s important to note that these banner ads should contain just enough context so the user knows what kind of app to expect — the rest of the banner design should focus solely on the icon.
This part is where you can decide which direction you want to take your design (e.g., symbols or mascots). As an example, you can run tests by altering icon background colors, in-icon shapes or mascots, and even tiny shadowing tweaks.
Does the cat move the needle for app installs with your target user base? It’s worth testing to find out.
As the data pours in, you’ll discover the best candidate. Go with the icon that’s getting the most clicks with your target user base.
That said, note that if you already have an app icon and you’re testing whether it’s worth changing to a new one, this approach still works. Just use the existing icon as your control and compare the new designs against it. If the old icon performs the best, no change is needed. Otherwise, go with the data-backed winner.
c. Target Different Consumer Segments
Facebook’s targeting is very robust, and you’re likely targeting a few segments with your mobile app. This is a great time to see if icon designs perform better (or worse) across different consumer groups that you’re hoping to target.
Some options for segmenting users are:
- Age
- Gender
- Geography (note: country- or state-level tends to work best)
- Psychopathics
- Industries
- Customer Revenue Size Potential
Refine your targeting with the above settings and you’ll be one step closer to finding the right app icon for the users you want. You might learn that your beauty product recommendations app performs better with females 18-25 when you include brighter colors or an image of makeup.
A quick note about sample sizes: when you’re refining your targeting, make sure the number of impressions you’re getting properly reflects your target population. You can see if you’re getting a statistically significant number of impressions using online calculators like this one.
d. Don’t Forget the “Off” Switch
Once your campaigns reach the sample size you want and you’re comparing roughly the same amount of impressions across each variation, make sure you pause the campaign.
Campaigns will run until a) they run out of budget or b) they’re manually flipped off. Don’t ruin your test by forgetting to stop the data.
A Few More Testing Tips
Experiment Boldly
If you want clear answers, your tests need to be bold. Test designs that are wildly different.
We reached out to Kongregate, a mobile games publisher, for some icon testing tips. “We’ve found the best app icon tests come from trying out completely unique variants from the original,” says Jeff Gurian, Sr. Director of Marketing at Kongregate. “Once we identify a winning image, we then test minor modifications until we arrive at what we consider an optimized winner. We often back-test the new designs against the original control icon to validate the results. This is especially important to do if the source mix or geographic make-up of the traffic is changing.”
Above is Kongregate’s icon performance testing results relative to the existing (control) icon. While there is often a clear winner (third test from the left), most tests resulted in some improvement relative to the control.
Jeff also mentioned that new icon designs can even lure people who previously passed over an older version. ”Our theory is the different or new icon could make people think it’s an entirely new game.” That’s the power of an app icon.
Keep Tests Laser-Focused
Don’t get carried away testing the nearly infinite amount of things you could change. Form a hypothesis and test changes based on that. Save tests for questions your team truly wants to answer.
If someone on your samurai RPG app team truly believes an icon with two katanas (instead of one) will have a massive uptick in downloads, test that specifically. Don’t be tempted to test your copy, title and icon in tandem.
Know Which Metrics Matter
Remember what you’re optimizing for beyond all else. If your app’s success is being measured in paying retained users and you’re only noticing an increase in app downloads, your app icon isn’t attracting the right kind of users. Optimize for the right metric — don’t just think in terms of installs.
While Google Experiments might not be able to attribute actions down to the activation or payment level, these are things to look out for in the future. Facebook testing, however, can deliver this level of tracking. We strongly suggest keeping this in mind when deciding how to test your icon.
Optimize against the metric that matters for your business.
Real Results: A/B Testing in Action
When testing our own app icon variations, the best-performing one got 15% more clicks. That’s up to 15% more users for your app (assuming they all convert) — a huge upside.
Our findings serve as evidence to a trend that’s widespread among all things digital — tweaking designs and testing can help get you more users. When you’re making your next app icon, make sure you give it the time and attention it deserves. Your users (and your business) will thank you for it.
Want More? App Annie Has It
Don’t miss a single app-related update — subscribe to our blog to get more free app industry reports, conference updates and guides. We love hearing your opinions and ideas (hey, sometimes we overlook some key tips), so share your comments and questions below or on Twitter.
