Highlighting the data science methodologies that support Game IQ game classification
With hundreds of thousands of gaming apps in the app stores, and thousands of new mobile games released each month, App Annie is looking at a huge number of games to be classified. On top of this, the high granularity of our taxonomy structure makes it quite time-consuming to manually figure out into which category each game should fit.
How can we cover the vast majority of gaming apps in global markets accurately, efficiently, and in a scalable manner? In this blog post, we will walk you through the data science model that we developed to automate the game tagging procedure. Learn more about Game IQ here.
What data science was used to build Game IQ?
Collecting features: from texts to metrics
Our machine learning model starts with the features that feed into the model. The final performance of the model will largely depend on the quality of the input features. So the question to ask is: what information from a game can help us classify the game into our taxonomy categories?
We dug through a wide range of data and features for each game to look for relevant signals. Eventually, we chose to use the following three types of features:
- Text features: app-related text information
- Metrics features: App Annie data on market estimations and app usage patterns
- Affinity features: cross-app affinity
Each of these features captures a different aspect of a game. Text features contain highly granular information about the contents and gameplay of apps; metrics features carry quantitative signals that segment apps on the higher levels of our taxonomy structure; and app affinities allow us to discover clusters of similar apps.
Hybrid model to boost accuracy
Because text, metrics, and affinity features are all pretty distinct from each other in format, we chose to build an individual model for each of the three types of features. We started by manually labeling thousands of games as a training dataset. Three individual models are then separately trained on this dataset, and can independently generate probabilities of a game being in each category. These probabilities are then fed into another model that is trained to output the final classifications. In data science, this is a type of ensemble learning method called model stacking, which is commonly used to boost the prediction accuracy of models by combining a set of strong but diverse models.
Confidence scores: avoid errors by human intervention
As with every machine learning model, our model will inevitably make mistakes on some occasions. Oftentimes, this occurs when the features of a game that are fed into the model simply don’t contain enough information to draw a solid conclusion of the game category. To deal with these scenarios, we developed a unique proprietary method to calculate a confidence score for each category prediction, so that we know whether the model is confident or not about each prediction. For those predictions with confidence scores below a cutoff value, we send the predictions to our manual tagging platform for further review. This flexible human-in-the-loop setup further ensures the quality of our classifications.
In addition to creating an efficient and scalable labeling solution, the data science model has the added benefit of reducing the wiggle room for subjectivity. With data, we are able to abstract away as much as possible the need for subjective judgement calls on many apps that had seemed otherwise ambiguous to classify.
One such app that stumped us initially was Coin Master by Moon Active. Coin Master is an incredibly successful game that incorporates many different core gaming loops, from building and attacking villages to chance-based slots mechanics.
Coin Master screenshots and gaming loops:
Predictably, this hybrid-genre build was difficult to classify when simply looking at the app screenshots, reading the app description notes, and playing the game. The data, however, told a less ambiguous story:
When comparing the relationship between the Revenue:Active Users and Revenue:Downloads ratios, Coin Master exhibited market performance characteristics typically displayed by apps in the Casino class segment. By incorporating our market estimates into our classification model, we had the unique advantage and benefit of removing (as much as possible) instances where subjective judgement calls had to be made.
How will data science improve Game IQ moving forward?
We will continue to improve our data science model moving forward. Feedback from our clients on the game categorization is extremely valuable to us! Your feedback is treated as data additions that are fed into our model, enabling us to continuously improve the reliability and accuracy of our predictions.
Who is the App Annie Data Science team behind Game IQ?
March 23, 2021Product Announcements