Evolution of Search Ranking at Thumbtack

Millions of customers use Thumbtack every year to find and book professionals and local businesses to help them care for their home. Customers can find professionals for around 500 categories of home services like House Cleaning, Plumbing, Fitness Training on Thumbtack. When a customer searches for professionals (pros) in their neighborhood in a specific category, we provide an ordered list of search results of pros in their neighborhood. At Thumbtack, search ranking refers to the problem of finding the most relevant pros for a customer’s search request and ordering them based on their relevance. The more relevant the pro, the easier it is for customers to find and book the right pros for their job. As a data-driven, consumer facing technology company with millions of end users, Thumbtack places a strong emphasis on improving the experience for our customers through controlled online experimentation (e.g A/B tests). Recently, we successfully A/B tested and productized an ensembled Deep Cross Network (DCN V2) for search ranking. This was the first time we productized a near state-of-the-art neural-network based machine learning (ML) model. In this talk, we share learnings from evolving experimentation maturity around a complex problem space like search ranking, while continuously delivering business impact in a consumer facing technology marketplace like ours. We will share our learnings around challenges involving experimentation processes & model complexity. We will walk through the feature space evolution, how we measured & controlled for position bias, as well as why we chose to productize the DCN V2 model.