Combining Millions of Products Into One Marketplace Using Computer Vision and NLP

ODSC - Open Data Science
4 min readMar 14, 2019


Editor’s note: Ali Vanderveld will present on this topic at ODSC East 2019! Check out her talk here.

ShopRunner is an e-commerce company that receives feeds of product data from over 140 different retailer partners, including large department stores and retailers that specialize in clothing, electronics, appliances, nutritional products, and more. We present this marketplace to our members using both a web-based platform and our new mobile app, District. In order to provide an excellent user experience, we need to have one easy-to-navigate method for classifying products that effectively combines all of them into one coherent shopping experience. As a part of that experience, we want to make it as easy as possible to drill down into very specific details like color, pattern, fabric type, neckline, et cetera.

From our retailer partners, we receive at least one image and multiple text fields (including name and description) for each product. One method to categorize everything could be to pass each product to a human labeler. However, this process obviously does not scale well to a large marketplace like ShopRunner, which has a continuously shifting catalog of millions of active products at any given time. Instead, we’re having machine learning do the work for us.

[Related article: Essential NLP Tools, Code, and Tips]

This situation raises a lot of questions, including:

  • How do we deal with the possibility that the data provided may be insufficient for machine learning?
  • How do we deal with new categories, where we do not yet have enough training data?
  • Should you use separate, simpler models for overall classification vs specific attributes, or a larger monolith that handles everything at once?
  • How do you best combine predictions from images and from text, given that some product attributes are better learned from one or the other?
  • How can we monitor model performance over time?

Classification with images and text

The ShopRunner taxonomy has several different levels, with increasing levels of specificity. For example, a product could be classified as “womens:clothing:bottoms:shorts:denim-shorts.” In this and in most cases, both the image and text are important. For example, in this case, an image-only model could confuse mens, womens, and unisex denim shorts. Similarly, an image-only model could confuse men’s vs women’s running shoes and slacks vs colored jeans. On the other hand, the text data that we receive tends to be far messier, more limited, and in some cases, either the image or the text could be corrupted.

For all of these reasons, we choose a combined model. This involves passing image data and text data through separate computer vision and natural language processing models to condense each down into an embedding vector, which are then combined and passed into a final modeling stage. With this methodology, we can further add in any additional information that we want, for example from behavioral data that quantifies which types of members have been purchasing each product.

[Related article: Create Your First Face Detector in Minutes Using Deep Learning]

Tagging auxiliary attributes

In a similar vein, combining images with text can help us automatically detect the sorts of finer details that are useful for guiding customers as they drill down into the details of our products. Some of these are better predicted from images, such as color, pattern, and sleeve length. However, some attributes are better predicted from text, such as material and petite vs plus sizing. These problems are particularly well-suited to multi-task models (see this excellent blog post from one of our team’s senior data scientists), where a single model optimizes to solve a series of related problems.

Come learn the details of how we do this and the lessons we learned along the way at ODSC East 2019 this April 30 to May 3 in my talk “Combining Millions of Products Into One Marketplace Using Computer Vision and NLP.”

— — — — — — — —

Read more data science articles on, including tutorials and guides from beginner to advanced levels! Subscribe to our weekly newsletter here and receive the latest news every Thursday.



ODSC - Open Data Science

Our passion is bringing thousands of the best and brightest data scientists together under one roof for an incredible learning and networking experience.