November 24, 2019

383 words 2 mins read

Improving user-merchant propensity modeling using neural collaborative filtering and wide and deep models on Spark BigDL at scale

Improving user-merchant propensity modeling using neural collaborative filtering and wide and deep models on Spark BigDL at scale

Sergey Ermolin and Suqiang Song demonstrate how to use Spark BigDL wide and deep and neural collaborative filtering (NCF) algorithms to predict a users probability of shopping at a particular offer merchant during a campaign period. Along the way, they compare the deep learning results with those obtained by MLlibs alternating least squares (ALS) approach.

Talk Title Improving user-merchant propensity modeling using neural collaborative filtering and wide and deep models on Spark BigDL at scale
Speakers Sergey Ermolin (Intel), Suqiang Song (Mastercard)
Conference Strata Data Conference
Conf Tag Big Data Expo
Location San Jose, California
Date March 6-8, 2018
URL Talk Page
Slides Talk Slides
Video

Constructing marketing campaigns, targeting them to specific retail customers, and evaluating campaign effectiveness is a perennial problem for merchants and data processors. One of the key parameters of such campaigns is a user’s propensity to shop at a specific merchant in the future. A traditional machine learning methods of solving the aforementioned problem can be broken down into four steps: This traditional approach requires extensive feature engineering and user-behavior analysis during a model’s creation and tuning. As such, it often involves creating handcrafted features and demands an intimate knowledge of the dataset. Sergey Ermolin and Suqiang Song demonstrate how to use Spark BigDL wide and deep and neural collaborative filtering (NCF) algorithms to predict a user’s probability of shopping at a particular offer merchant during a campaign period. Along the way, they compare the deep learning results with those obtained by MLlib’s alternating least squares (ALS) approach. The proposed approaches reduce feature engineering workload and perform better than traditional feature-based ALS as measured by precision and recall metrics. However, these convolutional networks require significantly larger computational resources than traditional approaches, hence the logical requirement for a distributed compute infrastructure such as Apache Spark and a scalable deep learning framework such as BigDL. Sergey and Suqiang share work based on a real-life dataset that covers 12 months of data, between 1 and 10 million distinct qualified consumers, between 2 and 20 billion distinct known transactions, and between 50 and 200 target merchants (offers or campaigns) for benchmarks. Using this dataset as an example, they offer a detailed overview of the merchant-user relationship, share an in-depth outline of the deep learning algorithms they used, and discuss compute resources required.

comments powered by Disqus