Building a multitenant data processing and model inferencing platform with Kafka Streams

Each week 275 million people shop at Walmart, generating interaction and transaction data. Navinder Pal Singh Brar explains how the customer backbone team enables extraction, transformation, and storage of customer data to be served to other teams. At 5 billion events per day, the Kafka Streams cluster processes events from various channels and maintains a uniform identity of a customer.

Navinder Pal Singh Brar provides an overview of the architecture for data processing and triggering models, which is inbuilt for scalability and reliability. As a multitenant platform, each client’s models (such as bid models, fraud detection, and omnichannel reorder) may be interested in certain events, such as search, add to cart, transactions, etc., and whenever such an event is processed, the model interested in that particular event is triggered. Navinder details how an event lands into the system from Kafka, is processed and saved internally, and how the interested models are triggered on such events. Models use the internal persistent state (on RocksDB) for feature extraction and store their own model outputs in the platform, which can be used across teams as features. You’ll explore the architecture of the models, specifically ensuring fairness among the models, providing isolation and reusing features and inferences across models at the same time, dynamically updating global data (such as the product catalog) needed to run models on each node, customizing models to either trigger on each event or a as batch after frequent time intervals, implementing data archival and TTL policies and other features developed to save money, and advantages and limitations of the platform.