How do you evolve your data infrastructure?

Developing data infrastructure is not trivial; neither is changing it. It takes effort and discipline to make changes that can affect your team. Neelesh Salian discusses how Stitch Fix's data platform team maintains and innovates its infrastructure for the company's data scientists.

Stitch Fix has come a long way, both as a company and as a data science–heavy team. Its business challenges its data teams in terms of scale and complexity. Neelesh Salian discusses how Stitch Fix’s data platform team maintains and innovates its infrastructure for the company’s data scientists and evolves the ecosystem as the business continues to expand. The team’s charter has always remained to build a self-service data platform for data scientists, empowering them to be full stack and responsible for their own workflows. To accomplish this goal, the team prioritizes impactful changes, invests time in prototyping and testing, and uses its own infrastructure to test and innovate. This enables the team to be autonomous as developers of data infrastructure while also focusing on the larger team mission. The team takes a microservice architecture–based approach to solving critical problems. Neelesh shares details about such approaches the team took to change the ecosystem: adapting for data lineage, changing the reading and writing interfaces to the data warehouse, and improving the execution of Spark workflows, for instance. These were larger cross-functional projects that required careful planning and execution. Join in to explore lessons the team learned during this evolution that have helped them grow and make better decisions for the future.