Finding a Needle in the Haystack
Expedia's mission is helping people go places through the power of technology. To accomplish this we run hundreds of microservices, but what happens when one or more services fail at the same time? To …
| Talk Title | Finding a Needle in the Haystack |
| Speakers | Ashish Aggarwal (Principal Software Engineer, Expedia Group), Shreya Sharma (Technical Product Manager, Expedia Inc.) |
| Conference | Open Source Summit North America |
| Conf Tag | |
| Location | Vancouver, BC, Canada |
| Date | Aug 27-31, 2018 |
| URL | Talk Page |
| Slides | Talk Slides |
| Video | |
Expedia’s mission is helping people go places through the power of technology. To accomplish this we run hundreds of microservices, but what happens when one or more services fail at the same time? To improve the observability of system and provide high quality of service, we realize a need to connect these failure points across the distributed topology to reduce mean time to know(MTTK) and resolve (MTTR). In this talk, we will present our journey of distributed tracing that started with Zipkin and ended in building our own solution in open source. We designed the system using Kafka (KStreams), Cassandra, ElasticSearch and MetricTank with the key focus on extensibility and OpenTracing principles. We will deep dive on distributed tracing, trending and statistical alerting components and demonstrate how containerizing these components on Kubernetes helps us in ingesting terabytes of tracing data per day.
