The Great Cardinality Disasters of Our Time
Many Cloud Native tools generate Prometheus metrics; together they form a great combination to operate and monitor your infrastructure. But sometimes things go wrong: a quirk in the metric labels can …
| Talk Title | The Great Cardinality Disasters of Our Time |
| Speakers | Bryan Boreham (Director of Engineering, Weaveworks), Chris Marchbanks (Senior Software Engineer, Splunk) |
| Conference | KubeCon + CloudNativeCon North America |
| Conf Tag | |
| Location | San Diego, CA, USA |
| Date | Nov 15-21, 2019 |
| URL | Talk Page |
| Slides | Talk Slides |
| Video | |
Many Cloud Native tools generate Prometheus metrics; together they form a great combination to operate and monitor your infrastructure. But sometimes things go wrong: a quirk in the metric labels can make the volume of data explode, and, soon after, your Prometheus will explode too.Chris and Bryan will share their war-stories such as receiving 46,000 simultaneous alerts or squashing the source of 100kB label values. Then, they will provide top tips to avoid this happening to your tools in the future.
