Scalable Monitoring Using Prometheus with Apache Spark
As spark applications move to a containerized environment, there are many questions about how to best configure server systems in the container world. In this talk we will demonstrate a set of tools t …
| Talk Title | Scalable Monitoring Using Prometheus with Apache Spark |
| Speakers | Diane Feddema (Principal Software Engineer, Red Hat), Zak Hassan (Senior Software Engineer - AI/ML CoE, CTO Office, Red Hat Inc.) |
| Conference | KubeCon + CloudNativeCon Europe |
| Conf Tag | |
| Location | Copenhagen, Denmark |
| Date | Apr 30-May 4, 2018 |
| URL | Talk Page |
| Slides | Talk Slides |
| Video | |
As spark applications move to a containerized environment, there are many questions about how to best configure server systems in the container world. In this talk we will demonstrate a set of tools to better monitor performance and identify optimal configuration settings. We will demonstrate how Prometheus, a project that is now part of the Cloud Native Computing Foundation (CNCF), can be applied to monitor and archive system performance data in a containerized spark environment. In our examples, we will gather spark metric output through Prometheus and present the data with Grafana dashboards. We will use our examples to demonstrate how performance can be enhanced through different tuned configuration settings. Our demo will show how to configure settings across the cluster as well as within each node.
