Scaling Massive, Real-Time Data Pipelines with Go

Big data often means huge processing pipelines. Passing 10,000 messages a second between apps and datastores is tough. Add on filtering and parsing and youre bound to run into problems. At Pivotal Cl …

Big data often means huge processing pipelines. Passing 10,000 messages a second between apps and datastores is tough. Add on filtering and parsing and you’re bound to run into problems. At Pivotal Cloud Foundry, we have to handle several factors of that load each day, which means our programs need to be highly concurrent and very careful with state. Let’s look at some patterns and best practices that can be taken advantage of in order to handle this load.