framework

Stream analytics in the enterprise: A look at Intels internal IoT implementation

December 10, 2019

Moty Fania shares Intels IT experience implementing an on-premises IoT platform for internal use cases. The platform was designed as a multitenant platform with built-in analytical capabilities and based on open source big data technologies and containers. Moty highlights the lessons learned from this journey with a thorough review of the platforms architecture.

The future of column-oriented data processing with Arrow and Parquet

December 9, 2019

In pursuit of speed, big data is evolving toward columnar execution. The solid foundation laid by Arrow and Parquet for a shared columnar representation across the ecosystem promises a great future. Julien Le Dem and Jacques Nadeau discuss the future of columnar and the hardware trends it takes advantage of, like RDMA, SSDs, and nonvolatile memory.

Chainer: A flexible and intuitive framework for complex neural networks

December 6, 2019

Open source software frameworks are the key for applying deep learning technologies. Orion Wolfe and Shohei Hido introduce Chainer, a Python-based standalone framework that enables users to intuitively implement many kinds of other models, including recurrent neural networks, with a lot of flexibility and comparable performance to GPUs.

Interactive learning systems: Why now and how?

December 6, 2019

Alekh Agarwal explains why interactive learning systems that go beyond the routine train/test paradigm of supervised machine learning are essential to the development of AI agents. Along the way, Alekh outlines the novel challenges that arise at both the systems and learning side of things in designing and implementing such systems.

Lessons learned from deploying the top deep learning frameworks in production

December 6, 2019

By building a marketplace for algorithms, Algorithmia gained unique experience with building and deploying machine-learning models using a wide variety of frameworks. Kenny Daniel shares the lessons Algorithmia learned through trial and error, the pros and cons of different deep learning frameworks, and the challenges involved with deploying them in production systems.

Making AI a reality for the enterprise and the physical world

December 5, 2019

Aman Naimat and Mark Patel present an analysis of the current adoption of AI in industry based on a systematic study of the entire business Internet at over 500,000 companies. Drawing on this data, Aman and Mark offer a new economic framework to discover, measure, and motivate future use cases for AI.

Frankenstein's microservices: How to avoid the monster

December 2, 2019

Many companies adopt microservices to break down monoliths, but they soon uncover a hidden cost: How do you manage all these new interconnected things popping up? Michael Hamrah explains how to avoid creating Frankenstein's monster by understanding elements of a microservice platform. . .so you can sleep at night.

Better bash: Unit and integration testing

November 27, 2019

C.J. Jameson explains how to test all the things, even your bash scripts. Using the BATS framework, it's cheap to write both high- and low-level tests to drive out more modular and readable bash scripts.

Building and operating containerized applications at scale (Day 1)

November 27, 2019

The advantages of containerized applications are increasingly recognized. Michael Hausenblas provides a gentle introduction to building and operating containerized applications at scale. The first day focuses on the basics of building app using containers; the second day expands this knowledge, focusing on the operations (monitoring, upgrades, etc.) of these apps.

Container orchestration wars

November 27, 2019

The orchestration space is fast moving and full of competing products, platforms, and frameworks. How do you choose the right one for your requirements? Karl Isenberg explores the features of several container orchestrators, breaking down the feature sets and characteristics into categories and scoring multiple solutionsincluding Kubernetes, Marathon, and Docker Swarmagainst each other.

Effectively adding analytics to your monitoring

November 26, 2019

Effective monitoring for todays agile environments is both science and art. (Analytics can provide the science while experts and business context can provide the art.) There is no perfect solution, but a framework for integrating these varied information sources as collaborators can drive continuous improvement. Elizabeth Nichols highlights (anonymized) examples from real environments.

My cloud makes pretty pictures

November 24, 2019

Data is beautiful when made visible. Matthew Brender and Raj Dutt offer a demo of Snap, an open telemetry framework designed to gather an increasingly diverse amount of measurements from the cloud, and illustrate how to visualize data in Grafana with unprecedented ease.

Troubleshooting without losing common ground

November 22, 2019

Common ground, an important concept in recent teamwork research, helps us understand why collaborative troubleshooting breaks down over time, leading to wasted effort and mistakes. Drawing on common ground as well as some ideas from medical diagnosis, Dan Slimmon demonstrates that by extending ChatOps, we can make troubleshooting much easier without losing the benefits of fluid team conversation.

Breaking Spark: Top five mistakes to avoid when using Apache Spark in production

November 20, 2019

Spark has been growing in deployments for the past year. The increasing amount of data being analyzed and processed through the framework is massive and continues to push the boundaries of the engine. Drawing on his experiences across 150+ production deployments, Neelesh Srinivas Salian explores common issues observed in a cluster environment setup with Apache Spark.

Bringing big data and design to policy making

November 19, 2019

Cat Drew explains how the UK's Policy Lab and GDS data teams are bringing more of a data, digital, and design approach to policy making, showcasing some of the Policy Lab projects that have used ethnography and data science to create fresh insight to change the way we think about policy problems.

Data wants to be shareable

November 19, 2019

Data has more potential value when it can be shared. In order to monetize data, it must first be made shareable: shareable data is an asset that can be sold, traded, or used to create new data marketplaces. Mona Vernon outlines a framework to structure thinking about data shareability and monetization and explores these new business opportunities.

Stream analytics in the enterprise: A look at Intels internal IoT implementation

November 17, 2019

Moty Fania shares Intels IT experience implementing an on-premises IoT platform for internal use cases. The platform was based on open source big data technologies and containers and was designed as a multitenant platform with built-in analytical capabilities. Moty highlights the key lessons learned from this journey and offers a thorough review of the platforms architecture.

Blocks in containers: Lessons learned from containerizing Minecraft

November 15, 2019

Its clear containers are here to stay for building reliable, scalable applications. But what about applications developed prior to the advent of containerization? Using Minecraft as her example, Julia Ferraioli explains why retrofitting technology for containers poses conceptual and practical challenges that require an approach different than starting container native.

Build your child their very own Node.js Frozen bot

November 14, 2019

Gabrielle Crevecoeur dives into the world of NodeBots. Using Node.js, the Johnny-Five framework, and an Arduino, Gabrielle walks participants through creating a bot, step by step. Although the NodeBot you build will respond to voice recognition, you'll leave able to create NodeBots with any functionality you like.

Lessons from writing HTraced: A high-performance network daemon in Go

November 8, 2019

HTraced is a new trace collection daemon for the Apache HTrace project, written in Golang. Colin McCabe shares his experiences with the Go programming language and explains the design of HTraced. Colin explores libraries, frameworks, and build systems and discusses what was fun, what was tricky, and how the team optimized overall system performance.

Migrate your traditional VM-based clusters to containers

November 8, 2019

Arun Gupta explains how to migrate a traditional VM-based cluster to a container-based infrastructure. You'll learn the gotchas of Docker Cloud, Kubernetes on Google/Amazon, OpenShift, and Mesos + Marathon.

Open sourcing the lean startup methodology

November 7, 2019

Kishau Rogers proposes a framework for launching enterprise-wide innovation centers that use open source technology in all aspects of business innovation, from rapid prototype development and open collaboration to customer discovery, while following a roadmap for commercial product development.

Practical performance tips to make your cross-platform mobile apps faster

November 6, 2019

Apache Cordova is one of the most popular frameworks for cross-platform mobile development. To build Cordova apps that perform well, its important to understand how to use the technologies in the most efficient ways. Doris Chen outlines what impacts "native performance," demonstrates how to measure mobile app performance, and shares practical tips for building faster Cordova apps.

Scalable bidirectional communication over the Web

November 5, 2019

Recent advances in transports and frameworks have made it possible to deploy large-scale, low-latency bidirectional communication over HTTP without losing original benefits like scalability and safety. Wenbo Zhu offers an overview of the use cases for enabling bidirectional communication as well as a demo that runs a distributed, shared whiteboard application from a browser.

Stop writing JavaScript frameworks

November 5, 2019

JavaScript frameworks seem like death and taxesinevitable and unavoidable. But that's not the way it needs to be, and actually, it must stop. Joseph Gregorio outlines the Zero Framework Manifesto and discusses his implementation experience building and shipping framework-free applications for the past two years.

Designing a reactive data platform: Challenges, patterns, and antipatterns

November 1, 2019

Extracting value from data goes beyond robust ingestion pipelines and flexible storage. A solid data architecture addresses the needs of analysts and engineers by providing a simple, self-service ecosystem capable of handling any workload. Alex Silva discusses how the platform team at Pluralsight has been conquering these challenges by designing a platform using distributed, reactive services.

Let's make the pain visible

October 30, 2019

We've been trying to explain the pain of technical debt for generations, but we've never been able to measure it. What if we could measure the effects of learning curves, collaboration pain, and problems building up in the code? Janelle Klein outlines the Idea Flow Learning Framework, a strategy for measuring the friction in developer experience that introduces a whole new world of possibilities.

Let's not rewrite it all

October 30, 2019

Every engineer dreams at some point of rewriting a legacy system. Rewrites introduce bugs, duplicate effort, and delays. Architectural evolution or incremental rewrites can be an alternative, but only if done properly. Michelle Brush explores good and bad patterns for evolving architectures, offering a framework for deciding what approach to take.