eng.age

Server optimization at scale - Eliminating packet drops and improving capacity

Improving the performance of our technology for the benefit of our customers and their audiences is an ongoing course of action at Verizon Media. For example, over the past two years, our performance and kernel engineers have eliminated virtually all packet drops (over 98% removed), improved performance health checks on our edge servers by 50%, and increased server capacity by up to 40%.

Continue Reading

Video Encoding in the Cloud to Stream Large-Scale Live Events

As advertising dollars and viewers continue to move away from traditional television, content owners and broadcasters are looking to ad-supported live event streaming as a way to engage new audiences and grow revenue through over-the-top (OTT) streaming. OTT offers new distribution opportunities, which enable a publisher to air and monetize licensed content that previously had no place on broadcast television. But before a publisher can begin to stream live events, technical hurdles in the first step of the streaming video workflow must be considered:

Continue Reading

Using StackStorm to automate support for 20,000 servers

In the last three years, our network has supported hundreds of thousands of live-streamed sporting events, massive software downloads, billions of hours of streaming video content, and demands from thousands of web applications that all require real-time responsiveness and globally consistent performance. To support this massive growth, since 2015, our global network has grown to 71 Tbps, requiring thousands of new servers across our 135+ PoPs.

To keep up with the challenge of deploying a network footprint that is always growing and changing, our Operations teams developed an IT automation platform using StackStorm. It’s a giant leap forward that has transformed how our teams deploy, change, repair, and decommission server infrastructure, spanning more than 95% of the server population across our global network.

Continue Reading

Running a Multi-Tenant WAF at the Edge

Web Application Firewalls (WAFs) are a critical layer in modern web security, providing a website’s first line of defense against vulnerabilities. WAFs can be used to defend against and notify on attempted exploits, allowing for mitigations faster than organizations can patch vulnerable software. For a global CDN, this functionality must be implemented in a way that is sensitive to performance, providing response times on the order of milliseconds. When we first introduced a WAF engine to the VDMS stack three years ago, we selected the ModSecurity Rules Engine, which we found to be first-rate for individual WAF use cases. Furthermore, ModSecurity’s support of the OWASP Core Rule Set (CRS), powerful rule language, and API access to the HTTP traffic stream in real time offered significant flexibility.

Continue Reading

All Kinds of Connectivity

Our friends over at DE-CIX have recently published a white paper examining the benefits of using peering links, like those available via their Internet Exchange. In particular, the white paper highlights results from a study done by Adnan Ahmed and Zubair Shafiq at the University of Iowa along with our research team here at VDMS. The original study appeared in the 2017 proceedings of the IEEE International Conference on Networking Protocols (ICNP)

Continue Reading