Enhancing Deployment Safety at GitHub with eBPF Monitoring

The Circular Dependency Challenge at GitHub

GitHub operates on a unique principle: it hosts its own source code on github.com. While this dogfooding approach allows the team to test changes internally before rolling them out to users, it introduces a critical vulnerability. If github.com ever goes down, the very code needed to fix it becomes inaccessible. This creates a stark circular dependency: to deploy GitHub, one needs GitHub to be running. To mitigate this, GitHub maintains a mirror of its source code for forward fixes and built assets for rollback scenarios. However, this is only a partial solution—deployment scripts themselves can introduce new circular dependencies that compromise reliability.

Enhancing Deployment Safety at GitHub with eBPF Monitoring — Source: github.blog

Understanding Circular Dependencies in Deployments

Imagine a MySQL outage that prevents GitHub from serving release data. To resolve the incident, a configuration change must be deployed to the affected MySQL nodes via a script. During this process, three distinct types of circular dependencies can emerge.

Direct Dependencies

The deploy script might attempt to pull the latest release of an open source tool from GitHub. Since the outage blocks access to release data, the script cannot finish, stalling the entire deployment.

Hidden Dependencies

Even if a servicing tool exists locally, it may check GitHub for updates upon execution. If GitHub is unreachable, the tool could fail or hang, depending on its error handling—delaying or preventing the deployment.

Transient Dependencies

A deploy script might call an internal API, such as a migrations service, which in turn tries to fetch a new binary from GitHub. The failure cascades back to the original script, amplifying the outage’s impact.

The Traditional Approach and Its Limitations

Previously, GitHub relied on individual teams to manually review deployment scripts and identify circular dependencies. This process was error-prone and time-consuming, often missing subtle interactions. With the design of a new host-based deployment system, a more robust solution was needed—one that could detect and block these dependencies automatically at runtime.

How eBPF Provides a Solution

eBPF (extended Berkeley Packet Filter) is a kernel technology that allows safe, programmable inspection and modification of system behavior. GitHub engineers discovered they could use eBPF to selectively monitor and block network calls made by deployment scripts. This provides a fine-grained security layer that prevents scripts from accidentally creating direct, hidden, or transient dependencies on GitHub itself or other internal services during an incident.

Implementing eBPF for Deployment Safety

The implementation involves attaching eBPF programs to network hooks in the kernel. The programs inspect outgoing requests from deployment scripts, comparing them against a whitelist of allowed destinations. If a request targets an IP or domain that could create a circular dependency (e.g., github.com during an outage), it is either logged, blocked, or rerouted. The system also handles transient dependencies by tracing API calls across processes. Key steps include:

Defining a set of critical services that must remain reachable only through approved paths.
Writing eBPF programs that filter network traffic at the socket or system call level.
Integrating with the existing deployment pipeline to log and alert on blocked calls.
Testing the eBPF rules in staging environments to ensure they don’t break legitimate operations.

This approach shifts the responsibility from manual script review to automated enforcement, increasing deployment safety without requiring changes to every team’s scripts.

Key Takeaways

GitHub’s use of eBPF demonstrates how modern kernel tools can solve longstanding operational challenges. By proactively blocking circular dependencies during deployments, the platform becomes more resilient to outages. Teams can now deploy with confidence, knowing that their scripts won’t inadvertently amplify failures. For those interested in building similar protections, eBPF offers a powerful, low-overhead way to monitor and control system interactions—far beyond traditional network policies.

Tags: