Boosting Deployment Safety at GitHub with eBPF Technology

The Unique Challenge of a Self-Hosted Platform

GitHub hosts its own source code on github.com, a decision that turns the company into its own most demanding customer. While this internal dogfooding ensures new features are tested under real conditions before reaching external users, it creates an unusual vulnerability: if github.com ever goes down, GitHub loses access to its own codebase. This circular dependency is a nightmare scenario—unable to deploy fixes because the very platform needed for deployment is unavailable.

Boosting Deployment Safety at GitHub with eBPF Technology — Source: github.blog

To mitigate this, GitHub maintains a mirror of its code for emergency patching and pre-built assets for quick rollbacks. However, this addresses only the most obvious dependency. More subtle circular dependencies can lurk inside deployment scripts themselves.

Uncovering Hidden Circular Dependencies

When designing a new host-based deployment system, the team wanted to ensure that deployment scripts wouldn't inadvertently create their own circular dependencies. For example, a deployment script might attempt to download a binary from GitHub—a request that would fail if GitHub were inaccessible. Or it could call an internal service that, in turn, tries to fetch an update from GitHub. These indirect dependencies are often overlooked.

The team turned to eBPF (extended Berkeley Packet Filter), a powerful Linux kernel technology that allows safe, programmatic inspection and modification of system calls. With eBPF, they could selectively monitor and block any network calls or file accesses made by deployment scripts that might lead to circular dependencies.

Types of Circular Dependencies Identified

Consider a hypothetical MySQL outage that prevents GitHub from serving release data. To fix the problem, a deploy script must apply a configuration change to the affected MySQL nodes. During this process, three types of circular dependencies can emerge:

Direct dependencies – The MySQL deploy script tries to download the latest release of an open source tool from GitHub. Because GitHub is down, the script cannot fetch the required binary and fails.
Hidden dependencies – The deploy script uses a servicing tool already on disk. That tool, however, checks GitHub for an update before running. If it cannot reach GitHub, it may hang or error out.
Transient dependencies – The script calls an internal API (e.g., a migrations service) that itself attempts to pull a binary from GitHub. The failure propagates back to the original script.

The Traditional Mitigation Approach and Its Shortcomings

Previously, each team owning stateful hosts was responsible for manually reviewing their deployment scripts to identify and eliminate circular dependencies. This process was error-prone and time-consuming. Many dependencies go unnoticed until an actual outage occurs—a lesson best avoided.

Moreover, the review approach cannot protect against third-party tools or scripts that silently introduce new dependencies. A simple update to a command-line utility could suddenly make it try to contact a remote server, breaking the deployment in a crisis.

How eBPF Provides a Robust Solution

eBPF allows GitHub to define fine-grained policies at the kernel level. The deployment scripts are sandboxed, and any system call that could create a circular dependency—such as making an HTTP request to a GitHub URL—is either blocked or logged. The team can whitelist safe operations (e.g., reading local configuration files) while denying potentially dangerous calls.

This approach shifts the burden from manual script review to runtime enforcement. Because eBPF programs run inside the kernel, they can inspect every network packet and file access without modifying the application code. The overhead is minimal, and the security guarantees are strong.

Additionally, eBPF provides observability. GitHub can monitor exactly what network connections and file accesses each deployment script makes. This data helps identify hidden dependencies even when no outage is happening, allowing teams to fix issues proactively.

Getting Started with eBPF for Deployment Safety

GitHub shared how other organizations can adopt similar techniques:

Start with existing eBPF tools – Tools like bpftrace and BCC provide high-level interfaces for writing eBPF programs without deep kernel knowledge.
Identify critical system calls – Focus on calls that could lead to external network access (connect, sendto) or file reads from potentially remote mounts.
Implement a policy engine – Define rules such as “block any DNS resolution for github.com during deployment” and enforce them with eBPF.
Test in a staging environment – Simulate failures and verify that the eBPF rules do not interfere with legitimate operations.

By leveraging eBPF, GitHub has made its deployment system more resilient against the very real risk of circular dependencies. The result is a safer, more reliable platform—both for its own engineers and for the millions of developers who depend on GitHub.com.

Tags: