Microsoft Brings eBPF to Windows
If you want to run code to provide observability, security or network functionality, running it in the kernel of your operating system gives you a lot of power because that kernel can see and control everything on the system.
That’s powerful, but potentially intrusive or dangerous if you get it wrong, whether that’s introducing a vulnerability or just slowing the system down.
If you’re looking for a way to take advantage of that kind of privileged context without the potential danger, eBPF is emerging as an alternative — and now it’s coming to Windows.
Not Just Networking
Originally eBPF stood for “extended Berkeley Packet Filter”, updating the open source networking tool that puts a packet filter in the Linux kernel for higher performance packet tracing (now often called cBPF for classic BPF).
But it’s now a generic mechanism for running many kinds of code safely in a privileged context by using a sandbox, with application monitoring, profiling and security workloads as well as networking, so it’s not really an acronym anymore.
That privileged context doesn’t even have to be an OS kernel, although it still tends to be, with eBPF being a more stable and secure alternative to kernel modules (on Linux) and device drivers (on Windows), where buggy or vulnerability code can compromise the entire system.
The usual implementation is a kernel-based virtual machine for low-level packet processing; on Linux that lets you change the behavior of the kernel without recompiling it, to load your own event-driven code that will be executed in response to “hooks” — triggers like network events, system calls, function entries and kernel tracepoints.
That code can run a variety of functions by using helper calls, and it can change what the kernel would usually do: an eBPF program might mean a network packet is dropped, a system call is refused or an event is recorded for tracing.
It’s still something of a specialized technique, but developers get to work in a relatively high-level language like C++, Go or Rust but get the impact of working directly with the kernel.
“One of the reasons Microsoft has been investing in eBPF is because we see the importance of making the data plane itself programmable and controllable by software,”
— Dave Maltz.
eBPF is used for high-performance networking and load balancing, delivering application-specific routing or Quality of Service, protection against denial of service attacks and enforcing container runtime security. Kubernetes networking security using Calico or Cilium is extremely popular because it provides visibility into HTTP traffic traditional security monitoring can’t see.
There are some interesting opportunities to use eBPF in the new open networking models that are being developed around the open source, Linux-based SONiC network operating system Microsoft (a founder member of the eBPF Foundation) created to simplify building its Azure infrastructure, which is now supported by many network hardware vendors for smaller cloud providers and enterprises, especially in verticals like financial services, as well as hyperscalers.
“One of the reasons Microsoft has been investing in eBPF is because we see the importance of making the data plane itself programmable and controllable by software,” SONiC founder and Azure Networking engineering lead Dave Maltz told the New Stack.
Disaggregating the network stack means network architects could choose which protocols they want to run and use just the software modules required for that; not running software that provides features you don’t need means less overhead and fewer potential bugs.
It also allows the network OS to use public APIs to provide functionality; the code that provides that API can be improved but code that depends on it won’t have to be updated when that happens, which allows for much more innovation in the networking stack.
DASH (Distributed APIs for SONiC Hosts) is a network project to do that for the software-defined networking data plane, using SmartNICs and other hardware to improve network performance for cloud services by doing more work in the network itself — like encryption or key management — in ways that merge compute and networking.
“We need to expose higher-level APIs for the control of that SDN data path and eBPF is a great way to implement that,” Maltz said.
On Linux, eBPF is the evolution of a capability that’s been around the 90s but now has enough features to be broadly useful for an increasingly wide range of applications.
“Over the years, it’s grown from we can run some arbitrary code when a packet arrives, to we can run some arbitrary code when some other thing [happens]. We’re seeing more and more places where you can hook into the kernel,” Isovalent’s Chief Open Source Officer Liz Rice told us.
eBPF is ideal for debugging, application tracing and performance troubleshooting, getting observability data without the usual intrusiveness and overhead — and for creating workarounds and compatibility fixes for limitations in software you can’t change.
Most of what New Relic’s open source Pixie Kubernetes observability platform does relies on eBPF and Splunk recently donated its Flowmill eBPF collector application and kernel telemetry to the Cloud Native Computing Foundation‘s OpenTelemetry project.
Those telemetry collection components will help developers stitch together telemetry data from different layers and could help to make eBPF simpler to work with, suggested Dave Thaler, partner software engineer at Microsoft.
“eBPF is a very efficient and safe means of obtaining information about processes, with good flexibility around what and how information should be collected,” Thaler said.
Netflix has been using eBPF to get much deeper metrics for analyzing performance on Amazon Web Services‘ than AWS itself originally thought was possible, but when it first started that work it was still a tool for experts. “It seems to be crossing into the mainstream now,” RedMonk co-founder James Governor told The New Stack.
It’s so useful and mainstream that developers and Microsoft customers started asking when they could get something similar in Windows.
Why Windows
“eBPF has emerged as a revolutionary technology enabling greater programmability, extensibility and agility,” Thaler told us. That’s as useful on Windows or any other OS as it is on Linux, and using common tooling and frameworks across Windows and other platforms provides engineering efficiency, he noted.
“For [developers] who already use eBPF on Linux, it is attractive to use eBPF on Windows to enable the same type of solutions to work on both platforms. Even for those who only use Windows, we believe that the programmability, extensibility, and agility benefits of eBPF will open up development to a wider audience.”
And he expects the same speedups eBPF can bring with SmartNICs with approaches like DASH should be possible for Windows in the future.
Windows already has ways of extending low-level functionality by calling public APIs, such as NDIS, Windows Filtering Platform, DTrace, the Driver Interception Framework (DIF) and so on. But extending the Windows kernel requires writing a driver and submitting it to Microsoft for signing which, Thaler points out, is a slow process.
“Since the eBPF verifier uses a formal methods based analysis to check code safety-based, eBPF enables agility of rapid extensibility without having to wait for a long [approval] process, which can be especially useful in time-critical scenarios such as debugging or DDoS mitigation.”
“eBPF for Windows will enable developers to safely consume frameworks like these using multiple existing well-known programming languages without writing a kernel driver, and leverage the eBPF ecosystem of cross-platform tools and experience.”
Rice agrees: “eBPF gives us this really powerful platform for building things like observability tools and there’s no reason why that shouldn’t be just as applicable on Windows as it is on Linux.”
Importantly, eBPF is being built to run on Windows rather than to be part of Windows, which means it will run on existing versions of Windows rather than needing an update to the OS. The open source eBPF for Windows project supports Windows 10, Windows Server 2016 and later, with Microsoft contributing code to existing open source eBPF projects so that they work with Windows as well as Linux (and potentially other operating systems in the future).
And when the project is mature, it will move to what Microsoft describes as “a community-governed foundation in the eBPF ecosystem.”
Building on Open Source
Windows uses drivers whereas Linux uses kernel modules and APIs rather than system calls, so eBPF needs to be implemented slightly differently. The modular architecture Microsoft chose also means eBPF can be used in more scenarios.
Before it runs, eBPF bytecode is checked by the open source PREVAIL static verifier that runs in user mode rather than in the kernel: if it passes all the safety checks, the code is either compiled to native code by the open source uBPF JIT compiler that also runs in user mode or passed straight to the uBPF interpreter. The interpreter and the compiled native code both run in kernel mode.
Running the verifier and just-in-time (JIT) compiler in user mode is a big difference from eBPF on Linux, but it makes sense, Thaler told us, and not just because Microsoft wanted to build on an existing community project that had already made the decision to build for user mode.
There’s an increasing trend to moving code out of the kernel, because if there’s a bug in the driver that causes it to crash it won’t take down the whole OS.
But it also means that eBPF can be used to extend user mode daemons, not just the kernel. You could even run eBPF on one device and have it provide functionality and performance improvements on another machine. Thaler explained:
“We believe that an important property of code for eBPF building blocks like the verifier, interpreter, and JIT compiler, is that the same code should be reusable not just across multiple platforms, but be usable in different contexts such as built into an OS kernel, or run in user space, or run inside a Trusted Execution Environment, or even run on a separate machine from the OS being extended.”
That means that the verifier has to be very reliable, and it needs to be protected so that it’s as secure as the kernel.
There’s already a test suite for the verifier and it will be fuzz tested before it’s ready for production. The safety properties that the verifier providers for code are also stronger than the tests that Microsoft typically runs on drivers before signing them.
Both the verifier and compiler run in a privileged system service on Windows, and user-space APIs can only be called by admin accounts, Thaler notes, “We are investigating future models where the verifier and compiler can run inside a separate VM, or even on another machine.”
“Much of the security hardening work for eBPF for Windows still remains, which is why it is not yet signed for used in production, only test environments,” he warned. Although you can install and try it out today, you’ll have to put your Windows PC into developer mode, with test-signed binaries enabled; that’s common for developers but not suitable for production systems for security reasons.
You’ll also have to stick to interpreted mode unless you turn off the HyperVisor-enforced Code Integrity (HVCI) hardware virtualization feature Windows uses to protect kernel mode processes like the logon service from attack. The JIT compiler doesn’t sign code with a signature that HVCI trusts, so that code won’t currently run.
“JIT compiled mode is more efficient than interpreted mode, if all other things are equal,” Thaler points out, so Microsoft is looking at how to get JIT compilation working with HVCI.
Cross-Platform Code
Developers can create eBPF bytecode for Windows with existing eBPF tools like clang, include it in a Windows application or just type it into the Windows netsh command-line tool (much as Linux users would use Bpftool); they all call a shared library that exposes Libbpf APIs (and passes the code on to the verifier).
Microsoft plans to enable anything that you can call as a public API and write a driver in Windows that’s relevant to eBPF to be exposed to eBPF. “The eBPF runtime does provide additional constructs from eBPF, such as using shared memory via support for a wide variety of ‘map’ constructs.”
That code might or might not be the same eBPF code you’d run on Linux because it depends on whether you’re doing something that’s supported on Windows.
“Some eBPF APIs are inherently Linux-kernel-specific since they interact with or extend functionality in a way that depends on knowledge of internal implementation details. Other eBPF APIs extend functionality such as TCP/IP that is common across platforms,” Thaler explained.
Call a network socket or bind to a common protocol like IPv4 or IPv6 and you can recompile Linux eBPF code to run on Windows.
“Our goal is to have a large set of cross-platform APIs, while enabling anyone to easily add additional APIs for their platform of choice, such as to extend functionality in their own drivers or user-space applications,” Thaler said.
That will mean if eBPF bytecode calls those APIs, the same code will work on Linux and Windows.
As for the hooks and the helper functions eBPF code can call, eBPF for Windows already supports two hooks and over 10 cross-platform helpers documented in bpf_helper_defs.h and ebpf_nethooks.h.
“For the core eBPF execution context, Microsoft has had to add implementations of various map types and helper functions since there was not an existing open source project that could be immediately leveraged,” Thaler said. “Our hope is that over time such implementation will move to a cross-platform project such as the generic-ebpf project, that eBPF for Windows can then use.”
Microsoft is focusing on the most commonly used ones first to enable popular applications, but any hooks and helpers that call public APIs can be contributed from the community, he said.
Because eBPF has so far only been available on Linux, eBPF tools like Libbpf, Bpftool or Kinvolk’s Inspektor Gadget suite for inspecting Kubernetes clusters (now owned by Microsoft) assume you’ll be using Linux and rely on Linux-specific functionality and implementations even of things that are available on other platforms.
“A number of helpers and hooks on Linux are inherently Linux-specific or expose structures that use details of the Linux implementation. Some of them wouldn’t apply to any other platform, and some would apply but other platforms use a different native format,” he explained.
For those that apply but simply use a different native format, some cross-platform functionality can still exist by copying the data into the format exposed by the existing eBPF helper, with a slight performance cost.” Microsoft is working with the community to separate the native and cross-platform functionality in Libbpf and Bpftool so they can be used on Windows as well.
Higher-level tools like the L3AF lifecycle management and orchestration project for eBPF networking apps (including load-balancing, rate limiting, traffic mirroring, flow exporter, packet manipulation and performance tuning) that Walmart contributed to the LF Networking group are also currently only for Linux; again Microsoft is working with the L3AF community to bring support for Windows.
The number one request, ahead of even the observability and debugging features eBPF will provide, is the denial of service protection it will offer on Windows.
Another big opportunity for eBPF on Windows is for organizations to support their own legacy in-house applications when they need to tweak things like port redirection or socket handling. If you need to use port 8080 when the application expects port 80, or you need to work with a socket and the middleware only exposes an http abstraction, eBPF makes it easer to create workarounds — without needing to write a custom driver and get it signed.
That’s something a lot of organizations with Windows servers will find useful.
