PART 1 OF A SERIES.

The primary goal of an attacker is to achieve a specific objective without being detected. This often involves establishing a foothold in an environment and then moving laterally. In a real-life compromise an attacker operates with little to no information about the target’s security controls. As a result, they must adjust their tactics to focus on remaining undetected by an unpredictable and potentially wide spectrum of security controls. Controls mature over time and new technologies have been developed, prompting attackers of all types to adapt and improve their tactics to maximize their likelihood of success.

From the perspective of an attacker, one of the more challenging of these newer technologies is endpoint detection and response (EDR), which is often touted as the future of antivirus (AV). Traditional AV is designed primarily to facilitate prevention and detection of malicious code using a combination of signatures and heuristic analysis. EDR, however, has two primary jobs: detection of malicious behaviors or other signatures and facilitation of analysis and incident response (IR). Because of this, EDR solutions have become a de facto requirement for comprehensive defense against attackers.

EDR is specifically designed to detect suspicious behaviors occurring on the endpoint. These behaviors may include a range of attack techniques, such as process execution or injection and image loads in memory. Once those behaviors have been identified, the second role of EDR comes into play, as the tools are leveraged by defenders and incident responders to take action based on the detection. This triage response may include isolating the compromised host from the network, facilitating the collection of endpoint logs, reviewing the timeline of events, collecting and documenting threat indicators or even allowing for the termination of suspect processes.

Attackers respond to the frequent deployment of these technologies by developing new, highly sophisticated techniques to evade detection on disk and in memory. These techniques extend beyond the traditional initial compromise vectors and are often utilized in post-exploitation techniques to prevent any type of detection throughout the attacker’s operation. The key to these capabilities, wherein the EDR technology makes many decisions in milliseconds, is derived from its ability to hook into all running processes on the host.

This project would not have been possible without the high-quality work from those before us. Many people have publicized ground-breaking research in reviewing EDRs and memory hooks, focusing on ways to circumvent specific products. The intent of this article is to go deeper on the subject, not focusing on techniques that only work on specific products but instead identifying systemic issues across all EDR products and how attackers can utilize them to bypass any EDR product without needing prior knowledge of a client’s security stack.  This involves diving deeper on these previously discussed concepts as well as discovering new ones.

If you are interested in learning more, here are a few additional sources:

What is hooking?

Hooking is a technique to alter the behavior of an application, allowing EDR tools to monitor the execution flow that occurs in a process, gather information for behavior-based analytics and detect suspicious and malicious activity. This allows for more accurate detection rates of post-initial compromise techniques (i.e. code execution) as well as post-exploitation techniques (i.e. privilege escalation, lateral movement or ransomware activity).

These hooks send the data to the EDR agent running on the endpoint for the telemetry to be processed in real time. The agent is often installed at the kernel level, the highest privileged access. There are two major reasons for this; the first is to avoid being shut down or removed by an attacker, as it isn't easy to gain access to services running in the kernel. To be exploited they typically require some sort of vulnerability or the attacker must already have obtained high-integrity privileges on the endpoint. Since attackers operate with a “black box” mentality, it’s typically assumed that the initial foothold does not already have these privileges and they must be acquired through post-exploitation activity, which may be caught by an EDR if not carefully executed. Another reason these products run in the kernel is that it provides the EDR access to control and monitor the entire system.

Image

Figure 1: Privilege Map

Agents often place hooks into the process by loading a DLL they own, which remaps functions already loaded. This allows the agent to monitor each and every process, gathering telemetry. The agent is constantly receiving information monitoring all changes in the process, on disk and even network communications. The agent passes the data over to the product’s cloud-based platform. This is where all data telemetry is processed into actionable data that can be classified as either malicious or benign behavior. While a majority of the preventative controls are performed by the agent, often times modifications to attack techniques can circumvent the agent’s initial detections. This is where the capabilities of the EDR’s platform really shine in identifying malicious behavior based on all the data gathered in the wild.

Image

Figure 2: EDR Telemetry Diagram

For example, a process spawns a new process in a suspended state and modifies the memory permissions on that new process, attempting to execute a WriteProcessMemory procedure. The data written is encrypted and doesn’t trigger any malicious indicators for the agent; however, the telemetry sent to the EDR platform can still view this data and make a determination to identify these events as a potential process hollowing technique. All of this happens in less than a second, ensuring threats are not successful, which means the agent needs numerous data points to make these analytic decisions.

To further understand how this data flow works, we need to know a bit about Windows architecture. For starters, the Windows system offers a large set of functions and API calls that an application can utilize to execute code. The primary function of the Windows APIs is to align all the stack registers before calling the syscall to execute the low-level instruction.

Syscalls such as NTAllocateVirtualMemory provide a low-level interface to allow a process to interact with the operating system. These are low-level assembly instructions transition to the kernel and are used to tell a CPU to perform an action such as allocating memory, creating a file or writing data stored in a specific buffer to disk. These syscalls reside in the ntdll.dll dynamic linked library (DLL), and while many of them are not documented, syscalls cannot be directly called as they only perform a low-level assembly instruction.

Image

Figure 3: Execution Flow Example

When a process is executed, system DLLs are loaded, at which point the EDR agent hooks specific API functions and syscalls such as VirtualAlloc and NTAllocateVirtualMemory. It is important to note that each EDR platform hooks different functions and syscalls, which provides different telemetry which in turn yields different information and different detections. As the execution flows, the EDR hook triggers, forcing the execution to jump from the system DLL to the EDR DLL, at which point the EDR performs a series of instructions and then returns the execution to the system DLL.

Image

Figure 4: Execution flow Example – EDR Hooked

As you can see, the same syscall is called twice. This is due to the transition between user mode and kernel mode. System calls may be prefixed with the characters NT or ZW. NT syscalls represent calls from User Mode, while ZW syscalls indicate a Kernel Mode call. Regardless of the prefix, the underlying syscall instruction is identical.

Image

Figure 5: NT and ZW Prefixes

So why do EDRs hook user mode?

While kernel mode is the most elevated type of access, it does come with several drawbacks that complicate EDR effectiveness. In kernel mode, visibility can be quite limited as there are several data points only available in user mode. Also, third-party kernel-based drivers are often difficult to develop and if not properly vetted can lead to higher chances of system instability. The kernel is often regarded as the most fragile part of a system and any panics or errors in kernel mode code can cause huge problems, even crashing the system entirely.

User mode is often more appealing to attackers as it has no way of directly accessing the underlying hardware. Code that runs in user mode must use API functions that interact with the hardware on behalf of the application, allowing for more stability and fewer system-wide crashes (as application crashes will not affect the system). As a result, applications that run in user mode need minimal privileges and are more stable.  Suffice to say, a lot of EDR products rely heavily on user mode hooks over kernel mode, making things interesting for attackers.

Since the hooks exist in user mode and hook into our processes, we have control over them. Since applications run within the user’s context, this means everything that's loaded into our process can be manipulated by the user in some form or another. It’s important to note that some sensitive regions of memory are set to Execute, Read (ER-), which prevents the modification of these regions. We’ll discuss some techniques to navigate around this below.

Bypassing EDRs in memory

Before we can even look for memory hooks, we need to identify the EDR’s DLL. This is key to narrowing down our searches, as a simple application can have numerous different DLLs loaded and they can change based on what functions a process needs. For instance, the DLL ws2_32.dll is required for any Windows socket connection.

Identifying the EDR’s DLL can be as simple as looking at the name of the DLL or description. Other techniques can involve looking for the path of the DLL or reviewing the code-signing certificate of the DLL. There are several great tools to do this but for our case, we will use ProcessHacker2, a free process monitoring software. As we can see, there are several DLLs loaded but one stands out both by the name and description, as the name contains the EDR product’s name and the description alludes to its purpose.

Image

Figure 6: Execution flow of a Procedure – EDR Hooked

Now that we have identified the EDR DLL, the next step is searching for these hooks. For that we need to have a deeper understanding of these system-level functions and how they operate on the stack. If we take a look using the opensource debugger x64dbg, we can see that each DLL has a series of functions they export. These are the functions that an application can utilize.

Image

Figure 7: Sample Export Functions of Ntdll

We know that syscalls are the only way for applications to access the kernel to perform low-level instructions; let’s examine how these instructions look at the assembly level. In x64 bit architecture, every syscall should begin with the value currently stored in RCX moved to R10, followed by a hex value moved into the EAX register. This hex value is a system service number and is unique for each syscall.  The kernel has no insight into what actual instruction is executed and just looks up the unique syscall ID to determine the corresponding instruction to perform. In this example, you can see how a typical syscall would look in assembly. The (SYSCALL_ID) will always be unique to that special syscall.

With these pieces of information, we now know the EDR’s DLL name and what these system calls are supposed to look like, so we can go looking for differences in these functions. We also know that the EDR redirects the flow of execution by modifying the first five bytes, starting with a jump (e.g a jmp instruction, which is “E9” in hex). We can use this to search for potential hooks by searching for any E9 patterns in ntdll, that jump to the EDR’s DLL.

Image

Figure 8: Execution flow of a Procedure – EDR Hooked

Here we can see clearly the hooked syscalls.

Image

Figure 9: Execution flow of a Procedure – EDR Hooked

Now this is not always the case as some products try to mask their jumps by first jumping to a different region within the same system DLL or a thread in in the current process before jumping to the EDR itself. In many cases the results are the same. We still can identify these instructions by looking at the syscalls for any modifications to the assembly.

Image

Image

Image

Figure 10: EDR Indirect Jumps

With the knowledge of all known hooks, we can then restore these values by patching the correct bytes, overwriting the EDR’s hooks. This prevents any telemetry from being transmitted from the process to the agent, as the EDR's DLL has no data to send.

This method is great in theory; however, several factors can make this difficult to execute. The first major issue: there's a significant chance that overwriting all the hooks one after another will be detected by the EDR before this process can be completed. Another major issue is knowing exactly what functions each EDR hooks. Each product hooks different sycalls and an attacker never knows what products are in use (depending on their OSINT). Containing the cleaned versions of each possible function that could be hooked for each version of Windows can also increase the size of your payload exponentially. Additional research is required to determine exactly where to make these targeted modifications without detection across all products.

A simpler method involves reloading sections of your process’s memory to flush the EDR out. This works because we know the EDR’s hooks are placed when a process spawns. We also know which DLLs EDRs typically hook. We can target these DLLs and manipulate them in memory by using the API function VirtualProtect, which changes a section of a process’s memory permissions to a different value. These values can be a combination of Read, Write and Execute.

We begin by first opening the system DLL located on disk. These files are located in C:\Windows\System32\ or SysWow64 for 32-bit. These DLLs stored on disk are unhooked or “clean,” as an EDR only hooks content that is loaded into memory. Once the DLL has been opened in memory, we can grab the bytes that make up the .text section of the DLL. This section of a DLL contains the executable assembly - in NTDLL’s case, the executable code with the syscalls.

Image

Figure 11: Execution flow of a Procedure – EDR Hooked

While this provides a copy of the clean bytes, we still need to make sure we write the bytes into the right location in the process’s memory. This is a problem due to address space layout randomization (ASLR), which ensures that the positions of libraries on the heap and stack are randomly positioned in memory. Since the process has already been established in memory, this can be a problem because if we don’t apply the proper bytes to the right memory address, when the application tries to execute code it will crash. Instead of trying to remap or modify the process any further, we can rely on the offsets for each of the functions stored in the .text. Each function has an offset which denotes the exact number of bytes from the base address where they reside, providing the function’s location on the stack.

Image

Figure 12: Execution flow of a Procedure – EDR Hooked

Using VirtualProtect we can change the loaded version of ntdll’s .text region to be writeable. While this is a system DLL, since it has been loaded into our process (that we control), we can change the memory permissions without requiring elevated privileges. It is also important to note that we are only changing a section of the DLL to avoid detection. Once this is completed, we can then write the bytes based on the proper offsets and restoring the correct, unhooked bytes to the proper memory addresses.

Image

Figure 13: Execution Flow of a Procedure – EDR Hooked

Now that the syscalls have been restored to the values set prior to the EDR’s hooks being placed, execution can continue without any hooks on the syscalls. The EDR’s DLL is still present but it will not receive any telemetry from the hooks since they are no longer there. The EDR is unaware that its hooks are no longer in place as there is typically no integrity check performed to ensure they are still active.

Image

Figure 14: Before Reloading Occurs – Hooks Still in place

Image

Figure 15: After Reloading– Hooks are Gone

This unhooking approach is quite effective. However, changing the permissions of memory regions can still lead to detection, such as identifying calls to the VirtualProtect function. While the unhooking attack is successful, monitoring controls can indicate the presence of this attack. This can be overcome by using other techniques to overwrite sections of memory, bypassing the need to call any Windows functions to change the permissions on these unwritable regions of memory. By using low-level operating system primitives, we can overwrite and restore the assembly of system functions without modifying the Execute, Read (ER-) permissions.

Another technique involves using our own assembly syscalls rather than relying on the operating systems. This is an effective method to avoid triggering EDR hooks as an EDR only hooks what it knows and, in this case, it only knows what exists in the system DLLs. This means we can avoid detection by creating our own function calls to set up the registers, along with our own syscall in the application’s sections of memory, rather than in the loaded system DLLs. For us to do this we need assembly code that will contain our setup, set the variables and registers with the correct values and execute our syscalls. These assembly instructions will be contained in a standalone assembly file that will be loaded into our code when we compile our payload. It is important to have these syscall values be dynamic as the specific syscall ID value changes depending on the Windows architecture. Different versions of Windows have different syscalls depending on the operating system. For example:

Image

If you’re curious, a great reference to have when developing your own syscalls is j00ru’s Windows-Syscall. For the most part the code should look like this.

Image

Figure 16: Examples of Custom Syscalls and Assembly

The next bit of code we need to include is our own version of the WinAPI function to align the registers with proper values prior to our syscall executing; this is due to the low-level nature of syscalls. These procedures are typically stored in either kernelbase.dll or kernel32.dll. Let’s continue with the example of NTAllocateVirtualMemory.

Each of these values need to be stored in a register on the stack, in a specific order, prior to the syscall’s execution. When an application calls the VirtualAlloc function, there are only four attributes required: address, size, type and protection.

These values are always going to be dynamic based on the use cases. The one thing that will be static is the handle that uniquely identifies the VirtualAlloc function. It’s also incredibly hard for a product to read every single string of bytes to determine the presence of APIs or syscalls outside of system DLLs.

Image

Figure 17: Examples of Syscall Custom Assembly

Image

Figure 18: Custom Assembly Location in Memory

While this sounds simple to develop, there are a lot of small dependencies that, if not properly developed into your code, can lead to issues or kernel panics. Earlier we reviewed the execution flow of normal requests and how the WinAPIs and syscalls interact with the kernel. When developing your own syscalls and functions, you need to also provide references for all possible response codes and hex values associated with each of those values. When the instruction is returned, our code needs to know how to interpret the responses; obviously these responses will not be “ALL GOOD” or “this value you caused an error.” These are often hex values that the system interprets into error codes.

Image

Figure 19: Error Code Example

While we’ve highlighted some of the techniques that attackers can use to circumvent EDR products’ prevention and detection capabilities, this doesn’t mean the products are done. EDR products are still remarkable products that are important to the detection and prevent of cyber threats. The conversation should not be focused around replacing these products for something else; instead, it should be on what can be done to improve these products, as they are not going away any time soon.

What can be done about this?

While we only discussed several effective techniques, there are many different ways to achieve the same goal. This may seem very daunting to an organization that relies on EDR products to help detect and prevent real-world cyber threats.

The hard truth is that there is no easy fix. Much of the remediation would involve continued overhauling and improvement of these products. The first would be implementing anti-tamper sensors, which monitor the application behavior looking for specific characteristics of indicators that would indicate changes in the process’s memory, specifically focusing around the loaded system DLLs. There are very few cases when a process would need to modify the system DLLs, as they are only used to interact with the operating system.  In essence, these sensors would work by monitoring for any modifications to system DLL bytes or the permissions after the initial process is created and DLLs are loaded.

Another more extensive route is to move away from relying on hooking the user mode in favor of living more in the kernel. There are many benefits to moving to this approach, as the kernel is the first part of the operating system to load. Because of this, it can be positioned to monitor any attempts to manipulate the user mode hooks. Most modern EDR products already have some sort of functionality existing in kernel mode, whether it be for network connections or file system monitoring. Expanding on these optics can not only provide additional telemetry but also help in preventing any tampering by an attacker, as modifying kernel mode resources requires some sort of privileged access. In addition, mechanisms such as PatchGuard prevent any modifications to resources operating in the kernel. As I mentioned before, it does take a lot more effort to develop a stable product that runs in the kernel space, but the trade-off is much greater than in user mode.

Attackers can clear event logs or spoof command line arguments to obfuscate the telemetry, making it more difficult to get the right information. Windows Event Tracing (ETW) makes doing so extremely difficult. ETW provides a way to trace and log events from both user and kernel mode. This data is extremely granular in nature. While ETW operates primarily in the kernel, it relies on syscalls stored in ntdll to provide telemetry. These syscalls pass along telemetry that is used as data points by ETW. What makes ETW very unique is that it does not rely on hooks, as these are all native to Windows with the primary purpose of feeding data into the kernel. This means there is never a need to modify or hook these syscalls. Instead, ETW determines what data is relevant and what isn’t. By utilizing ETW instead, a product can make the same determinations as it would have based on data coming from user mode hooks.

While there are techniques publicly disclosed on how to circumvent ETW, the fact remains that elevated privileges are not required for these types of attack. This makes execution of these techniques difficult in the real world as typically an attacker’s initial point of compromise will often not have elevated privileges and are typically a post-exploitation goal. Capturing some form of ETW telemetry from hosts can provide blue teams a tremendous amount of data necessary to triage the detection and understand the risk posed by the threat. It also helps incident responders understand how and when the attack occurred, which in turn informs response processes.

Implementing EDRs is great but relying on them as your singular source of detection against cyber-attacks can create a single point of failure. Additionally, employing a comprehensive set of security controls that are complementary to EDRs can be quite effective in filling in those gaps to help prevent and detect cyber-attacks. This can include making sure that event logs are being collected from all endpoints, not just critical systems or servers, so defenders are able to correlate with activity indicative of an attack. These events may not clearly indicate an EDR bypass attack but can provide other useful artifacts that are needed for detecting and preventing attacks.

Additionally, network-based controls are just as important in detecting malicious activity. Network-based controls that perform deep packet inspection and TLS intercepts can detect malicious activity over the wire. Even if attackers are able to bypass the security controls and suppress logging events, network controls can detect the outbound connections that can lead to the detection of C2 channels.  Finally, application whitelisting or other methods of restricting the execution of applications can hamper an attacker’s ability to bypass EDRs.

While these are all great starting points to help improve your detection and prevention capabilities, the truth remains that this is very much a cat and mouse game. Attackers will always build new techniques to evade detection and defenders will have to adapt to overcome them. EDR products aren’t exempt from this ever-adapting dynamic.

Please be sure to continue with part two of this series. In addition, we will be releasing a series of tools that highlight the EDR techniques discussed here. These code projects will be found on Optiv’s GitHub.

Disclosure Timeline

References:

Here's a review of related posts on this series: