How to Detect Fileless Malware with Endpoint UEBA

Stop dangerous and undetectable attacks by focusing on behavioral clues—not rules.

Traditional malware attacks were carried out by malicious software installed on the target machine. With ‘fileless’ malware, tools that are built-in to the host machine are used to carry out attacks. Since there is no software installed, it is very difficult for traditional signature-based defenses to detect that a breach is in progress, ultimately leading to significant time before the breach is discovered.

Fileless malware attacks have the same goals and a similar attack lifecycle as traditional ‘dropped’ malware but occur using legitimate tools installed on every machine, such as PowerShell and Windows Management Instrumentation (WMI)—a technique known as “living off the land.” After the initial compromise, instead of dropping software onto the victim’s machine, native system tools, such as PowerShell, are spawned to execute a payload, carrying out the next phase of the attack. Since PowerShell is a system tool, its malicious use is not detected by traditional security approaches and even highly experienced security analysts. Understanding the normal behavior of users, systems, and processes through unsupervised machine learning on rich endpoint data, however, is a powerful tool in detecting malicious use of legitimate system tools.

Figure 1: Powershell executes normally, but payload is malicious.
Example: Command-and-Control Attack Using DNS

DNS is one of the most commonly used protocols on enterprise networks and is very difficult to detect when used for tunneling covert command and control payloads. Throughout the stages of a breach, established normal baselines given by unsupervised machine learning for users, hosts, and processes can help to identify abnormal activity.

Initial Compromise

Consider a common phishing attack with an infected Microsoft Word document where an attacker can ultimately get a shell on the victim’s host computer. Code is passed to a PowerShell command through the command line, thus allowing its execution without being written to disk. The following table shows some interesting behaviors that are outside of the norm for a process, user, and system.

Fileless Malware Chart

Establishing Persistence

More interesting activity continues as the script passed to PowerShell in the initial compromise executes. One of the goals of the script is to establish persistence by querying and setting registry paths that are commonly used for this purpose. Depending on what type of account this script is running (administrator versus normal), attempts will be made on run keys in the following paths:



Establishing persistence can also be achieved with a scheduled task. This would involve the following key:

HKLM\Software\Microsoft\Windows NT\CurrentVersion\Schedule\

There are a number of additional sensitive areas in the registry where modification would be suspicious for certain users and processes. With established baselines on registry activity using unsupervised machine learning, anomalous activity would be detected through the introduction of a number of rare activity models:

Fileless Malware Activity and Model 2

C2 Activity

Whether an organization has tightly controlled outbound DNS or not, an attacker has options to get additional code to the victim’s computer through DNS responses. In the case where all outbound DNS traffic is allowed, a direct connection to a C2 server can be tunneled through DNS. Similarly, when only trusted DNS servers are allowed, an adversary could register a domain and designate a controlled server as the authority for this bogus domain. This would result in traffic being forwarded and returned to the compromised host, allowing for ongoing control. The network traffic logs for this activity—even in the eyes of the most skilled analyst—would likely just blend in with the rest of the DNS traffic on the network. However, with unsupervised machine learning in place, detection is fully automated since destination, frequency, and payload of DNS traffic would prove to be abnormal. 

C2 Activity PowerShell
Figure 2: Unusual activities appear as PowerShell uses DNS.


The endpoint is often where a breach originates, whether through an exploited operating system vulnerability, a user that has fallen victim to a phishing attack, or an insider with ill intent. A security approach that centers around patterns of previously seen attack techniques is a challenge because attack methods are ever more sophisticated. And a security approach that relies on the manual investigation of data cannot scale. With unsupervised machine learning and advanced models, fueled by a rich stream of endpoint data, previously undetectable attacks can be surfaced.  

Want to learn more? Read about Interset’s newest endpoint detection capabilities here, and contact us to schedule a demo.

Ron Chittaro is a senior principal software engineer at Interset.