Insights
K2: Detecting Syscalls

SysWhispers, HellsGate, HeavensGate, SidewaysGate, SpoofGate, TFGate, DoomGate, whatever gate your tool is being detected before the initial handle fully opens. How do EDR's detect & deny direct and indirect syscalls?

20-05-2026 • 8damon
8 min read

K2: Detecting Syscalls

Intro

Welcome to my new piece on the endless reimplementation of Direct, Indirect, SpoofDirect whatever name you can throw at syscalls.

Let's rediscuss this 10+ year old technique and why all of its re-implementaitons and complexities are missing the fundemental flaw in the design "design"...

The Initial Problem

At some point some very smart person realized that a very invasive EDR had decided to switch Microsoft's normal system call (syscall) prologue with some sort of redirect that is causing them to become detected.

Native sycsall stub;

NtOpenProcess
MOV        r10,rcx
MOV        eax,26h
TEST       byte ptr [7FFE0308h],1  ; KUSER_SHARED_DATA.SystemCall | STATUS_WAIT_1
JNE        short 0000000180162165h  ; → RVA 0x00162165
SYSCALL
RET
INT        2Eh
RET

Hooked by EDR:

NtOpenProcess
MOV        rax, EDR | NtOpenProcess
JMP        rax

INT3
INT3
INT3
INT3
JNE        ntdll.dll+addr
SYSCALL
ret
INT        2Eh
RET

Now very smart person Isn't too happy they're detected, and they need another way to call API's without EDR's noticing. Notice anything about the syscall prologue?

MOV eax,26h, this loads a number into EAX. That number is the "SSN", AKA "Syscall Service Number", which routes down to the SSDT (System Service Descriptor Table), which then routes it to the releveant kernel function (this is irrelevant, you can ignore it)

Ok so what If we just copy the bytes from ntdll.dll+NtOpenProcess into our own stub and execute that?

void* stub = VirtualAlloc(NULL, 0x1000, MEM_COMMIT | MEM_RESERVE, PAGE_EXECUTE_READWRITE);

memcpy(
    stub,
    GetProcAddress(GetModuleHandleA("ntdll.dll"), "NtOpenProcess"),
    0x20
);

((NTSTATUS(NTAPI*)(HANDLE*, ACCESS_MASK, POBJECT_ATTRIBUTES, PCLIENT_ID))stub)(
    &hProcess,
    PROCESS_ALL_ACCESS,
    &oa,
    &cid
);

And it works! I feel like a genius. But wait! How are we supposed to get that clean stub when the one in our memory is hooked? Fear not! Endless repositories of Python scripts generating syscall numbers & stubs for you exist!

Back to smart guy, smart guy has now obtained one of these many repositories and has embedded a bunch of these syscall stubs into his binary, and he's ready to infiltrate.

...Unfortunately the file never executed because a YARA scan just sniped the stubs;

$syscall_stub_simple = { 4C 8B D1 B8 ?? ?? ?? ?? 0F 05 C3 }
$syscall_stub_ntdll_like = {
    4C 8B D1
    B8 ?? ?? ?? ??
    F6 04 25 08 03 FE 7F 01
    75 ??
    0F 05
    C3
}

$syscall_ret = { 0F 05 C3 }

Thats unfortunate. Smart guy now needs a different way to resovle these SSN's, so he can build his own "direct" syscall stubs. What if we load another ntdll.dll from disk and get the syscall stubs from there? Okay now the EDR's wondering why smart guy's double-loaded ntdll.dll. Smart guy ignores this, its a "tradeoff", and "not inherently suspicious". Sure, moving on.

Smart guy now calls his NtOpenProcess stub, on whatever process he's targeting. And unfortunately for smart guy the EDR's Object Manager (Ob*) callback has just seen a PROCESS_ALL_ACCESS handle, or VM_WRITE whatever you think is stealthy into the target.

But how? The kernel! Enter NtKernel;

API Call Map for NtOpenProcess  [1 call site(s)]:
  └── 0x162162  SYSCALL  ntdll!NtOpenProcess [syscall]
      kernel: ntoskrnl!NtOpenProcess
      └── 0x84E8EE  CALL  PsOpenProcess [internal]
        .....
          ├── 0x84EE6F  CALL  ObOpenObjectByPointer [internal]

See that Ob* call? The kernel provides this wonderful API called ObRegisterCallbacks, which allows EDR's and alike to see things such as OB_OPERATION_HANDLE_CREATE, OB_OPERATION_HANDLE_DUPLICATE, or even POB_PRE_OPERATION_CALLBACK which allows EDR's, or whatever else down there to block on callback creation. Which means not only can the EDR see who is opening a handle, to what, and with what rights, but also the attached stack. You see where I'm going with this?

I should also mention that these callbacks are in no way limited to handles. Microsoft provides everything from image loads, thread creation, access checks, network ops & more.

EDR's will, granted, only sometimes, and only some, walk the stack. Now the stack looks pretty strange when you've got the syscall originating from some random private process memory, instead of the correct ntdll.dll!ExportStub. So now not only has smart been flagged by PROCESS_ALL_ACCESS, but also a direct syscall.

So instead of coming up with a less noisy way, or leveraging something else, or thinking outside the box, smart guy decides to try to legitimize his stack-frames instead. This is where these newer implementations of Direct/Indirect syscalls come in, they stack spoof, abuse vectored exeception handlers, jump around mid-control flow, ROP chain, basically anything but executing the EDR hook, without concerning themselves with anything going on in the kernel. Even unhooking the EDR's hooks, which sounds smart until the EDR runs a image hash check and figures out they've been unhooked, and now the SOC rooms red.

Anyway. K2, a roided up kernel driver built for detecting syscalls and detecting syscalls only, using the exact stack-frame technique just discussed.

Enter K2

Let's register some fat callbacks so we can see everything;

status = PsSetCreateProcessNotifyRoutineEx(K2ProcessNotifyEx, FALSE);
if (!NT_SUCCESS(status)) {
    return status;
}

status = PsSetCreateThreadNotifyRoutine(K2ThreadNotify);
if (!NT_SUCCESS(status)) {
    PsSetCreateProcessNotifyRoutineEx(K2ProcessNotifyEx, TRUE);
    return status;
}

RtlZeroMemory(g_OperationRegistrations, sizeof(g_OperationRegistrations));
g_OperationRegistrations[0].ObjectType = PsProcessType;
g_OperationRegistrations[0].Operations = OB_OPERATION_HANDLE_CREATE | OB_OPERATION_HANDLE_DUPLICATE;
g_OperationRegistrations[0].PreOperation = K2PreOperationCallback;
g_OperationRegistrations[1].ObjectType = PsThreadType;
g_OperationRegistrations[1].Operations = OB_OPERATION_HANDLE_CREATE | OB_OPERATION_HANDLE_DUPLICATE;
g_OperationRegistrations[1].PreOperation = K2PreOperationCallback;

RtlZeroMemory(&registration, sizeof(registration));
registration.Version = ObGetFilterVersion();
registration.OperationRegistrationCount = RTL_NUMBER_OF(g_OperationRegistrations);
registration.Altitude = g_Altitude;
registration.OperationRegistration = g_OperationRegistrations;

status = ObRegisterCallbacks(&registration, &g_ObRegistrationHandle);
  • PsSetCreateProcessNotifyRoutineEx for process creation
  • PsSetCreateThreadNotifyRoutine for thread creation
  • ObRegisterCallbacks for process and thread handle create / duplicate operations

Callback Events

Each callback maps to an event specification.

That specification tells K2 which event is being inspected, whether the top user frame must strictly match one of the expected exports, and which exports are valid for that event.

static const K2_EVENT_SPEC g_ProcessCreateSpec = {
    "process-create",
    FALSE,
    1,
    { "NtCreateUserProcess" }
};

static const K2_EVENT_SPEC g_ThreadCreateSpec = {
    "thread-create",
    FALSE,
    2,
    { "NtCreateThreadEx", "NtCreateThread" }
};

static const K2_EVENT_SPEC g_ProcessOpenSpec = {
    "process-open",
    TRUE,
    3,
    { "NtOpenProcess", "NtCreateUserProcess", "ZwAlpcOpenSenderProcess" }
};

static const K2_EVENT_SPEC g_ThreadOpenSpec = {
    "thread-open",
    TRUE,
    4,
    { "NtOpenThread", "NtCreateUserProcess", "NtCreateThreadEx", "ZwAlpcOpenSenderThread" }
};

Annoyingly Windows has like 5 entrypoints to some of the same system calls, it's tempting to say "a process handle open should always come from NtOpenProcess", but no. Windows has legitimate paths that produce process and thread handles through other syscall exports.

For example:

  • A process handle can legitimately arrive through NtOpenProcess
  • A process handle can also arrive through NtCreateUserProcess
  • ALPC sender APIs can produce process or thread handles too
  • Thread handles can naturally arrive through NtOpenThread, NtCreateThreadEx, NtCreateUserProcess, or ZwAlpcOpenSenderThread

If a detector treats every non-NtOpen* origin as malicious, it creates noise from normal Windows activity, WMI, CLR, CSRSS, CTF, and other boring but complicated system paths.

So K2 does not just ask "is this NtOpenProcess?"

It asks "is this one of the syscall exports that can reasonably produce this event?"

From Callback to Stack Inspection

The callbacks themselves do almost nothing, they check the event conditions, verify the current IRQL is safe, then call the analyzer.

Process creation:

VOID
K2ProcessNotifyEx(
    _Inout_ PEPROCESS Process,
    _In_ HANDLE ProcessId,
    _Inout_opt_ PPS_CREATE_NOTIFY_INFO CreateInfo
    )
{
    UNREFERENCED_PARAMETER(Process);

    if (CreateInfo == NULL) {
        K2InvalidateProcessModuleCache(ProcessId);
        return;
    }

    if (KeGetCurrentIrql() > PASSIVE_LEVEL) {
        return;
    }

    K2InspectCurrentThread(&g_ProcessCreateSpec);
}

Thread creation:

VOID
K2ThreadNotify(
    _In_ HANDLE ProcessId,
    _In_ HANDLE ThreadId,
    _In_ BOOLEAN Create
    )
{
    UNREFERENCED_PARAMETER(ProcessId);
    UNREFERENCED_PARAMETER(ThreadId);

    if (!Create || KeGetCurrentIrql() > PASSIVE_LEVEL) {
        return;
    }

    K2InspectCurrentThread(&g_ThreadCreateSpec);
}

Object pre-operation callbacks:

if (OperationInformation->KernelHandle || OperationInformation->Object == NULL) {
    return OB_PREOP_SUCCESS;
}

if (OperationInformation->ObjectType != *PsProcessType &&
    OperationInformation->ObjectType != *PsThreadType) {
    return OB_PREOP_SUCCESS;
}

if (KeGetCurrentIrql() > PASSIVE_LEVEL) {
    return OB_PREOP_SUCCESS;
}

isProcess = (OperationInformation->ObjectType == *PsProcessType);
if (OperationInformation->Operation == OB_OPERATION_HANDLE_CREATE) {
    K2InspectCurrentThread(isProcess ? &g_ProcessOpenSpec : &g_ThreadOpenSpec);
} else if (OperationInformation->Operation == OB_OPERATION_HANDLE_DUPLICATE) {
    K2InspectCurrentThread(isProcess ? &g_ProcessDuplicateSpec : &g_ThreadDuplicateSpec);
}

That PASSIVE_LEVEL check exists because the analysis path is not just a few register comparisons. K2 may attach to the current process, walk usermode loader lists, probe user memory, query virtual memory information, and parse PE export tables.

That is not work you do at arbitrary IRQL.

Capturing the User Stack

K2's first real signal is the user stack.

ULONG
K2CaptureUserFrames(
    _Out_writes_(FrameCount) PVOID* Frames,
    _In_ ULONG FrameCount
    )
{
    __try {
        return RtlWalkFrameChain(Frames, FrameCount, K2_SYSCALL_STACK_FLAG_USER_MODE);
    } __except (EXCEPTION_EXECUTE_HANDLER) {
        return 0;
    }
}

K2_SYSCALL_STACK_FLAG_USER_MODE is 0x1, which tells RtlWalkFrameChain to return usermode frames.

The analyzer stores up to 16 frames:

frameCount = K2CaptureUserFrames(frames, RTL_NUMBER_OF(frames));
if (frameCount == 0) {
    Analysis->UserStackUnavailable = TRUE;
    return FALSE;
}

Analysis->FrameCount = frameCount;
RtlCopyMemory(Analysis->Frames, frames, sizeof(PVOID) * frameCount);
Analysis->Frame0 = frames[0];
Analysis->Frame1 = frameCount > 1 ? frames[1] : NULL;
Analysis->Frame0InvalidUserAddress = !K2IsLikelyUserAddress(Analysis->Frame0);
if (Analysis->Frame0InvalidUserAddress) {
    return Spec->StrictExportMatch;
}

K2 cares about two frames immediately:

  • Frame0: the top user frame, which should usually be the syscall-facing frame
  • Frame1: the caller behind that frame, which is useful for memory classification

If Frame0 is not a plausible usermode address, strict events can be flagged immediately.

For non-strict events, K2 is more conservative. This is a PoC, not a "log everything weird and drown in your own telemetry" speedrun.

Finding The Real ntdll

Once K2 has a top user frame, it needs the current process' real ntdll.dll base.

This matters because simply checking an address against a hardcoded module base is useless. Every process has its own user address layout. ASLR exists. The process may have win32u.dll loaded. The user stack may reference image-backed regions that are not in a simple global table.

K2 resolves ntdll from the current process PEB:

PVOID
K2GetCurrentProcessNtdllBase(
    VOID
    )
{
    PEPROCESS process;
    PVOID ntdllBase;

    process = PsGetCurrentProcess();
    if (process == NULL || PsGetProcessWow64Process(process) != NULL) {
        return NULL;
    }

    if (K2LookupProcessModuleCache(PsGetCurrentProcessId(), &ntdllBase, NULL)) {
        return ntdllBase;
    }

    if (!K2PopulateCurrentProcessModuleCache(process, PsGetCurrentProcessId(), &ntdllBase, NULL)) {
        return NULL;
    }

    return ntdllBase;
}

The current build is focused on native x64 processes. If PsGetProcessWow64Process returns a WoW64 process, K2 does not try to analyze that path.

The module lookup attaches to the target process and walks the PEB loader list:

peb = (PK2_PEB)PsGetProcessPeb(Process);
if (peb == NULL) {
    return NULL;
}

KeStackAttachProcess(Process, &apcState);
__try {
    ProbeForRead(peb, sizeof(*peb), sizeof(UCHAR));
    ldr = peb->Ldr;
    if (ldr == NULL) {
        __leave;
    }

    ProbeForRead(ldr, sizeof(*ldr), sizeof(UCHAR));
    head = &ldr->InMemoryOrderModuleList;
    for (entry = head->Flink; entry != head; entry = entry->Flink) {
        PK2_LDR_DATA_TABLE_ENTRY module;

        ProbeForRead(entry, sizeof(*entry), sizeof(UCHAR));
        module = CONTAINING_RECORD(entry, K2_LDR_DATA_TABLE_ENTRY, InMemoryOrderLinks);
        ProbeForRead(module, sizeof(*module), sizeof(UCHAR));
        if (module->DllBase == NULL || module->FullDllName.Buffer == NULL) {
            continue;
        }

        ProbeForRead(module->FullDllName.Buffer, module->FullDllName.Length, sizeof(WCHAR));
        if (K2EndsWithUnicodeInsensitive(&module->FullDllName, ModuleName)) {
            result = module->DllBase;
            break;
        }
    }
} __except (EXCEPTION_EXECUTE_HANDLER) {
    result = NULL;
}
KeUnstackDetachProcess(&apcState);

There are three details worth calling out.

First, K2 uses KeStackAttachProcess, because the PEB and loader list are usermode data in the process address space.

Second, every user pointer is probed under structured exception handling. Usermode lies. Usermode memory can disappear. Kernel code that trusts it deserves what happens next.

Third, K2 caches ntdll and win32u bases per process:

#define K2_MODULE_CACHE_SLOTS 16

typedef struct _K2_PROCESS_MODULE_CACHE_ENTRY {
    HANDLE ProcessId;
    PVOID NtdllBase;
    PVOID Win32uBase;
} K2_PROCESS_MODULE_CACHE_ENTRY, *PK2_PROCESS_MODULE_CACHE_ENTRY;

Process exit invalidates the cache:

if (CreateInfo == NULL) {
    K2InvalidateProcessModuleCache(ProcessId);
    return;
}

Resolving Exports

Ohhh but stack spoofing exists, so I can't just stop at ntdll.dll.

Is Frame0 inside ntdll.dll?

An operator can try to make the top user frame appear to land inside ntdll even though the actual syscall path was custom. If the defender only checks "inside ntdll", the spoof works.

K2 goes one layer deeper and resolves which exported func contains the frame.

The analyzer resolves the expected exports for the current event:

static
VOID
K2ResolveExpectedExports(
    _In_ PVOID NtdllBase,
    _In_ const K2_EVENT_SPEC* Spec,
    _Out_ PK2_SYSCALL_ANALYSIS Analysis
    )
{
    ULONG i;

    for (i = 0; i < Spec->ExportCount && i < RTL_NUMBER_OF(Analysis->ExpectedExports); ++i) {
        (VOID)K2TryResolveExpectedExport(NtdllBase, Spec->Exports[i], &Analysis->ExpectedExports[i]);
    }
}

Then it resolves the actual top frame:

if (K2ResolveExportForAddress(Analysis->NtdllBase, Analysis->Frame0, &exportBase, &exportSpan, Analysis->ResolvedExportName)) {
    Analysis->Frame0InNtdll = TRUE;
    Analysis->ResolvedExportBase = exportBase;
    Analysis->ResolvedExportSpan = exportSpan;
    Analysis->Frame0InExpectedExport = !Spec->StrictExportMatch || K2FrameMatchesExpectedExport(Analysis);
    Analysis->Frame0InDifferentNtdllExport = Spec->StrictExportMatch && !Analysis->Frame0InExpectedExport;
}

K2ResolveExportForAddress parses the module export directory, finds the nearest export RVA at or before the frame RVA, then computes a span using the next export RVA.

That means K2 can say:

  • Frame0 is inside ntdll
  • Frame0 resolves to NtClose
  • This event expects NtOpenProcess
  • Therefore this is an indirect or spoofed export path

The matching check is simple:

static
BOOLEAN
K2FrameMatchesExpectedExport(
    _In_ const K2_SYSCALL_ANALYSIS* Analysis
    )
{
    ULONG i;

    for (i = 0; i < Analysis->ExpectedExportCount; ++i) {
        if (K2AddressInRange(Analysis->Frame0, Analysis->ExpectedExports[i].Base, Analysis->ExpectedExports[i].Span)) {
            return TRUE;
        }
    }

    return FALSE;
}

If the event is a process open and the top user frame lands in NtAllocateVirtualMemory, it doesn't take a genius to figure that out. K2 doesn't need a usermode hook to notice that. The callback already fired .

Caller Memory Classification

Direct syscall stubs usually have an address problem.

If the stub is not in ntdll, where is it?

Probably:

  • The malware image
  • Shellcode in private executable memory
  • RWX memory
  • JIT-style memory
  • A manually mapped image that does not appear as normal MEM_IMAGE

K2 classifies the caller frame with ZwQueryVirtualMemory:

BOOLEAN
K2QueryAddressMemory(
    _In_ PVOID Address,
    _Out_ PMEMORY_BASIC_INFORMATION Mbi
    )
{
    SIZE_T returnedLength;
    NTSTATUS status;

    RtlZeroMemory(Mbi, sizeof(*Mbi));
    if (!K2IsLikelyUserAddress(Address)) {
        return FALSE;
    }

    status = ZwQueryVirtualMemory(
        ZwCurrentProcess(),
        Address,
        MemoryBasicInformation,
        Mbi,
        sizeof(*Mbi),
        &returnedLength);

    return NT_SUCCESS(status) &&
           returnedLength >= sizeof(*Mbi) &&
           Mbi->State == MEM_COMMIT;
}

static
VOID
K2PopulateCallerAnalysis(
    _Inout_ PK2_SYSCALL_ANALYSIS Analysis
    )
{
    if (Analysis->Frame1 == NULL || !K2QueryAddressMemory(Analysis->Frame1, &Analysis->CallerMbi)) {
        return;
    }

    Analysis->CallerFrameExecutable = K2IsExecutableProtection(Analysis->CallerMbi.Protect);
    Analysis->CallerFramePrivate = ((Analysis->CallerMbi.Type & MEM_PRIVATE) != 0);
    Analysis->CallerFrameWritable = K2IsWritableProtection(Analysis->CallerMbi.Protect);
    Analysis->CallerFrameImage = ((Analysis->CallerMbi.Type & MEM_IMAGE) != 0);
}
  1. Frame0 points at a small syscall stub outside ntdll
  2. Frame1 points back into executable private memory
  3. The event is a process/thread/open/duplicate operation
  4. K2 records the event as suspicious

This also catches softer variants where Frame0 appears to be in ntdll, but the caller behind it is still executable private memory or writable executable memory.

That is an important distinction. A stack spoof that only fixes the first frame is useless.

Detection Policy

The final policy is simple;

static
BOOLEAN
K2ShouldFlagAnalysis(
    _In_ const K2_SYSCALL_ANALYSIS* Analysis
    )
{
    if (!Analysis->StrictExportMatch) {
        if (!Analysis->Frame0InNtdll && !Analysis->Frame0InWin32u) {
            return Analysis->CallerFrameExecutable &&
                   (Analysis->CallerFramePrivate || Analysis->CallerFrameWritable || !Analysis->CallerFrameImage);
        }

        return Analysis->CallerFrameExecutable && K2IsSuspiciousCallerMemory(&Analysis->CallerMbi);
    }

    if (!Analysis->Frame0InNtdll) {
        if (Analysis->Frame0InWin32u) {
            return Analysis->CallerFrameExecutable && K2IsSuspiciousCallerMemory(&Analysis->CallerMbi);
        }

        return TRUE;
    }

    if (Analysis->Frame0InDifferentNtdllExport) {
        return TRUE;
    }

    return Analysis->CallerFrameExecutable && K2IsSuspiciousCallerMemory(&Analysis->CallerMbi);
}

Detection Reasons

K2 turns the analysis into short reason labels.

  • frame0-outside-ntdll: the syscall-facing frame is not in ntdll or win32u
  • indirect-export: the frame is in ntdll, but not in the expected export
  • private-caller: the caller frame is in private memory
  • wx-caller: the caller frame is writable executable
  • non-image-exec: the caller is executable but not image-backed
  • no-ntdll: K2 could not resolve the current process' ntdll
  • invalid-user-frame: the captured frame does not look like a valid user address

Conclusion

Direct syscalls suck stop overcomplicating move on to something better.

No but seriously, the wrong thing is being focused on. Hiding your API calls is next to useless when all of your operations are routed via the kernel regardless, and hiding them and having the kernel notice provides more of a detection then it would've without any sort of hide at all.

Bringing in encryption, stubs, private-memory mapping, double/late-load ntdll.dll, weird export/unwind dir parsing, unhooking and all of these different TTP's floods your binary with fun new detections.

K2 was written as a prelude to Blackbirds internal syscall detection system, publicized because I wanted to focus on such a common misconception, and I know me & a lot of fellow offsec/defsec researchers are tired of seeing a new syscall implementation weekly.