0xmrerror@terminal:~/writeups

WinAntiDbg0x300 — UPX, a Watchdog, and an Infinite Loop

UPX unpacking, a child-process watchdog and an infinite loop that never actually exits. A full GDB walkthrough.

MediumpicoCTFReverse engineering · 15 min readupxanti-debuggdbchild-processwindowsreveng

The finale of the series fights back: a UPX-packed GUI binary that spawns a child watchdog every ~5 seconds to attach to — and kill — your debugger, then hides the flag behind an infinite loop. Register patching won't cut it here. Goal: unpack, patch the loop, and let the flag fall out.

1 · Unpack

file WinAntiDbg0x300.exe
# PE32 executable (GUI) Intel 80386, for MS Windows, UPX compressed

# Also included: a .pdb (debug symbols) and a config.bin
upx -d WinAntiDbg0x300.exe -o unpacked.exe

2 · Static Analysis with the PDB

Import the unpacked binary in Ghidra and attach the supplied .pdb (File → Add to Program) — original symbol names come back for free. Trace the flow from wWinMain:

PrintDbgBanner → ReadConfig → ComputeHash(3)
DetectDebuggerAtLaunch   # NtQueryInformationProcess(ProcessDebugFlags) + PEB.BeingDebugged
EnableDebugPrivilege
ManageChildProcess       # the "fighting back" part
MyRegisterClass → CreateThread(ChallengeThreadFunction, ...)

ManageChildProcess creates a child of the same binary with the parent's PID as its argument; the child calls DebugActiveProcess on the parent and TerminateProcess your debugger. That's why attaching a debugger just gets you bounced.

3 · The Watchdog Loop

ChallengeThreadFunction never returns — a do { CreateProcess; wait; Sleep(5000); } while(true) that re-tests every ~5 seconds (child exit code 0xfe aborts, 0xfd shows a warning box). Just below the loop sits the flag-decrypt code — unreachable because the loop's closing JMP is unconditional.

# documented landmarks in ChallengeThreadFunction
0x004037c0  loop head: CreateProcessA / WaitForSingleObject / Sleep(5000)
     ...    unconditional JMP back to 0x004037c0   ← the jump to kill
0x004038e0  flag decryption (unreachable while the JMP stays)
0x00403929  OutputDebugStringW — "you got the flag"

4 · Patch the JMP to NOPs

In Ghidra's Listing, right-click the unconditional JMP at the end of the loop → Clear Code Bytes → select the bytes → Ctrl+Shift+G → Patch Instruction, and write NOP over each byte (JMP rel32 is 5 bytes). Execution now falls through straight into the decrypt path.

# in place: Clear Code Bytes → Patch Instruction → NOP × 5
E9 9B FF FF FF   →   90 90 90 90 90    # JMP rel32 → five NOPs

Export the patched binary with File → Export Program → Windows PE — the 'Original File' export silently drops in-place patches.

5 · Run Under DebugView

The binary still kills attached debuggers, so observe passively: start DebugView (Microsoft Sysinternals, 32-bit mode) and run the patched executable as administrator. The ComputeHash rounds take about 5 seconds, then the flag appears both in a message box and in DebugView's output pane.

# DebugView output (the OutputDebugString stream)
ComputeHash...
### you got the flag
picoCTF{Wind0ws_antid3bg_0x300_<HASH>}
# the trailing hex varies per build

6 · What Made It Work

Step 2 saves most of the work. The challenge ships its own .pdb, so ManageChildProcess and ChallengeThreadFunction arrive named instead of as FUN_00403xxx, and locating the loop stops being guesswork.

The rest is that a binary which kills attached debuggers stops being a problem the moment you stop attaching. The watchdog targets the debugger, not the process, so patching the loop and watching OutputDebugStringW from outside wins without a fight.