0xmrerror@terminal:~/writeups

Keygenme — MD5 Hex Spliced Into a License Key

The binary assembles the full 36-character key on the stack out of two MD5 digests — so the lazy solve is to break in and read it out.

HardpicoCTFReverse engineering · 18 min readmd5keygenreverse-engineeringgdbstackpicoctf

"Can you get the flag?" — picoCTF 2022 is done teasing: most RE challenges ask you to break a check, this one asks you to produce the key that passes it. The catch is the binary already knows the full 36-character key: it assembles it on the stack out of a hardcoded prefix, two MD5 digests and a closing brace, right before comparing it against your input. So the laziest solve isn't writing a keygen at all — it's breaking inside the checker and string-searching the fully-built key from memory.

1 · First Run — It Wants a License Key

gdb keygenme
gef➤ r
Starting program .../keygenme
Enter your license key: AAAAAAAAAA
That key is invalid.

No flag anywhere in sight. The binary is stripped, so there is no main symbol for GDB to break *main on — but the very first challenge is relocating the interesting code.

2 · Finding main in a Stripped Binary

Trick: break on printf — every program prints something before doing work — and let the call stack walk back to the caller:

gef➤ b *printf
Breakpoint 1 at 0x10b0
gef➤ r
...
#0  __printf (format="Enter your license key: ") at printf.c:28
#1  0x5555555554be → mov rdx, [rip+0x2b4b]  # <stdin>
#2  0x7ffff7b16d0a → __libc_start_main(main=0x55555555548b, ...)

__libc_start_main exposes main = 0x55555555548b. Finish the printf, single-step twice, and the very next branch of code is the license-checker call:

gef➤ finish
gef➤ si
gef➤ si
→ 0x5555555554c9  mov esi, 0x25        # fgets(buf, 0x25, stdin)
   0x5555555554ce  mov rdi, rax
   0x5555555554d1  call fgets@plt
   0x5555555554dd  call 0x555555555209   # ← sub_1209(&buf) — the real work
   0x5555555554e2  test al, al

So main slurps up to 0x25 (37) bytes with fgets and funnels the buffer into the stripped function at 0x555555555209. That function is the whole game.

3 · Decompile the Checker

Binary Ninja recovers the function formerly known as sub_1209:

00001209 int64_t sub_1209(char* arg1)

00001242 var_98 = 0x7b4654436f636970     # "picoCTF{"
00001249 var_90 = 0x30795f676e317262     # "br1ng_y0"
0000125a var_88 = 0x6b5f6e77305f7275     # "ur_0wn_k"
0000125e var_80 = 0x5f7933               # "3y_"
00001265 var_ba = 0x7d                  # "}"

00001278 MD5(&var_98, strlen(&var_98), &var_b8, ...)   # MD5("picoCTF{br1ng_y0ur_0wn_k3y_")
000012bf MD5(&var_ba, strlen(&var_ba), &var_a8, ...)   # MD5("}")

# hex-encode the first digest, 16 bytes → 32 hex chars into var_78
00001328 while (var_cc <= 0xf) {
    sprintf(&var_78[var_d0], "%02x", var_b8[var_cc]);
    var_cc++; var_d0 += 2; }
# hex-encode the second digest into var_58
0000138e while (var_c8 <= 0xf) {
    sprintf(&var_58[var_d0_1], "%02x", var_a8[var_c8]);
    var_c8++; var_d0_1 += 2; }

# copy the 27-byte text prefix into the output key var_38
000013c6 for (var_c4 = 0; var_c4 <= 0x1a; var_c4++)
    var_38[var_c4] = var_98[var_c4];
# nine key positions overwritten from the MD5 hex strings + a "}"
var_15 = var_6f;  var_1c = var_78;  var_1b = var_5f;  var_1a = var_5e;
var_19 = var_5b;  var_18 = var_43;  var_17 = var_71;  var_16 = var_5f;
var_15 = var_ba.b;                                     # tail "}"

0000141d if (strlen(arg1) != 0x24)          # input must be exactly 36 chars
00001426 else while (true) {
00001457     if (arg1[var_c0_1] != var_38[var_c0_1]) break;  # byte-at-a-time compare
00001460     var_c0_1++; }
return (loop ran 36 times);

4 · What the Code Is Actually Doing

This is a key generator, not a key verifier — the correct key is synthesized on the stack at runtime:

  • Text prefix (27 bytes): the movabs constants spell picoCTF{br1ng_y0ur_0wn_k3y_. It is copied wholesale into the output buffer.
  • MD5(prefix) and MD5("}"): each digest is hex-encoded (%02x) into 32-char strings. Only a handful of those characters end up in the key.
  • Nine splices: the assembly later in this writeup copies nine bytes into key positions 27–35 — a mix of characters from the two hex strings and a literal } — producing the 36th (final) character.
  • The gate: strlen(arg1) != 0x24 rejects any key that isn't exactly 36 bytes, then a simple per-index compare loop validates your input byte-for-byte against the synthesized key.

Because the entire key exists in memory, you are one debugger command away from it.

5 · The GDB Disassembly — Where Everything Is

Break right at the top of the checker, then dump the whole function (display/120i $pc or layout asm). The relevant region:

gef➤ b *0x555555555209
gef➤ c
Enter your license key: AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA
→ 0x555555555209  endbr64
   0x55555555522e  movabs rax,0x7b4654436f636970   # "picoCTF{"
   0x555555555238  movabs rdx,0x30795f676e317262   # "br1ng_y0"
   0x555555555250  movabs rax,0x6b5f6e77305f7275   # "ur_0wn_k"
   0x55555555525e  mov DWORD PTR [rbp-0x78],0x5f7933   # "3y_"
   0x555555555265  mov WORD PTR [rbp-0xb2],0x7d       # "}"
   # ── MD5(prefix) and MD5("}") ──
   0x555555555294  call 0x5555555550f0 <MD5@plt>
   ...
   0x5555555552bf  call 0x5555555550f0 <MD5@plt>
   # ── hex-encode digest #1 (loop 0..0xf) ──
   0x5555555552da  movzx eax,BYTE PTR [rbp+rax*1-0xb0]
   0x55555555530e  call sprintf@plt            # "%02x"
   # ── hex-encode digest #2 (loop 0..0xf) ──
   0x555555555340  movzx eax,BYTE PTR [rbp+rax*1-0xa0]
   0x555555555374  call sprintf@plt            # "%02x"
   # ── copy 27-byte prefix into the key at rbp-0x30 ──
   0x55555555539c  movzx edx,BYTE PTR [rbp+rax*1-0x90]
   0x5555555553b4  mov BYTE PTR [rbp+rax*1-0x30],dl    # key[i] = prefix[i]
   # ── the nine splices (key[27]..key[35]) ──
   0x5555555553c8  movzx eax,BYTE PTR [rbp-0x67] ; mov [rbp-0x15],al
   0x5555555553cf  movzx eax,BYTE PTR [rbp-0x70] ; mov [rbp-0x14],al
   0x5555555553d6  movzx eax,BYTE PTR [rbp-0x57] ; mov [rbp-0x13],al
   0x5555555553dd  movzx eax,BYTE PTR [rbp-0x56] ; mov [rbp-0x12],al
   0x5555555553e4  movzx eax,BYTE PTR [rbp-0x53] ; mov [rbp-0x11],al
   0x5555555553eb  movzx eax,BYTE PTR [rbp-0x3b] ; mov [rbp-0x10],al
   0x5555555553f2  movzx eax,BYTE PTR [rbp-0x69] ; mov [rbp-0x0f],al
   0x5555555553f9  movzx eax,BYTE PTR [rbp-0x57] ; mov [rbp-0x0e],al
   0x555555555400  movzx eax,BYTE PTR [rbp-0xb2] ; mov [rbp-0x0d],al  # '}'
   # ── the length gate and per-byte compare ──
   0x555555555411  call strlen@plt
   0x555555555419  cmp rax,0x24                # ← break here
   0x55555555541d  je  0x555555555426
   0x55555555541f  mov eax,0x0               # invalid
   ...
   0x555555555432  movzx edx,BYTE PTR [rax]  # our byte
   0x555555555450  movzx eax,BYTE PTR [rbp+rax*1-0x30]  # expected byte

Note the destination offsets rbp-0x30 + 0x1b ... rbp-0x30 + 0x23: those are key positions 27 through 35, exactly the gap after the 27-char prefix. The tail characters are drawn from the two MD5 hex buffers — hex1[0], hex1[7], hex1[9], hex2[1], hex2[2], hex2[5], hex2[29], hex1[9] again — finishing with the literal }.

6 · Break, Then Search the Stack

Now that the display buffer is in place, all that's left is to break before the compare and search live memory for the assembled string. GEF's search-pattern scans loaded modules and the stack in one shot:

gef➤ b *0x555555555419
gef➤ c
...
→ 0x555555555419 cmp rax, 0x24
gef➤ search-pattern picoCTF
[+] Searching 'picoCTF' in memory
[+] In 'keygenme'(0x555555555000-0x555555556000), permission=r-x
    0x555555555230 - 0x555555555237 → "picoCTF[...]"
[+] In '[stack]'(0x7ffffffde000-0x7ffffffff000), permission=rw-
    0x7fffffffde00 - 0x7fffffffde1b → "picoCTF{br1ng_y0ur_0wn_k3y_"
    0x7fffffffde60 - 0x7fffffffde67 → "picoCTF[...]"
gef➤ x/s 0x7fffffffde60
0x7fffffffde60: "picoCTF{br1ng_y0ur_0wn_k3y_247d8a57}\377\177"

The first hit is the hardcoded picoCTF{ in the executable's own memory; the stack hits are the runtime-assembled key. The 36x bytes –0xde60.. print almost perfectly (the trailing \377\177 is just the stack's adjacent garbage). Typing that string back into the license prompt flips the verdict to valid.

7 · The Flag

picoCTF{br1ng_y0ur_0wn_k3y_247d8a57}
# baked into the binary → static across all instances

8 · What Made It Work

The premise is what does the work. This binary doesn't verify a key, it builds one and then checks it, so the answer is on the stack a moment before the compare. Recognizing that turns search-pattern into a legitimate extraction primitive rather than a hack, and the two MD5s never had to be computed by hand because the program computed them for us.

Finding the code in a stripped binary comes down to __libc_start_main's hidden main argument, or a printf breakpoint plus finish to walk the caller.