Torturette

It does not come to check your program. It comes to confirm what you already suspected.

Torturette is not just another tool. It is the elegant way of telling a program that, however proud it may feel, it is still vulnerable to the most basic errors. While other programs stroll through the world with their malloc and their free as if they were gods, Torturette walks in, takes off their mask and demonstrates to them, with a mixture of technical coldness and well-organised bad temper, that a small memory failure can be enough to turn a brilliant run into a perfectly documented disaster.

Because of course, there are people who write software as if it were all child's play. You compile, you run, you check that there are no warnings, and you already believe you own the universe. You even go so far as to think that "it compiles first time" is a synonym for "it is well written", which are completely different things and only one of them is true in your case. Then along comes Torturette, with the enthusiasm of someone who knows the program is going to fall, and shows the end user that their binary was not as invincible as it claimed to be.

It is not that Torturette has anything personal against you. It is that your code has not given it any reason to trust it either. Leaks, double frees, aborts, invalid accesses, fatal signals and other small displays of low-level incompetence: all of it ends up exposed with a clarity that makes the experience quite uncomfortable for anyone who thought their code was "almost stable". Almost stable is, in the opinion of the person writing this, the most elegant way there is of saying "unstable", only with less shame.

You are going to try to defend yourself. You are going to say that "that never happens in production", that "that branch of the code never runs", that "I already had that under control". Torturette has heard that same sentence from hundreds of different binaries, all of them now resting in a core dump. It does not argue with you. It simply runs, applies pressure, and lets the facts speak, since they are far less touchy than you are.

In short, Torturette does not aim to be pretty. It aims to be relentless. It is the kind of tool that asks the uncomfortable question of a program that thought it was fine: "And what happens to this when something really breaks?" And the answer, almost always, is a pile of obvious uselessness that is very hard to ignore.

![note] ONE note on the difference between valgrind and and valgrind observes one execution. We run our program, it does whatever it does, and Valgrind reports the leaks and invalid accesses that actually happened on that run. If our allocations all succeed - which they essentially always do on a normal machine. the failure branches in our code are never executed. Valgrind cannot report a bug in code that never ran. Torturette forces these branches ro run. For every allocation the target makes, it forks at the exact call site, make that one call return NULL, and lets the copy continue. That's 141 separate executions for the line reader, one per allocation, each exploring a > path valgrind would never reach. The bsq case is the concrete example. Under valgrind on a valid map, bsq is clean: 16 allocations, 16 frees, nothing lost. Torturette failing allocation #2 showed 32 bytes leaked - because on that error path the code returns without freeing the rows it already built. Valgrind would only find that if we could somehow make the second calloc fail, which normally means an out-of-memory The same held for get_next_line. It passes Valgrind clean, Writing one int past a heap block surived every mutation run - Torturette has no redzones and injects allocation failures rather instrumenting memory. valgrind catches that immediately and torturette never will. They're two different toold that are complement. Valgrind check that the path we took was memory correct. Two honest caveats though. Torturette is roughly cubic in live allocations, so 6400 allocations take about three minutes. And its leak detection compares total byte against the baseline, which can mask a real leak when a baseline leak is allocated late.

Index

• What it does
• How to use it
• What it offers
• Technical side
• Continuous integration
• Limitations
• Quick test

What it does

Torturette's workflow is relatively simple on the surface, but rather aggressive in practice. The program:

1. shows an initial banner and prepares the execution environment;
2. runs the target binary in a normal mode in order to observe its base behaviour;
3. launches a series of additional runs in the background, forcing allocation or memory-handling failure situations;
4. analyses the results and generates a report with the suspicious cases, the most relevant events, and the points where the failure appears to have occurred.

It does not limit itself to "retrying" innocently. What it does is subject the program to deliberate pressure in order to check whether it holds up or whether, on the contrary, it leaves evidence of its fragility in the form of errors, signals, aborts or undefined behaviour. Torturette is not curious, it is methodical.

How to use it

Torturette builds on its own libft, which lives in a separate repository. Clone it into the project root first:

git clone git@github.com:Univers42/libft.git libft

Then build the project with:

make

make compiles libft itself (into libft/build/lib/libft.a) before building torturette and the injector, so there is no separate step to remember. If the libft directory is missing, the Makefile says so and stops rather than failing with a wall of linker errors.

If something does not compile, it is not the Makefile's fault. Check your code first, then go back to blaming the Makefile if it makes you feel better.

make torture, the way this is meant to be used

Forget about preparing anything by hand. The idea behind Torturette is not that you point it at an already compiled binary and cross your fingers: it is that you leave your own code, the code you are writing right now, lying in the root of the project, and give it straight to:

make torture

With that, Torturette compiles all the .c files in the root, puts the result together into a single binary and immediately subjects it to the full round of memory failures. No intermediate steps, no having to remember to compile beforehand, no excuses along the way. One command, and your code is already being put in its place.

And if your program needs arguments, you pass them just like that, with no odd variables or syntax in between:

make torture arg1 arg2

That is all. Whatever you write after torture reaches your binary intact, as if you had run it yourself, only this time somebody else is looking over your shoulder to see whether it holds up. If you do not leave any .c in the root, Torturette is not going to invent a culprit: it will simply tell you that there is nothing to torture, because even for that you have to contribute something.

This is the flow intended for daily use: you write, make torture, you read the report, you cry a little, you fix it, you repeat. The rest of this section is for anyone who wants to do it more manually, or for anyone who has not yet understood that make torture already sorts it all out for them.

Manual use

If you prefer to compile the target binary yourself and pass it directly to Torturette, you can do that too:

./torturette <target_program> [arguments...]

For example:

./torturette tests/try_me/try_me

The rest of the Makefile targets

make          # builds torturette and the injector
make try      # compiles tests/try_me and subjects it to torturette
make bsq      # same for tests/bsq, on a valid map
make gnl      # same for tests/gnl, on a mixed fixture
make tests    # builds all three test programs and runs nothing
make ci       # the whole gate: every program on every fixture, then the mutation scores
make mutate   # mutation-tests all three (mutate-try / mutate-bsq / mutate-gnl for one)
make torture  # compiles ALL the .c files in the root and subjects them to torturette
make silence  # like torture, but with torturette keeping quiet.
make clean    # cleans up object files and test binaries
make fclean   # cleans everything. Not your code. Sometimes one has to be a decent person
make re       # fclean + all, in case you think that rebuilding it will fix your code
make help     # in case all of this has not made it clear enough

What it offers

• Normal execution of the target program.
• Automatic memory failure tests (malloc/calloc/realloc) and thread creation tests (pthread_create).
• Detection of problems such as:
  • leaks,
  • double frees,
  • crashes by signal,
  • aborts,
  • invalid accesses,
  • anomalous execution states,
  • threads that hang forever when something basic (memory, a thread) fails and your code was not expecting it.
• A final report with information useful for locating the problematic point.
• Integration with a simple test program for simulating failures manually.

Technical side

Torturette is designed as an analysis tool oriented towards execution and towards the observation of anomalous behaviour. Its design is based on a clear separation between the main orchestrator, the execution modules and the reporting modules.

General structure

The project is made up of several modules:

• main.c: coordinates the overall execution flow.
• engine.c / engine.h: encapsulate the logic for running the target program and managing the test cases.
• report.c / report.h: take care of showing the banner, summarising the results and presenting the failure reports.
• injector.c: implements the logic related to injecting test conditions and to the behaviour of the failure cases.
• common.h: contains the definitions and structures shared across the whole system.
• libft/: the project's own C library (git@github.com:Univers42/libft.git), built by the Makefile and used throughout instead of the standard library equivalents -- ft_printf, ft_snprintf, ft_strlen, ft_strdup, ft_strtok and the rest.
• tests/: one directory per victim program -- try_me (the interactive example), bsq (a BSQ solver reading its map with libft's get_next_line) and gnl (a bare get_next_line reader) -- each with its fixtures and a tools/mutate.sh. They deliberately stay on the plain standard library apart from the line reader, because they stand in for the victim: ordinary student programs using ordinary malloc and free.

Execution model

The flow can be understood as a combination of a base analysis and a stress phase. In the first, Torturette runs the target program in the usual way in order to obtain a behavioural reference. In the second, it enables artificial failure conditions in order to verify whether the binary behaves robustly or whether it leaves signs of hidden problems.

This strategy makes it possible to distinguish between:

• failures that appear immediately,
• failures that only show up under pressure,
• behaviour that looks correct but leaves resources badly managed,
• runs that end abruptly and must be reported as signal failures or aborts.

Handling of results

When a test detects a suspicious case, the tool tries to classify it and summarise it in a readable way. The report includes information useful for diagnosis: the type of problem, the execution context and, where possible, backtrace details or other data derived from running the target binary.

This approach is useful because it turns a set of raw failures into a sequence of events that can be interpreted far more quickly. In other words, you not only "see" that something exploded, but an attempt is also made to explain what kind of error it was, where it may have happened and what sort of behaviour produced it.

Threads that never wake up

pthread_create() can fail just like malloc() -- normally because the system has exhausted the user's thread/process limit (ulimit -u), something far easier to run into on a shared computer or in a restricted container than on your own machine. Torturette tests this too: for every real pthread_create() the target binary makes, it checks what would happen if that particular thread creation failed.

This is where the problem that really matters shows up: if your program does not check that failure, or checks it but some other already running thread ends up waiting (on a mutex, a semaphore, a barrier) for that thread which never came into existence, your program falls asleep forever. Without a time limit, Torturette would sit waiting alongside it, indistinguishable from having hung itself. That is why every test has a maximum waiting time (5 seconds by default; adjustable with the TORTURETTE_TIMEOUT_SEC environment variable if your machine is short on RAM and you need more headroom before calling something hung). If it runs out, the case is reported as HANG, along with the last thing the program managed to print before falling asleep.

Relationship with debugging

Although Torturette can work with binaries without debugging information, the value of the analysis increases considerably if the target program is compiled with debugging information. In that case, the messages and the reports can provide far more context about the exact point at which the failure occurred.

For that reason, this tool is more useful in development or analysis environments than in a final, unsupervised execution context. It does not replace a debugger or a deep audit, but it does offer a quick way to force error situations and observe how the software reacts.

Test program

The tests/try_me directory contains a minimal example designed to demonstrate the tool's behaviour. That program accepts the following commands:

Command	What it provokes
malloc	builds a linked list, correctly freeing anything half-built on failure (no leaks, so that you can see it can also be done properly)
calloc	builds a matrix with a real and very common bug: if it fails halfway through, it does not free what has already been allocated
mix	a buffer that grows with realloc(), with the classic error of losing the old pointer if realloc() fails
leak	a memory leak, on purpose
doublefree	freeing the same thing twice, just in case
segfault	a direct invalid access, without anaesthetic
abort	a controlled abrupt termination
exit	a normal exit, so that you can see it knows how to do that too

malloc, calloc and mix never fail on their own: it is Torturette that, testing each allocation one by one, decides whether your code knows how to survive them failing. malloc is deliberately well written (so that you can check that Torturette also knows how to say that something is fine when it is); calloc and mix have real and common bugs -- the kind you see constantly in school projects -- so that you can see how they look in the report.

It is especially useful for seeing how Torturette reports different kinds of problem without having to prepare a complex program from scratch.

Testing the tester

Running make try, make bsq or make gnl from a terminal asks whether to torture the program as it is or to mutate it first; make mutate skips the question. A tester that says OK to everything also says OK to correct code, so running it on correct code proves nothing. tests/tools/mutate.sh does what Google's mutation-testing system does: it plants one small bug on purpose -- a free dropped, a free doubled, a NULL check removed, a < turned into <= -- rebuilds, runs Torturette, and checks that the verdict changes. A mutant Torturette notices is killed; one it misses survives. Each program under tests/ has a tools/mutate.sh wrapper:

tests/gnl/tools/mutate.sh     # mutates get_next_line itself
tests/bsq/tools/mutate.sh     # mutates the program that calls it, over five maps

Every mutant is one sed command on one line, recorded in mutants/manifest.tsv, so you can read exactly what was broken. The report at the end is drawn with libft's own table and box widgets, and its last column says why a mutant died -- which input noticed, and whether it was a leak, a double free, a signal, a hang or a change in how far the program got. Read the survivors before drawing conclusions. Some are equivalent mutants (the change alters nothing observable), some sit on a path none of the inputs takes (add a fixture), some change only the program's output without ever leaking, crashing or hanging, and a few are silent out-of-bounds accesses -- Torturette has no redzones, so a write one element past a heap block that happens not to crash is invisible to it. That last one is precisely the line between what a memory-failure tester sees and what an address sanitizer does.

Continuous integration

Nothing about Torturette ages well on its own. It interposes malloc through LD_PRELOAD, walks return addresses out of backtrace(), and decides what belongs to your binary by comparing addresses against its load range -- all of which are promises made by the libc, the compiler and the linker rather than by this repository. A new glibc, a compiler that stops emitting what backtrace() walks, a libft commit, a distribution whose loader resolves preloaded symbols in a slightly different order: any of those breaks the tool without a single commit of ours. So .github/workflows/ci.yml runs the same jobs on every push and once a day, and the surprise shows up on the morning it happens instead of the afternoon someone needs the tool.

What runs, on Ubuntu 22.04 and 24.04 with gcc and clang, on x86_64 and aarch64, and again inside Debian bookworm/trixie, Ubuntu rolling, Fedora, AlmaLinux 9 and Arch containers:

make all                   # recursive: libft first, then torturette and the injector
.github/scripts/verify-artifacts.sh
make tests
tests/tools/ci.sh verdicts
make re                    # and again from clean, to catch a missing prerequisite

Only glibc distributions, and on purpose: the injector needs execinfo.h / backtrace(), which musl does not have, so Alpine is out of scope rather than broken.

verify-artifacts.sh checks one thing the build itself cannot. injector.so is preloaded into your program, so every symbol it exports interposes on your program's own -- and it links libft, which defines several hundred globals with no ft_ prefix. -Wl,--exclude-libs,ALL is the only thing keeping those local. The script compares the injector's exported surface against libft's symbol table on every build: if a toolchain ever stops honouring that flag, Torturette starts quietly replacing functions inside the program it is testing, and every verdict it prints becomes fiction -- green, plausible and wrong.

The gate: tests/tools/ci.sh

The same script CI runs is the one you run, with make ci. It asks two questions.

Are the verdicts still the right ones? Torturette is run on every program under tests/, on every fixture, and the OVERALL RESULT is compared with the verdict that program is supposed to get. Several of the test programs are broken on purpose, and those are the important ones. try_me leaks, double-frees, segfaults and aborts on command, and its calloc and mix paths drop the earlier allocations when a later one fails; so its expected verdict is KO, and the gate fails if Torturette says OK. They are not skipped -- skipping them would throw away the only cases that prove anything. A tester that says OK to correct code proves nothing; a tester that says OK to code it is supposed to reject has gone blind, and that is the only failure mode a tester really has. Where the kind of failure matters, the case also names the marker the report has to contain (DOUBLE FREE, SIGSEGV, SIGABRT, LEAK), so a report that finds a problem but the wrong one still fails. Neither check reads the exit status -- torturette always exits 0, on purpose -- so the verdict is parsed out of the report with the colour escapes stripped, and a run that has to be killed by the timeout counts as a hang.

Is it still noticing what it used to notice? The verdict gate cannot answer that: a tester that has quietly stopped seeing anything still agrees with every expectation about correct code. So the mutation suites above are run in full, and the mutation score of each target is checked against a floor recorded in mutation_floors(). Survivors are known blind spots, not failures -- the floor is what stops the score sliding a little further every month without anyone noticing. Raise a floor when a real improvement lands; never lower one to make CI green.

make ci                       # both, from a built tree
tests/tools/ci.sh verdicts    # just the verdicts (about two seconds)
tests/tools/ci.sh mutation    # just the mutation scores

Limitations

• It is not a production tool, nor a substitute for a full debugger.
• The analysis depends on the target program being compiled with debugging information if you want to see the exact origin of the failures.
• Some cases may end abruptly, which is to be expected in a failure-testing environment.
• Behaviour may vary depending on the system, the libc and the target binary's compilation options.
• Every test is made with a fork() at the exact instant of the call -- and fork() only preserves the thread that invoked it; the rest of the threads that already existed at that moment do not carry over into the test process. If your program creates several threads and then waits for the first ones (pthread_join) after a later one fails, the test of THAT later thread may give a result (a crash or a hang) that reflects this limitation of fork() rather than a real failure in your code. The test of the first pthread_create(), and in general any memory/thread failure that does not depend on previously live threads, does not have this problem.

Quick test

The tests directory contains small example programs intended to let the user try out the flow. Build them all with make tests, or build one and torture it straight away:

make try
make bsq
make gnl

That program allows you to try simple commands to see how the tool reacts and how the failures materialise in the reports. If after all this your program still "works perfectly", it is not that it is perfect: it is that we have not tested it enough yet.