Besides coming up with the ideas, I worked mainly on the python implementation and the PicoCTF and OverTheWire benchmarks. The Docker attackbox setup took a particularly large part of the engineering effort. I also took part in writing the final paper.
HackSynth is an autonomous penetration-testing agent built from two cooperating modules: a Planner that proposes the next command, and a Summarizer that digests what the command printed so the next step has a compact history to work from.
To measure it, we built a benchmark of 200 Capture-the-Flag challenges in total: 120 from PicoCTF and 80 from OverTheWire. The evaluation compares open-weight and proprietary models, looks at how temperature and top-p change behaviour and token use, and examines whether the agent’s actions stay safe. GPT-4o performed best among the models evaluated in the paper.
It was presented at ELTE’s TDK conference (first place, Dec 2024), the Eötvös Conference, and the 37th National Student Research Conference, where it won the cybersecurity session (April 2025). It also became the basis of a business plan for the Hungarian Startup University Program.
The follow-up, HackSynth-GRPO, explores teaching an agent through reinforcement learning rather than only prompting it.
Making the agent runnable
An agent needs more than a prompt. It needs a place to run commands, the tools those commands expect, and a way to reach the challenges. The attackbox setup was one of the parts I spent the most time on.
The public implementation’s docker_setup.py creates or reuses a Kali container, installs the tools, and sets up challenge connectivity, including VPN support. These are practical details, but without them there is no useful tool output for the agent to reason about.
Turning challenges into a benchmark
The benchmark work was a substantial part of my contribution. A challenge needs more than a title and a flag: the agent needs its starting instructions, files or network access, and a consistent way of checking whether it succeeded.
The PicoCTF benchmark includes solver scripts and a container with the tools those scripts need. Running them produces benchmark_solved.json, which can then be passed to the agent’s benchmark runner. The challenges cover general skills, cryptography, web exploitation, forensics, binary exploitation and reverse engineering.
For OverTheWire, the solvers cover Bandit, Natas, Leviathan and Krypton. Each wargame produces its own solution file; a script combines these into one benchmark. That makes the evaluation reusable across model configurations without manually rebuilding the task descriptions and expected answers each time.
Commands, observations and a limited memory
The Planner works from the task and the accumulated summary. Its command runs in the attackbox, then the Summarizer folds the output back into the next iteration. Long terminal output is therefore an engineering problem as well as a prompting problem: the agent needs enough evidence to choose a useful next step without carrying every byte of every command forever.
The repository includes the configurations used for the paper’s measurements, including experiments with observation length and sampling parameters. This is the part of agent research I find especially concrete: the surrounding tools, execution environment and benchmark determine what the model can actually attempt.