Three Kinds of Nothing: How to Handle Vague Output in Agentic Pentesting
How to fix up your pentesting workflows to differentiate between absence and silence
So recently I've been building a harness and tools to make my life easier as a pentester. And while it's been fun, I've hit a couple of snags along the way, particularly with output.
You see, most security tools have an issue or two describing when something goes wrong. And while as a human you might have experience and context, a model or beginner may not, and that makes both run around in circles to sort out one thing.
For instance, say you're running a subdomain scan, and it tells you nothing was found. But there are actually three possible definitions of "nothing" here:
- Nothing found: The tool asked, got an actual answer, and that answer was empty. Like 0 or Null.
- Nothing arrived: The tool asked and something quietly intercepted the request. Like a DNS timeout, a proxy ate the request, or the CT log was down.
- Nothing complete: The tool asked, got part of an answer and considered it worthless. Like a filled disk, the API paginated and it read page one, or the scan hit a rate limit at host 40 of 200.
Same output, three different verdicts. And I learned this the hard way. In one of my recent engagements, crt.sh reported a clean result for a domain. When I went through the logs, I discovered it had actually returned codes 502, 502, then 404 across three attempts. The tool asked three times, got told the server was unavailable the first two times, found no page the last time, and reported "no subdomains found".
Technically no one lied. But that's because there was no way to differentiate between silence and absence.
And with AI in the loop, it gets worse.
An experienced tester gets suspicious when a scan comes back clean. An LLM driving your operations might not. It reads "0 results", writes "no subdomains were identified", and moves on with total confidence. And it's not something you fix with better prompts, but with proper engineering. So I got to work.
The solution as it turns out is simple: make tools incapable of saying "nothing" without saying "which nothing".
Here's the idea in pseudocode:
run the thing
if no response ever arrived:
verdict = NOTHING ARRIVED # exit 75, not 0
say so loudly, and stop
if a response arrived but you couldn't read all of it:
verdict = NOTHING COMPLETE # say what fraction, and why
report partial, never empty
if a real, whole answer arrived and it was empty:
verdict = NOTHING FOUND # this one is a findingThree branches, one for each case. Now let's look at each case properly with live tool output, and build the code around it to see what changes. First in Python, because everyone reads Python, and then in Rust, because I like Rust 🦀.
Case 1: Nothing arrived
This is the dangerous one, so it goes first. Here's what my harvest tool printed on the engagement I mentioned:
harvest: ***************.com
harvest: writing to /work/artefacts/harvest/***************.com
subdomains found: 0
hosts written to: hosts.tsv
harvest: doneNothing in that output is false. It also gives you no way at all to know that this happened underneath it:
$ grep crtsh /work/logs/commands.log
crtsh attempt 1: HTTP 502
crtsh attempt 2: HTTP 502
crtsh attempt 3: HTTP 404The tool knew, but it had no vocabulary to say so, so the knowledge died inside the function that had it. The fix is to stop treating "how many results" as the only question, and start recording whether anything came back at all.
Case 2: Nothing complete
This one is sneakier, because the output usually looks healthier. Here's a scan that hit a rate limit partway through:
$ ffuf -u https://********.com/FUZZ -w words.txt -mc 200
:: Progress: [40/200] :: Errors: 160 ::
/admin [Status: 200, Size: 4821]
:: Progress: [200/200] :: Duration: 4s ::One result. A perfectly plausible one. The Errors: 160 is right there in the output and it is very easy to skim past, and an LLM summarising this will say "one endpoint was discovered" without blinking. What actually happened is that you tested a fifth of your wordlist and know nothing about the other four fifths.
Partial is not empty and it is not complete. It needs its own concept.
Case 3: Nothing found
And this is the one that is genuinely fine:
$ dig +short admin.**********.com
$The resolver answered. The answer was "no such name". That's a real, reportable negative, and you can put it in a client report and defend it. The whole point of separating the other two is so that this one keeps its meaning.
Building it in Python
The trick is what we keep track of. Most tools count results. We're going to count requests, and mark each one answered or not answered. Once you have that list, all three nothings fall out of it:
- If none of them were answered, nothing arrived.
- If some were, it's incomplete.
- If all of them were, then and only then does an empty result mean something.
Here's the whole thing. It runs as it is, so save it as verdict.py and try it.
#!/usr/bin/env python3
"""
File: verdict.py
Purpose: Three kinds of nothing, and a tool that says which one it means.
"""
# Exit codes, so a calling script can branch on a number instead of reading
# prose. These two come from sysexits.h, a Unix convention older than all of us.
EXIT_OK = 0 # we have an answer you are allowed to use
EXIT_NOTHING_ARRIVED = 75 # TEMPFAIL: we learned nothing, try again later
EXIT_NOTHING_COMPLETE = 65 # DATAERR: we learned something, but not enough
def attempt(what, answered, detail=""):
"""One request we made, and what happened to it.
what : the thing we asked. "crt.sh", or "word 41 of the list".
answered : True ONLY if a real response came back. A 502 is not an answer,
it's the server telling you it couldn't answer.
detail : why not, in words, when answered is False.
This is just a dictionary. You could use a class or a dataclass; the point
is that a request now carries its own fate around with it, instead of that
knowledge dying inside the function that had it.
"""
return {"what": what, "answered": answered, "detail": detail}
def verdict(attempts, found):
"""Turn the attempt log plus whatever we found into a sentence and an exit code.
attempts : every request we MEANT to make, answered or not.
found : the things we actually found. A list, and often an empty one.
The order of the three checks below is the whole idea. Each one is a reason
you're not allowed to talk about the target yet.
"""
# Split the log: which of our requests actually got a response?
answered = [a for a in attempts if a["answered"]]
# 1. Did anything come back at all? Ask this FIRST. The moment you look at
# how many results you got, you've already assumed the requests worked.
if not answered:
reasons = ", ".join(a["detail"] for a in attempts) or "no request was made"
return (
f"NOTHING ARRIVED: {len(attempts)} attempt(s), none answered "
f"({reasons}). This says nothing about the target.",
EXIT_NOTHING_ARRIVED,
)
# 2. Did all of it come back? Partial isn't empty and it isn't complete,
# so it gets its own word. Notice we don't have to be TOLD how much we
# planned to do: it's the length of the list we're holding.
if len(answered) < len(attempts):
missed = len(attempts) - len(answered)
return (
f"NOTHING COMPLETE: {len(answered)} of {len(attempts)} checked, "
f"{len(found)} result(s) so far. The remaining {missed} were never tested.",
EXIT_NOTHING_COMPLETE,
)
# 3. Every request answered. NOW an empty result means something real.
if not found:
return (
"NOTHING FOUND: every request answered and the answer was empty. "
"This is a finding.",
EXIT_OK,
)
return f"FOUND: {len(found)} result(s).", EXIT_OK
if __name__ == "__main__":
import sys
# The three cases from earlier in this article, as attempt logs.
# A CT log that was down: three tries, three refusals, nothing learned.
crtsh = [
attempt("crt.sh", False, "HTTP 502"),
attempt("crt.sh", False, "HTTP 502"),
attempt("crt.sh", False, "HTTP 404"),
]
# A wordlist scan that got rate limited at word 40 of 200. Words 1 to 40
# were answered, words 41 to 200 were not, and that's the whole story.
ffuf = [attempt(f"word {i}", i <= 40) for i in range(1, 201)]
# A resolver that answered properly, and the answer was "no such name".
dig = [attempt("admin.example.com", True)]
worst = EXIT_OK
for attempts, found in [(crtsh, []), (ffuf, ["/admin"]), (dig, [])]:
line, code = verdict(attempts, found)
print(line)
worst = max(worst, code) # the loudest verdict is the one that wins
sys.exit(worst)That's about forty lines of actual logic, and it changes what the tool is allowed to claim.
What that changes
Here's what those same three scans printed before, each tool counting only its results:
subdomains: 0
endpoints: 1
admin host: not foundAnd now run verdict.py:
NOTHING ARRIVED: 3 attempt(s), none answered (HTTP 502, HTTP 502, HTTP 404). This says nothing about the target.
NOTHING COMPLETE: 40 of 200 checked, 1 result(s) so far. The remaining 160 were never tested.
NOTHING FOUND: every request answered and the answer was empty. This is a finding.$ echo $?
75One of those three lines is evidence. The other two are homework. And the tool now says which is which, in a way that neither a tired human at 2am nor a language model can misread.
The exit code matters as much as the words, by the way. 75 means the calling script, the CI job, or the agent orchestrating your tools all stop rather than writing a summary.
The same thing in Rust
Rust is nicer here, because an empty result isn't a number you might forget to check. It's a variant, and the compiler makes you handle it.
use std::process::exit;
/// One request we made, and what happened to it.
pub struct Attempt {
pub what: String,
pub answered: bool,
pub detail: String,
}
/// The four things a tool is allowed to conclude.
pub enum Verdict {
NothingArrived { attempts: usize, reasons: Vec<String> },
NothingComplete { checked: usize, planned: usize, found: usize },
NothingFound,
Found(usize),
}
impl Verdict {
pub fn exit_code(&self) -> i32 {
match self {
Verdict::NothingArrived { .. } => 75,
Verdict::NothingComplete { .. } => 65,
Verdict::NothingFound | Verdict::Found(_) => 0,
}
}
pub fn line(&self) -> String {
match self {
Verdict::NothingArrived { attempts, reasons } => format!(
"NOTHING ARRIVED: {} attempt(s), none answered ({}). \
This says nothing about the target.",
attempts,
reasons.join(", ")
),
Verdict::NothingComplete { checked, planned, found } => format!(
"NOTHING COMPLETE: {} of {} checked, {} result(s) so far. \
The remaining {} were never tested.",
checked, planned, found, planned - checked
),
Verdict::NothingFound => "NOTHING FOUND: every request answered and the \
answer was empty. This is a finding."
.to_string(),
Verdict::Found(n) => format!("FOUND: {n} result(s)."),
}
}
}
/// Same three questions, same order.
pub fn verdict(attempts: &[Attempt], found: usize) -> Verdict {
let checked = attempts.iter().filter(|a| a.answered).count();
if checked == 0 {
return Verdict::NothingArrived {
attempts: attempts.len(),
reasons: attempts.iter().map(|a| a.detail.clone()).collect(),
};
}
if checked < attempts.len() {
return Verdict::NothingComplete { checked, planned: attempts.len(), found };
}
if found == 0 {
return Verdict::NothingFound;
}
Verdict::Found(found)
}
fn at(what: &str, answered: bool, detail: &str) -> Attempt {
Attempt { what: what.into(), answered, detail: detail.into() }
}
fn main() {
let crtsh = vec![
at("crt.sh", false, "HTTP 502"),
at("crt.sh", false, "HTTP 502"),
at("crt.sh", false, "HTTP 404"),
];
let ffuf: Vec<Attempt> =
(1..=200).map(|i| at(&format!("word {i}"), i <= 40, "")).collect();
let dig = vec![at("admin.example.com", true, "")];
let mut worst = 0;
for (attempts, found) in [(&crtsh, 0), (&ffuf, 1), (&dig, 0)] {
let v = verdict(attempts, found);
println!("{}", v.line());
worst = worst.max(v.exit_code());
}
exit(worst);
}Save that as verdict.rs, run rustc -O -o verdict verdict.rs, and ./verdict prints:
NOTHING ARRIVED: 3 attempt(s), none answered (HTTP 502, HTTP 502, HTTP 404). This says nothing about the target.
NOTHING COMPLETE: 40 of 200 checked, 1 result(s) so far. The remaining 160 were never tested.
NOTHING FOUND: every request answered and the answer was empty. This is a finding.Same three lines, same exit 75. The difference is what happens six months from now.
Add a fifth variant, say NothingAuthorised for a source that refused you on purpose, and every match in your codebase stops compiling until you've decided what it means. That's exactly the property you want from the thing that decides whether you're allowed to make a claim (or not).
Limitations
The examples here are quite basic, and in truth this is custom security tooling, but that's the beauty of it.
With AI, you can find out what those gaps are, and with a prompt, two minutes and a test, you can expand it to do so much more. Here are the three gaps I'd start with.
- It tells you your requests landed, not that they landed where you think. A redirect, a load balancer, or a CDN edge serving a cached page from six months ago all arrive perfectly well and answer a question you didn't ask. Handle it by recording what answered, not just that something did. So save the final URL after redirects, the response headers, a hash of the body. Then an attempt that resolved somewhere unexpected is a fourth verdict rather than a success.
- It can't tell you what you never thought to ask. If your wordlist has no entry for the admin panel, you check all 200 words honestly and get a confident NOTHING FOUND over a question nobody posed. Handle it by making the question part of the output. Print which wordlist, which sources, and which checks. "No admin panel found using a 200 word list" is a claim a reader can argue with. "No admin panel found" isn't.
- A tool can mark its own homework. Nothing stops you writing
answered=Trueeverywhere and reporting complete forever. Handle it by deriving `answered` from something you didn't choose, like the HTTP status or the resolver's response code, and by logging every attempt somewhere. A verdict you can check against a log is a sound verdict indeed.
None of those make the original tools untrustworthy. You're simply marking their limitations, and improving upon it yourself one level higher.
The takeaway
None of this is particularly new. It's about forty lines of if-else statements and a vocabulary of three words. But it changes what your tooling is allowed to claim, and when you've got an AI turning tool output into prose that ends up as facts in a report, that is the whole ball game.
See you in the next one. Happy Hacking ✍️.