OffSec Reality Check: Why Your Pentest Sucked (and How to Fix It)
Offensive Security (OffSec) is a broad discipline. It spans vulnerability research, reverse engineering, red teaming, application security, and more but let’s just call it penetration testing for now. Lately, I’ve seen more people express frustration after receiving subpar results from their penetration test vendors. If you’ve been there, this post is for you.
A Crowded, Noisy Market
There are over 5,000 penetration testing vendors out there. That’s a lot of noise. Many simply run automated scanners and hand over the results as if that counts as a proper engagement. Why? Because price pressure is intense, especially from customers who are just trying to check a compliance box. Often, the lowest bidder wins, and quality suffers.
Here’s the uncomfortable truth: good penetration testing is expensive. The quality of your results will correlate with the quality of the brain doing the work. All things being equal, talented practitioners cost more.
Who’s Buying?
Many large enterprises have internal OffSec teams. They only bring in external vendors when workloads spike or when rare skills are needed. That means a large portion of the penetration test market is made up of SMBs, local governments, school districts, and other resource-constrained entities.
Unfortunately, budget-constrained buyers tend to favor the lowest cost option. That often leads to poor outcomes and plenty of frustration.
The Core Problem with Penetration testing
Penetration testing is fundamentally a recursive search problem. One asset leads to 10 more, those lead to 100, and so on. A host reveals ports, which lead to web apps, which lead to URLs, which have parameters, and each one can expand the scope further. This branching factor makes the problem intractable for fully automated tools.
The attacker only needs a few wins to achieve total success. Prioritization and intuition are key, and that’s where skilled humans shine.
Typical elements that expand the search space:
Thousands of live hosts and services
Dozens of protocols and websites
Tens of thousands of URLs and parameters
Cloud infrastructure
Leaked credentials
E-mail addresses (phishing campaigns)
SaaS integrations and third-party risk
Shadow IT
Even with the best tools, the time adds up fast.
Managing Expectations
If you’re just trying to check a box, hire the cheapest vendor and move on. But don’t be surprised if the results are thin, vague, or misleading. Skilled testers will likely find real issues and generate real work for your already-stretched IT team.
Understand what you’re buying:
Assessment: Broad, shallow, fast. Good for finding the obvious stuff. Can be done internally with the right tools & skills.
Adversary Simulation: Deep, focused, realistic. Time-consuming and expensive. Useful if you’re mature enough to handle it.
Neither will give you absolute confidence. They can prove the presence of a vulnerability, not its absence.
And remember:
Networks change constantly. What’s secure today might be exposed tomorrow.
Penetration tests are point-in-time, best-effort evaluations based on incomplete information.
Vulnerability research is a different discipline entirely. Penetration tests look for known issues. Zero-days are another level of investment.
Common customer misunderstandings:
Expecting 100% coverage
Imposing fines for any missed vulnerabilities
Demanding fixed pricing for unclear scopes
Asking for guarantees that no vulnerabilities exist
Penetration tests cannot prevent breaches. This is important.
Some clients enter a penetration test engagement expecting $50,000 worth of depth, creativity, and thoroughness—while only allocating a $5,000 budget. This mismatch is a recipe for disappointment.
High-quality penetration testing isn’t just about running tools; it’s about the time, experience, and intuition that skilled practitioners bring to the table. When budgets are tight, testers are forced to cut corners: skipping manual exploration, ignoring edge cases, or focusing solely on known vulnerabilities instead of creative attack chains.
Depth takes time. Lateral movement, privilege escalation, chained exploits, and business logic flaws are rarely found in a few hours. If you’re not paying for the time it takes to explore those paths, you shouldn’t expect them in the report.
Budget appropriately, or start smaller with targeted objectives you can afford to explore meaningfully.
Collaboration & Partnership
Another common mistake clients make is treating a penetration test as a black-box “set-and-forget” engagement. They hand over a scope document, maybe an IP range, and then disappear—failing to answer questions, clarify boundaries, or provide critical context. This lack of collaboration handcuffs the tester and guarantees shallow results.
Remember: penetration testers aren’t clairvoyant. They can’t read your org chart, infer undocumented architecture decisions, or guess which web apps are the most business critical.
Effective penetration testing is collaborative. Great results flow when the client is responsive, transparent, and invested in helping the testers succeed. Something as simple as answering a question about authentication flow, clarifying intended access paths, or identifying high-value targets can save hours of guessing and redirect effort toward more impactful work. The more you treat your penetration test vendor like a partner, and not an adversary or a checkbox, the more likely you are to uncover meaningful, actionable insights.
Penetration testing is a recurring part of any mature security program. That means you’re likely going to engage with the same vendor at least once a year, if not more. Treating the relationship as a transactional checkbox exercise makes the process harder on everyone and diminishes long-term value. Instead, aim to build a working partnership.
Choose a team you can trust, communicate with easily, and rely on to understand your environment over time. A good partner will remember the nuances of your infrastructure, adapt to unexpected changes, and help you track progress year over year.
If engagement is painful, slow, or adversarial, it’s time to reassess. Penetration testing should be rigorous, but the process of working together should be painless.
How to Find a Good Penetration tester
Talk to the person doing the work. If you can’t, ask how quality is ensured.
Ask for example reports. See how they communicate findings.
Seek referrals. Your professional network is a better filter than a marketing website.
Check credentials. OSCP, OSWE, and OSEP are strong signals.
Understand the cost. Good people might cost more. Complex targets take time and experience.
You’re buying peace of mind, not a PDF. Prepare for sticker shock. There’s no way around that if you’re shopping in the same labor market as everyone else. But there is one way to potentially stretch your budget: leverage labor rate variance across regions.
Security talent in high-cost markets (e.g., the U.S., Western Europe) comes at a premium. But equally skilled practitioners exist in lower-cost regions like Eastern Europe, Latin America, Southeast Asia, and even the Caribbean; often at a fraction of the price. This isn’t about exploiting cheap labor; it’s about acknowledging that global talent markets are unevenly priced due to cost of living, local demand, and currency differences.
Of course, this approach requires diligence and comes with tradeoffs:
Ensure you’re not sacrificing quality for cost.
Vet the individuals or firms thoroughly. Ask for sample reports, references, and interviews with the actual people doing the work.
Alignment on time zones, and communication norms.
Doing business internationally can get complicated, especially when it comes to contracts, payments, and legal requirements.
When executed well, this strategy can give you access to top-tier talent without breaking your budget. But it’s not a shortcut—it’s a different procurement model that still demands careful vetting and clear expectations.
Consider Bug Bounties for External Testing
If your goal is to test your external footprint (i.e., internet-facing systems), consider starting with a public bug bounty. No, it’s not a full replacement for a traditional penetration test but it can be surprisingly effective. Look no further than the U.S. Department of Defense’s Hack the Pentagon program for proof.
That said, bug bounties aren’t without risk. You’re inviting thousands of strangers to attack your infrastructure. Trust is inherently low when working with unknown researchers. Report volume can be overwhelming. And if you aren’t ready to fix what they find quickly… you’ll have a bad time.
One major caveat: do not run bug bounties on your internal systems. That’s a trust boundary you really don’t want to cross unless you’re working with a vetted, tightly controlled partner.
Smarter Strategy
Start by asking: what’s the worst-case scenario if we get breached? Quantify the potential range of costs: financial, reputational, operational. If a breach cleanup could cost $1M, but a solid penetration test costs $20K, that’s a 50x ROI if it helps you avoid disaster. Conversely, if the potential impact of a breach is only around $10K, a full-scale penetration test may not be the best use of resources.
Don’t burn budget blindly. Start small and focused. Time-boxed engagements with clearly defined objectives will yield much higher value than a generic “test everything” approach. Target low-hanging fruit first. Emulate realistic attacker behavior and access paths (see MITRE ATT&CK initial access vectors). Iterate.
Make yourself just hard enough to move down the target list. You don’t need to outrun the bear, just the person next to you.
Final Thought
OffSec isn’t magic. It’s methodical, creative problem solving by humans. Set your expectations accordingly. Invest wisely. And if you’re not ready to act on what a penetration test finds, you may want to focus on foundational security controls first.
