What Changes Between Your January and June Pen Test?
Key Takeaways
Introduction
Most organizations treat penetration testing the way they treat a fire inspection. Someone comes in, runs their tests, hands over a report, and the certificate goes in a folder. Box checked. See you next year.
That model works. It answers a much narrower question than most people think it does.
In this article, our security and infrastructure leads share how point-in-time testing earns its keep, where it leaves you exposed, and why the fix is to stop asking it to do a job it was never built for. For the broader picture of how Derive approaches security, see our cybersecurity services.
Are manual and continuous pen testing competing options?
The first thing to get straight: these are not competing options.
“The two options are complementary. They’re definitely not mutually exclusive. It’s not one or the other. They’re addressing different needs within an organization.”
— Yaroslav Samoylenko
Manual penetration testing is fundamentally an audit motion. The ideal customer has governance requirements that must be met, and met in a way an external auditor will certify. The real deliverable is the attestation, more than the finding list. As Samoylenko put it, in practice it’s “more of an audit than it is a pen test.”
Automated, continuous penetration testing is aimed somewhere else entirely: at organizations trying to mature their threat and vulnerability management programs. Its output is context, fed to the team that has to decide what gets patched first across a large fleet of endpoints, rather than a certificate.
Large, regulated organizations often need both. They need the certified manual test to prove testing happened, and they need a continuous platform running on the network telling them what’s newly exposed this week.
What changes between two pen tests?
A manual pen test is a point-in-time engagement. The typical cycle: initial test, readout, roughly a 90-day remediation window, then a retest. The retest only covers the findings from the original report.
So what happens in the months between engagements?
Everything. Systems come online and go offline. Scope changes. New systems get stood up. New vulnerabilities get published.
Your threat model is moving too. A new data store. A new department. A new line of business. A new office that arrives with systems nobody has inventoried, which is exactly what happens during mergers, acquisitions, and divestitures. The assets change constantly, and when the threat model shifts, the patch cadence built around the old one becomes quietly out of date.
Then there’s speed. “A vulnerability could be discovered tomorrow that could be exploited by tomorrow night,” notes Mike Ilich. “You can’t wait until June to see if you’re vulnerable.”
The pattern is familiar to anyone who’s watched a Citrix advisory drop: nobody’s scanning their NetScalers, a vulnerability gets published, and suddenly everyone exposed is getting hammered at once. That’s opportunistic spraying, attackers scanning broadly for anyone who hasn’t patched yet. It’s precisely the category continuous testing is built to catch.
Is vulnerability scanning the same as pen testing?
These get used interchangeably, and they shouldn’t be. Scanning is a component of penetration testing. Most engagements begin with a scan, or target acquisition and enumeration, then progress through the later phases. Penetration testing encompasses everything underneath it, and the scan is one part of that.
Why isn’t a CVSS score enough to prioritize findings?
Once testing surfaces findings, the real work starts: deciding what to fix, in what order.
Most teams begin with the CVSS score (Common Vulnerability Scoring System), an industry standard that rates vulnerabilities from 1 to 10, with 10 reserved for the worst of the worst. Historically a 10 was rare; 9.5 and 9.8 were the practical ceiling. Lately, more genuine 10s have appeared.
CVSS is a reasonable starting point and a poor stopping point, because it describes the vulnerability, not your environment.
“That scoring system is not specific to your environment. It has no grounding in what your environment looks like, what your defenses look like, what systems you have in place.”
— Yaroslav Samoylenko
This produces the failure mode every security team has seen: two or three findings that individually scored as unremarkable, chained together, add up to a genuinely high-risk path through your network. A generic score will never tell you that.
That’s the second layer: the environment-specific score a platform like Horizon3.ai’s NodeZero assigns based on what it actually found in your network. Vendors label it differently, but the function is the same: translate a generic severity into a real consequence here.
Layer three is your threat model: how critical is the affected system to the business?
CVSS, then environmental impact, then business criticality. Run a finding through all three and you get something a decision-maker can act on.
What can you do with a finding?
At the end of the day there are only three ways to address a problem:
- Fix it: patch, remediate, or redesign.
- Defer it: insure against it, so if it happens you’re compensated.
- Accept it: a management decision that the cost of the risk landing is acceptable compared with the cost of fixing it.
Accepting risk is a perfectly legitimate choice. It just needs to be a deliberate one, and that’s an argument for continuous testing. Having a prioritized threat score against every single one of those items helps management make that decision with reason behind it, rather than just emotion.
Frequently Asked Questions
What is the difference between manual and continuous penetration testing?
Manual penetration testing is a point-in-time engagement that produces a certified attestation for auditors. Continuous penetration testing runs on an ongoing basis and gives the security team prioritized context on what is exposed now. They address different needs and most regulated organizations use both.
How often should you run a penetration test?
A single annual test proves you were secure on one day. Systems, scope, and published vulnerabilities change constantly between tests, so organizations with a large or changing footprint benefit from a continuous layer underneath their scheduled manual tests.
Is a CVSS score enough to prioritize vulnerabilities?
No. A CVSS score rates the vulnerability in general, not your environment. Several low-scored findings can chain into a high-risk path, so prioritize by CVSS, then environmental impact, then how critical the affected system is to the business.
Is vulnerability scanning the same as penetration testing?
No. Scanning is one component of a penetration test. Most engagements begin with a scan, then move through enumeration and later phases that a scan alone does not cover.
The Takeaway for IT Buyers
During our conversation about this, Ilich compared the two approaches to a primary care physician and a specialist. The specialist is who you want for the deep, targeted work, but the appointment is a production. Your regular doctor is down the street, checks the fundamentals far more often, and tells you when it’s time to escalate. Two related functions, with different rules and purposes. Use one and you get real benefit; use both and you have coverage.
Practically: if you’re a smaller organization without a large external service footprint, continuous automated testing is likely where your security dollar goes furthest. If you’re larger, with many services to maintain and custom web applications that make you a target for engineered attacks, you need the manual depth too, and the continuous layer underneath it.
What almost nobody should be doing is running one point-in-time test a year and calling that a security program.
Want to know what your environment looks like between tests? Let’s talk about where continuous testing fits alongside what you’re already doing. Book a free consultation.