Penetration Testing as a Service

Home - What - Why Pen Test - Why High Bit - Types - Reports - PTaaS - How Much?

Penetration Testing as a Service (PTaaS). A continual delivery model.

This page describes an alternative delivery model for penetration testing, known as "Penetration Testing as a Service", or by the acronym PTaaS.

The PTaaS acronym is a fairly recent development in a penetration testing context, and has come into common use only in the last couple of years. It really is something of a mis-nomer, because penetration testing has always been a service. The difference between PTaaS and the traditional penetration testing model is in how that service is delivered. In this article, we'll explain the difference between PTaaS as a conceptual delivery model, in comparison with the traditional penetration testing delivery model.

When done responsibly, there is little difference in testing methodology and no difference in quality, but Penetration Testing as a Service (PTaaS) will typically employ a different billing, scheduling and delivery model than traditional penetration testing, and these differences make PTaaS an attractive approach for some clients. PTaaS can be attractive to clients who perform or intend to perform penetration testing on a regular basis. It can be even more attractive to clients who produce B2B solutions and who are often asked by their own prospective clients, often at unpredictable times, to provide evidence of penetration testing. Sometimes the fastest way to understand a concept is to examine the pros and cons, and that is the approach we will use here.

Here are some of the advantages of Penetration Testing as a Service:

• Flatter spend. Most PTaaS contracts are structured on a monthly billing cycle. This helps long term budgeting by flattening your spend and can make budget approval much easier, since it is a regular and expected expense.


• Reduced administrative overhead for both the client and for the vendor. It takes time from both client and vendor decision makers just to confirm scope, pricing and terms for traditional model engagements. This factor can become more significant when you don't know the timing in advance. One such case occurs when an important prospect asks for evidence of penetration testing. In this situation, the need for penetration testing is at least partly driven by an immediate sales need, unpredictable in advance, and this puts timing pressure on decision makers. Penetration Testing as a Service is a continual, ongoing delivery model and requires no further scope approvals unless you make changes to your scope.


• Near continual monitoring and testing of networks and applications. The exact monitoring will depend on your scope, but typically includes an initial baseline engagement followed by monthly monitoring of all scoped networks, hosts and web applications. The monitoring includes examination for changes in exposed services as well as new vulnerabilities impacting existing services.


• Much shorter, almost real time, remediation cycles. The focus of PTaaS is on rapid and direct communication, rather than detailed, polished reporting. In our case, when we discover a new vulnerability, whether reported by a scan tool or discovered manually, we upload a simple report and notify you. Then we get on the phone and explain it (if needed) rather than spend a lot of time creating detailed reports with screen captures and long narratives. This allows you to immediately know what was found, and immediately begin a conversation focused on correcting it and confirming the correction. Conversely, in a traditional model, it is not unusual to have dozens of serious flaws reported all at once on a final report. This is often accompanied by heavy timing pressure, and that can be quite disruptive, taking your development staff out of the development business and into the remediation business in a frenzy of activity to correct a lot of flaws quickly. The Penetration Testing as a Service model means you can often remediate at a more reasonable pace, one or two at a time, with less impact on the day to day operations of your dev team.


• Early release detection and remediation. Depending on your compliance mandates, if any, and your architecture and SDLC model, it may be possible to deploy application changes to a security testing instance, which can be deployed at a strategic point in your SDLC release cycle, such as before any user acceptance or staging release. With Penetration Testing as a Service, detection and remediation can then occur prior to any further release. This approach can reduce the overall cost and risk exposure arising from unintended vulnerabilities by catching security faults earlier in the release cycle. Knowing that flaws have been caught prior to production release can go a long way toward reducing anxiety and time pressure.


• Immediate availability of current, public facing reports. When your important new prospect asks your sales team when your last penetration test was conducted, your sales team can tell them ‘last month’ or ‘last week’, and hand them your current report. This is a big advantage for our clients who are in the B2B space. Prospect requests for evidence of penetration testing is the single biggest driver of first time penetration testing for Business to Business vendors. It is also the single biggest factor driving short engagement windows, usually paid for at a premium. The Penetration Testing as a Service model allows B2B clients to present that evidence very quickly, while also demonstrating a significant commitment to continual penetration testing.


• Same quality. When done responsibly, you should still receive full, quality, manual testing. The diffence should be in the billing, frequency, and reporting. In our case, we will use automation to monitor your environment for changes and identify vulnerabilities that can be identified with automation, but any detected changes also get manual attention. Ongoing Penetration Testing as a Service testing activity is change driven, but you also have an annual, end to end, no assumptions, manual review of your scoped networks, hosts and web applications, just like you would expect with a traditional model.


• Ongoing documentation. A responsible PTaaS vendor should be providing log entries, and those log entries should be done by qualified testers who are actually monitoring and testing your environment. In our case, every time our testers run or review a monthly change scan, the tester creates a log entry with the date, time, testers name and review status. If changes occur and no security flaws are found, it is noted. If new (day zero) vulnerabilities are found on existing, unchanged systems, it is noted. If flaws are found and verified, a finding report is created, you are notified, and the current remediation status is noted and tracked. With Penetration Testing as a Service, you won't have a final report like you would have with a traditional model, but unlike the traditional model, you do have a running record of what has been done, when it was done, who did it and the current status of any open findings, all the time. For some clients, this has more value than a snapshot final report in a traditional model.

There are also disadvantages with the Penetration Testing as a Service model. Nothing is perfect for everyone, and PTaaS is no exception. Here are some of the factors that make PTaaS less desirable than a traditional delivery model for some clients:

• No full report. This is a plus for some, a minus for others. Under PTaaS, testing is ongoing and never completes, so there is a running record but no full private report. For some clients, especially those who are meeting compliance mandates that require detailed explanations about testing approach and technical summaries targeted for auditors, or intended to cover a specific ‘point in time’ snapshot, this can be important and requires a more traditional approach to reporting. With Penetration Testing as a Service, there is no detailed, private, final report. There is an ongoing public facing report, which, like all reports intended for public consumption, does not include sensitive details. The details are in finding reports that are generated individually, as they are identified, and these are geared toward rapid communication and remediation and are much less formal than the traditional model finding reports. There is no full, final report that ties all of these findings together in context at any historic point in time. There is a running record of where you are right now, and plenty of history and logging, but the overall context never gets wrapped up and neatly tied together. This is perfect for some clients, but may be less than desirable for others.


• Detrimental to a depth first approach. Like every other discipline, penetration testing has it's differing philosophies, and one of the more contentious and ongoing debates arises from depth versus breadth. This is not the place to dive into that debate, but PTaaS does have an impact here, especially if you lean toward a depth first philosophy. Covering the arguments surrounding this debate could easily consume hundreds of pages, but one factor is undeniable: penetrating deeply into an environment almost always involves chaining or combinining several vulnerabilities or even vectors together. This is something that is often done in traditional penetration testing, but doing so means the tester waited before reporting at least some of the findings. Testers don't get to combine flaws very often under the Penetration Testing as a Service model, unless the flaws occur together in the same release. Even then, the PTaaS model emphasizes immediate reporting and remediation, not waiting to report the finding because you want to try to combine it. Most clients who are interested in PTaaS have no position on the depth vs. breadth issue and when explained, lean toward a breadth first approach anyway, but if you are inclined toward depth first testing, PTaaS may not be for you.


• Third party timing restrictions. Some hosting providers require permission before you can perform penetration testing of any kind. One major provider with such a policy is Amazon Web Services. AWS requires that you obtain authorization before testing, and the maximum window they allow, as of this writing, is 12 weeks. This policy appears to be based on an assumption of a traditional delivery model for penetration testing, and assumes 12 weeks should be sufficient. This does not mean that you cannot use PTaaS in an AWS environment, but it does mean that you need to resubmit for approval 4 or 5 times per year. AWS may eventually adapt their policy to allow for Penetration Testing as a Service delivery models, but as of this writing, 12 weeks is their maximum and reapprovals will be necessary. Other providers also have restrictions. You will need to check your contracts for restrictions no matter which approach to penetration testing you take, but short authorization windows can be a recurring annoyance under the PTaaS model.


• Sensitive data handling and retention. Vendors will differ in their handling of sensitive data, but if they have solid practices for archiving and encrypting engagement data, with solid key management practices, Penetration Testing as a Service will cause complications. A PTaaS model requires that automated change detection and vulnerability scanning solutions run on regular basis. It is a rapid and continuing need, and is usually managed in a scheduling queue. Your data should still be encrypted and subject to access controls, but some of that data is in a more or less continually live state by necessity, and is seldom static long enough to be archived at rest under key, especially with solid key management policies like split knowledge of keys, as you may be accustomed to with a traditional penetration testing engagement.


• Altered applications. Some applications, especially those requiring multiple factor authentication or CAPTCHA protected login functions, may need to be modified in order to allow change detection and scanning software to run unattended. Basically, any control that is intended to prevent automation could, perhaps not surprisingly, cause problems with automated software in Penetration Testing as a Service engagements. To balance this, methods that are often problematic for normal human users, such as source IP restrictions, long complex tokens and client certificates can be used. Automated change detection and scanning traffic should always come from known IP addresses and can use source filters in addition to authentication mechanisms more suitable to automation. You may also benefit from updating version banners when you release, and placing those banners where they are visible prior to authentication, so that first order processes can move recently changed applications to the top of the queue for closer inspection. The essential point is, some application modifications may be necessary or beneficial, to allow automated systems to easily monitor for changes.


• Scope creep. It has to be said, any agreement that sets a price for a recurring service has to deal with the issue of scope creep. If your applications are changing and new functionality is added, that should be expected to a certain degree and is one of the advantages of the Penetration Testing as a Service model. On the other hand, if the application starts to resemble several applications bundled together, or starts using non-standard components that were not originally considered and priced, or adds web services that require special testing, or in general, becomes an application that would have been priced differently in an original quote, you should expect increased pricing. In our experiance, most of our clients who are interested in the PTaaS model in the first place are very familiar with the concept of scope creep as it applies in software development, and they understand that major changes may require price adjustment in penetration testing as well. Nonetheless, if it happens a lot it can eliminate some of the advantages of PTaaS by requiring recurring administrative overhead and budget approval. Some clients with rapidly evolving applications may be better served by a traditional model, at least until the solution stabilizes.

Things to be wary of with PTaaS offerings

Finally, and this is neither an advantage nor a disadvantage, but a caution: Penetration Testing as a Service as a model lends itself well to deception. There are two deceptions in particular that are made much easier under a PTaaS model:

• Disguising a product as a service. There have always been pentetration testing vendors, some with big names, who have been willing to 'dress up' automated scan results and sell those results as a quality penetration test. If that meets your needs, no harm no foul, but if you are looking for quality manual penetration testing, be wary. The Penetration Testing as a Service delivery model, with it's emphasis on easily consumable, near real time, abbreviated reports, makes this deception even easier than it is under the traditional approach. The screen captures, detailed narrative explanations and supporting documentation that are typical of quality manual testing under a traditional model are missing by design in the PTaaS delivery model.
  
  If you are not careful, you could be buying nothing more than automated scans created by a scan product, with the only service being delivery of automated reports over a web portal. Solutions like this deliver no actual manual penetration testing service at all, at most just a team of consultants who answer occasional questions about scan results or do a little false positive removal and add a little of their own verbiage before you see those results.
  
  This kind of deception can end up costing you, even if you are prepared to accept automated testing and it seems to be cheaper. If you are B2B and are testing to satisfy a request from a prospect or client, check with a few clients first to make sure that your target market is willing to accept automated testing results. Many large coporate purchasing departments now have policies requiring third party manual penetration testing. If you are testing for a compliance mandate, you should also check the standard - PCI-DSS as of version 3, for instance, specifically requires manual testing.

  If you get the wrong answers from a Penetration Testing as a Service vendor, find another one.



• Hiding underqualified personel. This is another deception, quite common in traditional penetration testing, made even easier under PTaaS. The PTaaS model seems to justify putting a web portal between you and the testing team. It's important to note that there really is nothing in the Penetration Testing as a Service delivery concept that even requires a portal, but the concept does lend well to a portal delivery and it is almost universally used. That portal can easily be utilized as a barrier to visibility. If the portal is your only communication channel, it is particularly easy to hide the identity, and qualifications, of the testing team members. You could end up with a team of 'consultants' who may or may not have any penetration testing certifications, who could be anywhere in the world, identified only by a name and a photo, and who only communicate with you over a portal.

Before contract signing, be especially wary of:

• Sales documentation using terms like researchers, and consultants, rather than certified penetration testers. PTaaS is a great model for hiding underqualified resources.


• Sales documentation emphasizing your access to qualified personel. Access to the tester should never be in question because it is the tester who should be contacting you. Make sure the access being touted includes the testers company email address and a phone number, and not just access through your portal account.


• Penetration Testing as a Service contracts that do not clearly specify actual, manual penetration testing work by personel with the right qualifications, or contracts that seem to imply it without actually specifying any manual work.


• Long term contracts without an easy cancellation.

After contract signing, assuming you have an easy cancellation clause (which you should have), be wary if:

• In practice, the service does not seem to provide log entries generated by humans, and at least occasional screen captures and supporting files, along with manually generated notes, at the time of the initial reporting of a finding.


• Your follow up questions for specific findings seem to be answered in general terms, by someone who seems to lack specific knowledge of your environment or applications.


• You get a different tester every time. This may mean that the vendor is crowd sourcing your testing, which is something that is particularly easy to do under the Penetration Testing as a Service model. Some clients are comfortable sharing their most sensitive data with a revolving door of global contactors, but most clients are not, and it's a factor that should at least have been disclosed in advance.


• You are constantly wading through false positives, or always seem to be the one who has to contact the vendor for clarification, instead of the other way around.


• You are only able to contact the tester through service portal interaction, and are not provided with the tester's company email address, phone number or other contact channels.

If you find yourself in an uncomfortable relationship, that's what the cancellation clause is for.

Penetration Testing as a Service (PTaaS) Summary.

The PTaaS delivery model is focused on nearly continual change monitoring, testing and rapid reporting and communication. It sacrifices polished, detailed reporting in favor of rapid remediation and ongoing testing. It reduces administrative overhead, can shorten remediation cycles and catch flaws earlier in the release cycle. It can have strong sales advantages for B2B clients, flattens budgeting and can simplify budget approval. It is not cheaper when done correctly, because the same manual effort is required from the same highly qualified testers, but it may increase ROI for the same spend, depending on your needs. For many of our clients, this is a delivery model better suited to their needs than a traditional model.

The approach can also cause problems if you need detailed reporting, can impact sensitive data handling and retention, may be complicated by third party approval factors, may require changes to applications and is not compatible with all penetration testing philosophies. For some clients, the traditional model is still the best.

If you do decide that Penetration Testing as a Service is the right approach for you, you still need to be careful that you are actually getting what you think you are getting, and should insist on a contract that allows you to bail out easily. In our case, for example, our PTaaS contracts allow you to cancel at any time, and we insist that we start out with a baseline traditional engagement so that you get to know us and our penetration testers, and we get to know you and your systems. For us, this is a relationship and not just a service.

If you are interested in the PTaaS delivery model, please ask us about it. This has been only a brief treatment of the subject and we will be happy to further explore the options with you.

Ask us for a free, quick, no hassle quote using the contact form above.