Though I believe penetration assessments to be important in assessing an organization’s overall security posture, I think they are 1) being performed poorly and 2) the results from them are being disseminated in a wrong way. The goal of any security assessment is to help an organization become MORE secure than they were before the assessment was performed thus reducing their overall risk. Many penetration assessments are performed by identifying vulnerabilities and breaking into as many systems as possible by exploiting these vulnerabilities. The report is then issued which has the list of these vulnerabilities, a perceived risk rating, and finally recommendations on how to remediate the vulnerabilities. What many pentesters lose sight of is the objective for performing the penetration assessment to begin with: to help the client become MORE secure. This type of penetration assessment provides absolutely no value to the client and certainly does not make them any more secure.
Many pentesters don’t see an unencrypted service enabled on a firewall which protects an organization’s PCI zone and wonder “why” this service is allowed but rather how I can use this service to break into this system. The recommendation for such vulnerability would be to use a more secure service; however, what is lost is “why” the vulnerability occurred in the first place and the impact to the business if such vulnerability was exploited especially with regard to the environment in which it was discovered. A penetration assessment needs to be just as much interview based (if not more) as it does technical. Without understanding the underlying reasons as to why such vulnerabilities occurred in the first place, it is impossible to provide any other recommendations other than tactical to the client. The client will then tactically remediate the vulnerability maybe by updating a system with a specific patch or shutting down a specific service and then a year later vulnerabilities of a similar nature will resurface. Why? Because the underlying reasons as to why such vulnerabilities occurred in the first place are unknown. Is it a patch management problem? Is it a change management problem? Are there no policies and procedures or minimum security baselines preventing such vulnerabilities? Is it a management problem? Is it a line-of-business problem? Is it a combination of the above? The list goes on and on, but without trying to understand the “why” it is impossible to truly help the client. It is no longer acceptable to report that the entire compromise of an organization’s Windows domain was obtained without at least attempting to
understand “why” it was possible and how to protect against future occurrences.
Today’s market has become so diluted with companies and individuals claiming they can perform penetration assessments (if you don’t believe me attend Defcon one year). Organizations need to have a better understanding as to how these hired service providers are actually performing these assessments. If a company performs security assessments with little or no interaction with their client, be very skeptical of using this company. As the old cliche goes, you get what you pay for. Bottom line is, penetration testing is no longer for the geeky technical guy who only cares about breaking into systems or for someone who knows how to run a vulnerability scanner. It’s for professionals who truly understand security and are interested in really helping an organization reduce their overall risk.
Tuesday, March 16, 2010
Subscribe to:
Post Comments (Atom)
Posts
10 comments:
What are you proposing as a solution? Proactive assurance? Risk management? More expensive penetration testing?
I also challenge the perceived value of penetration testing as a whole. I feel that clients are building a false sense of security, in that they hire a pentester to "break-in" and achieve an objective, and then fix the security issues that were found making them falsely secure.
My approach to security control evaluations or assessments is quite different. First the scope of the activity should be set to maximize the return on investment for the client. Pick those controls which are most relevant to the threats they intend to address. Worried about web-application hackers? include code development security testing processes, production change management, and preventative web application layer measures.
Once a scope has been agreed-to complete the assessment by evaluating the effectiveness of the control. Use automated vulnerability assessment tools to identify where patch management processes are ineffective, use static code analysis tools to identify coding practice weaknesses, etc.
Once vulnerabilities are identified, determine root-cause by identifying if the control is improperly designed or just operating ineffectively.
This will allow clients to become more secure, without enabling the false-sense of security which comes with pentesting.
I think the point of the article is kind of like that line from The Matrix, "There is no spoon." While there are certainly different and better ways to do pentesting, this is NOT about the pentesting. It's about maximizing the value of the results to the business that makes sense to the non-tech people. It's making about being able to tell how THIS pentest affects THEIR business. In other words, they need to know about risk, which means discussing meaningful impact, not vulnerabilities (assuming the simple: risk = threat x vulnerability x impact)
The solution that I am proposing is that instead of focusing so much time on the technical side of penetration testing, that the pen tester needs to also understand the business, its data, the owners, priorities, concerns, risk perception, etc. This will help to truly provide an impact statement that is meaningful and help to identify the strategic problems affecting the organization, hence providing much better recommendations in the report. Does this mean more expensive pen testing? No, but it could. It just means trust and credibility in the organization which companies hire to perform these assessments. It means trusting that the company performing the assessment is truly interested in helping an organization reduce risk. The problem that I have seen is that most pen testers don't have the social skills needed in order to perform an interview based assessment to truly understand the security issues (again, attend Defcon one year). Most feel a sense of accomplishment by breaking into as many systems as possible without ever once interacting with the client. There are only so many systems that can be broken into before the client says “We get it; we have a poor security posture”. In addition, it’s not strictly about breaking into systems but also about the information that can be obtained. This makes the pen tester go even deeper in understanding the organization in which they are performing the pen test. What are the different regulatory requirements that the organization has to adhere to (i.e. PCI, HIPPA, SOX, GLBA, etc)? What are executives going to respond more to, the fact that you were able to compromise every single system on their network or the fact that you were able to get unencrypted credit card information for a Level 1 merchant under PCI requirements?
Mark, I address some of these issues in a previous blog titled “To Pen Test or not to Pen Test, that is the question…”. I think what you are proposing is a risk based approach to pen testing. I think this approach makes sense in some cases but mainly in those organizations which have a limited budget to put toward security. The main problem with such an approach is that the threats are changing every day. As soon as a company feels that their biggest risk is externally at the application layer, a recession hits and layoffs occur. Now all of a sudden, internal hackers start to become a real concern. I still believe the best approach is a thorough assessment which addresses all controls and all threats with the focus on those that pose the biggest risk to an organization. As you mentioned, root cause analysis is definitely a must to truly help an organization reduce their overall risk.
I completely agree with "the understanding the business behind what they are testing" approach. It is very important to be able to understand the ramifications of your intrusion and explain it in business terms to those that contracted you in the first place. this can be achieved in a half day meeting. Understand the system, its usage, its users, its data, its connections.
However I disagree with the statements "A penetration assessment needs to be just as much interview based (if not more) as it does technical." & "what is lost is “why” the vulnerability occurred in the first place". In each penetration test I have a scope and goal. After my first interview (1/2 day), I draft what is needed so as to thoroughly test the systems susceptibility and reach my goal. Suppose a 2 week penetration testing engagement. using your argument of "equal or more interview" that would result in double the cost + two weeks of the clients personnel time (1 assigned) to the pen-testing. Taking a week out of the technical part and devoting it to interviews will result in poorer technical results + one week of the clients personnel time (1 assigned) to the pen-testing. Finally as a pen-tester ( at the pen-testing phase of the system ), I don't care as to why an unencrypted service exists, I care that it exists and is there for my taking. They should have consulted me at the design phase of the system.
My 2c.
I may have overstated how much time should be dedicated to actually understanding the business, but the point is clear. In order to provide value to your client an attempt to understand the business needs to be made. This may change what skills a prototypical pen tester needs in order to perform a value added assessment.
perhaps it would be better to insist that a solid risk assessment be performed long before you send in the trigger pullers (your Defcon crowd you keep mentioning).
I agree with you that risk assessments/evaluations should occur. i disagree that you should lump them into the same 1 or 2 week period as your technical assessment. its infeasible to think you can really understand a business with 1/2 or 1 day of interviews unless you are the luckiest and smartest guy on the planet and got just the right people lined up with the right information on the first try.
I see where you are going with this, but its nothing new. plenty of good pentest shops already do this.
First, let me stipulate that all those approaches, procedures, methodologies, etc. mentioned in the base article have a place in the security lifecycle. The statement that the ultimate goal of a PT is to improve the client's security is absolutely right on. Having said that, most of what is described is simply NOT Penetration Testing, unless you simply want to redefine the term (and unfortunately there are already lots of definitions floating around). Not to oversimplify, but a PT is not a Vulnerability Assessment. Period. It has different purposes, methodologies, and endgames. As with a VA, in the end, if the results are communicated poorly, the client loses. On the other hand, if the result and recommendations are communicated perfectly, but the client doesn't take appropriate action, the result is the same. Neither of those possibilities is limited to a PT - you can make the same statement about the value of a VA, code review, architectural review, etc. Hopefully all those other security requirements are already in place, and the more effectively they are implemented, the higher the value of a PT will likely be, as it won't just pick off low hanging fruit. Rather it may expose "unusual" attack vectors that haven't been considered. In fact, most "defenders" and their defenses, fail to think as an attacker, which is where the PT can really add value.
CG, although I agree that a Risk Assessment should occur unfortunately that would be in a separate assessment. Just because a client chooses an internal penetration assessment as opposed to a risk assessment for whatever reason, doesn't then imply that the company actually performing the assessment shouldn't try to provide value. Although, you may not be able to completely understand the business in a limited amount of time, doesn't mean that you shouldn't at least try. For example, if I am doing a penetration assessment and the client is missing 1000 patches across 100 different boxes, I try to understand why this is occurring as opposed to issue a report stating the client has a patch management problem.
Secondly, I think you hit on it "GOOD pentest shops" do already do this. GOOD being the keyword. Notice how I left out the word "plenty". Though I am sure there are a number of great pen test shops that perform value added assessments, there are much more that do not. I have talked to many clients who have had penetration tests performed and as soon as the pen testers compromised their domain, web server, etc, they go home or stop the assessment as if the work has been completed. This is not a value added assessment and in no way helps the client other than to simply say "your security is no good".
"The solution that I am proposing is that instead of focusing so much time on the technical side of penetration testing, that the pen tester needs to also understand the business, its data, the owners, priorities, concerns, risk perception, etc."
Agree with your general idea completely, from the perspective that the most greatest value from pentesting is in helping your client understand *why* they got pwnd. It's not because they missed one patch or had a bug in their web app. It's because that failure represents a failure in the controls that should have come earlier.
The kind of root cause analysis that says, for example, your SDLC is broken in these places, here are ways to address it... That offers a lot more value than telling them which fields in their app aren't doing html entity output encoding each quarter. ;)
Post a Comment