Vulnerability Assessments are not Penetration Tests!

Too often I, as well as many of my co-workers, go into a client and throughout whatever assessment I am working on, general questions come up like, “when’s the last time you’ve had a pen test?” And the client responds, “Ohhh, we do those annually with ‘Some Corporation.’ ” And after looking at ‘Some Corporation’s’ website and seeing what they consider to be a penetration test, I am again disgusted to see that they show up with a vulnerability scanner, run it, validate some findings, and are off to their next client.


Now I know these are some brash comments made toward some random security companies, but let’s be honest here: If you’re going to do something, do it right the first time and provide your client the value of the assessment they paid for. Additionally, give the client what they paid for. If I go to a salesman to buy a sports car and he tries to sell me a Honda Civic, I’m going somewhere else to get what I asked for. On the other side of the coin is the fact that a lot of companies that want a penetration test don’t really understand what it is to begin with. It seems to me that gone are the days of true pen testing when the dreaded “Red Team” shows up to strike real fear into the hearts and minds of Security Practitioners at Fortune 1000 companies.


Any kid in their parent’s basement with savvy computer skills can fire up a Nessus scanner, Web Application Scanners, or Qualys Guard against a network and some of those people can actually interpret the results to make sense out of them. Trust me, everyone on SecureState’s Profiling Team can do that with their eyes closed, but how many security companies out there can actually run a legitimate pen test? I’m not calling anyone out and challenging them, but in all reality, I just want to know how many companies are willing to admit that what they call a “Penetration Test” is actually just a vulnerability assessment? Even worse is the number of companies who perform so called “penetration tests” and truly believe that a vulnerability assessment is the same thing as a pen test.

So let’s all be clear here: a true penetration test is over 85 percent manual and the remaining 15 percent can be a vulnerability scanner to get some other findings in a report in order to provide additional value to the client. And let’s also define manual attacks as to not be ruling out all tools. Using a port scanner is way different than using nCircle, Qualys, or Nessus. Automated scanners like these are the tools that don’t really help a pen test. And just because you use a tool like the Metasploit Framework and many of the tools in Back|Track 4, doesn’t mean you are running a vulnerability scanner. NMAP has the ability to run scripts as well, but again, it doesn’t belong in the Vulnerability Scanner category.

Many times, companies perform Attack and Penetration's due to compliance, or potentially other reasons, which is a bad idea. It gives those companies the opportunity to choose malicious compliance over the desire for truly assessing the security of the entire company. Malicious compliance is a term used when companies do the bare minimum in order to achieve a stamp of approval for whatever standards they are trying to satisfy the needs of. When companies choose to perform pen tests on only their systems affected by compliance, such as PCI or HIPAA systems, they are missing entire networks of systems which aren’t tested. When this happens, companies aren’t getting the true value of what a Pen Test can provide.

SecureState is a trend setting company, and this is where we are going to step in and say, “We Pen Test!” The PCI DSS Council has at least defined what they consider a penetration test. In section 11.3 the Council defines it to be: “vulnerability assessment simply identifies and reports noted vulnerabilities, whereas a penetration test attempts to exploit the vulnerabilities to determine whether unauthorized access or other malicious activity is possible.” Even the EC Council states that, “Penetration testing simulates methods that intruders use to gain unauthorized access to an organization’s networked systems and then compromise them. Penetration testers may use proprietary and/or open source tools to test known technical vulnerabilities in networked systems. Apart from automated techniques, penetration testing involves manual techniques for conducting targeted testing on specific systems to ensure that there are no security flaws that may have gone undetected earlier.”

The SecureState Profiling Team utilizes lower risk vulnerabilities in some systems with additional vulnerabilities in other systems and links them together into larger attacks. By pulling off an attack in this fashion, the Profiling Team utilizes what is called the Vulnerability Linkage Theory in which we can show why it's important to maintain system baselines and other security measures. The Vulnerability Linkage Theory shows how the attack was pulled off by coupling vulnerabilities in many systems to result in the end compromise. For instance, username enumeration from a website, coupled with a brute force attack on the mail system, could allow SecureState to access mail from a company. From here we can email the tech support team and social engineer them into divulging information on how to access the corporate VPN and voila: access to the internal corporate network. There is no way a vulnerability scanner can do that.

Penetration tests zero in to specific systems in order to break in and see what information can be divulged. Pilfering computers and file shares will explain the benefits of Pen Tests by finding the important documents and unencrypted data. Even finding password protected Microsoft Office files can be cracked to release potentially serious data about a company we’re hacking into. Pen Tests can also be used by Security Departments to show why things need to be fixed and get budget to move forward.

There are conflicting views on Pen Tests and Vulnerability Scans. Pen Tests aren’t performed to find vulnerabilities; they are done in order to compromise systems and networks. The main difference between the two is that in a pen test the attackers are actually exploiting vulnerabilities in systems, adding user accounts, and compromising machines across the network. A full or total compromise, which means total control over the entire network, is the end goal of a pen test. Throughout a pen test, the attackers will inevitably generate a list of findings. Many of these findings may be the same as what a vulnerability assessment will also come up with, but there are many vulnerabilities that scanners just can’t find, which leads to the fact that tools can’t think; consultants can. Consultants are able to interpret results and decide on how to use them in order to leverage certain attack vectors against machines and networks.

Don’t get me wrong: I am not discounting the need, want, or value of a vulnerability assessment. These assessments, as well as pen tests, have their place and need. What I am saying is that these assessments need to be better understood in order to know how and when they should be performed. Additionally, there have been companies that run regular vulnerability assessments and the same vulnerabilities keep coming up every single scan. These companies are either overwhelmed with the amount of vulnerabilities present in their networks and don’t know how to fix them, or they don’t see the value or need in fixing them. Penetration tests can enforce the reasoning. In turn, by better understanding what the difference is, the clients will understand what to expect as a final product and won’t be dissatisfied with the results of each test.

Read more!

Passing the PCI Buck

If you've been following any of the recent PCI (Payment Card Industry) breaches, you'll see two trends coming from the breached organization. One one hand, they'll say that the PCI standard is flawed since it they were compliant and still got hacked. On the other hand, they'll say the problem was their QSA (Qualified Security Assessor) messed up. Although both the standard could be better and a QSA can be more thorough, it's time for organizations to stop passing the buck and admit that they screwed up.

Now I understand that there are certainly differences of quality in each QSA. I've seen everything from rubberstamped reports that make you wonder if the QSA can spell 'PCI' to a tinfoil hat QSA who uses the standard to create new, crazy requirements. But in the end, I sometimes wonder if the organizations understand truly what makes a good QSA and what their role is. Ultimately, the QSA is supposed to be an auditor (yes, I know technically we are 'assessors'). That means the QSA is trying to make sure the processes the organization established are working properly. In order to do that, the QSA needs to typically sample the controls and do some digging around. This should allow the QSA to determine that the control is generally operating correctly. However, the QSA is NOT the master of the organization's destiny for security. Heck, the audit is only once a year and cannot turn over every stone. The organization owns and is responsible for the processes, the controls, monitoring, and reacting. If they break down, that is the organization's problem.

Now as far as the standard goes, I hear a lot of people (generally non-QSA) stating that the PCI DSS is flawed. First of all, let's just understand that this was written to push compliance, not security, to a certain level since it was severely lacking in most organizations. Just to be clear, compliance is just a hurdle and not a ceiling. What I don't understand is then what other framework or compliance is better than PCI? For example, ISO 27001 is still too general and based on business decisions. The standard is still the most thorough, prescriptive, and layered one out there. This is not only for the controls within it but also the testing. For example, testing applies both externally and internally and escalates from scans, to pentests, and even web app reviews. For all the naysayers out there, I'd love to see a better version from them. Remember, PCI was the first and still the onyl one out there that has pushed hard for web application security such as the OWASP Top 10. Now for those who think the standard isn't detailed enough, give me a break. It isn't going to tell you exactly how to do X with product Y in a Z architecture. There are just too many variables.

So let's look at the latest details on the big breaches (http://www.wired.com/threatlevel/2009/08/tjx-hacker-charged-with-heartland/ ). Now we are finally starting to get enough details to understand what happened. Let's go with the assumption that the downfall to Heartland/Hannaford was SQL injection and see where the problem hypothetically is. Well the standard clearly states that it needs to be properly coded in requirements 6.5.2. Keep in mind that a QSA will not and may not even be able to review all your code. Oh yeah, that is the organization's requirement in 6.3.7. This also needs further tested through a web application review (possible third party problem) or prevented through a web application firewall (organization's problem) in 6.6. Not to mention that this would or should be detected during the vulnerability scanning (ASV problem) and penetration testing (probably third party problem) from requirements 11.2 and 11.3. So how exactly did the standard fail here when there are at so many layers of controls and testing designed to stop this?

This brings me back to yet another ranting point. It is unbelievable the number of organizations who are constantly trying to find the absolute cheapest vendor for any of the stuff we just discussed including the QSA, the ASV, and other testing. There is little going into understanding the quality or qualifications of the vendors being selected by the organization. This absolutely leads to 'getting what you pay' for quite a bit. But just remember, that is a choice of the organization to pick a deficient vendor.

We also need to realize that even if SQL injection had worked, there were still plenty of other controls within PCI that could have stopped the breach. First of all, there is a anti-malware requirement. Yes, I know that only recently was it clearly required for all platforms, not just those 'commonly affected'. But we have seen Linux malware and it's supported by all commercial anti-virus vendors and even free ones. So did the organization just not do it because it wasn't clearly dictated as a hurdle within the standard? I guess so. There is also a requirement to restrict firewall traffic both inbound AND outbound. So why was the malware allowed to talk to the internet from payment systems? There is no good business reason there. So the standard certainly isn't as flawed as people want to assert, though there it could be improved.

In the end, we certainly can conclude a few things. First of all, despite the fact that these organizations had been assessed as compliant, the organizations clearly were not compliant. Though it is possible that a QSA missed something, that is still the organization's problem. Also, the standard cannot be perfect but it is very well built and, if followed, would have stopped these. In security, like any process, it may only take one flaw for a breach to occur - even if that was the standard or the QSA. But that's why security has to be layered and has to be based on quality, not compliance. Regardless, the whole security process is owned by the organization and they need to take responsibility when it breaks. Passing the buck doesn't undo the breach, but does make for some interesting blamestorming to watch.

Read more!