Vulnerability Assessments are not Penetration Tests!

Too often I, as well as many of my co-workers, go into a client and throughout whatever assessment I am working on, general questions come up like, “when’s the last time you’ve had a pen test?” And the client responds, “Ohhh, we do those annually with ‘Some Corporation.’ ” And after looking at ‘Some Corporation’s’ website and seeing what they consider to be a penetration test, I am again disgusted to see that they show up with a vulnerability scanner, run it, validate some findings, and are off to their next client.


Now I know these are some brash comments made toward some random security companies, but let’s be honest here: If you’re going to do something, do it right the first time and provide your client the value of the assessment they paid for. Additionally, give the client what they paid for. If I go to a salesman to buy a sports car and he tries to sell me a Honda Civic, I’m going somewhere else to get what I asked for. On the other side of the coin is the fact that a lot of companies that want a penetration test don’t really understand what it is to begin with. It seems to me that gone are the days of true pen testing when the dreaded “Red Team” shows up to strike real fear into the hearts and minds of Security Practitioners at Fortune 1000 companies.


Any kid in their parent’s basement with savvy computer skills can fire up a Nessus scanner, Web Application Scanners, or Qualys Guard against a network and some of those people can actually interpret the results to make sense out of them. Trust me, everyone on SecureState’s Profiling Team can do that with their eyes closed, but how many security companies out there can actually run a legitimate pen test? I’m not calling anyone out and challenging them, but in all reality, I just want to know how many companies are willing to admit that what they call a “Penetration Test” is actually just a vulnerability assessment? Even worse is the number of companies who perform so called “penetration tests” and truly believe that a vulnerability assessment is the same thing as a pen test.

So let’s all be clear here: a true penetration test is over 85 percent manual and the remaining 15 percent can be a vulnerability scanner to get some other findings in a report in order to provide additional value to the client. And let’s also define manual attacks as to not be ruling out all tools. Using a port scanner is way different than using nCircle, Qualys, or Nessus. Automated scanners like these are the tools that don’t really help a pen test. And just because you use a tool like the Metasploit Framework and many of the tools in Back|Track 4, doesn’t mean you are running a vulnerability scanner. NMAP has the ability to run scripts as well, but again, it doesn’t belong in the Vulnerability Scanner category.

Many times, companies perform Attack and Penetration's due to compliance, or potentially other reasons, which is a bad idea. It gives those companies the opportunity to choose malicious compliance over the desire for truly assessing the security of the entire company. Malicious compliance is a term used when companies do the bare minimum in order to achieve a stamp of approval for whatever standards they are trying to satisfy the needs of. When companies choose to perform pen tests on only their systems affected by compliance, such as PCI or HIPAA systems, they are missing entire networks of systems which aren’t tested. When this happens, companies aren’t getting the true value of what a Pen Test can provide.

SecureState is a trend setting company, and this is where we are going to step in and say, “We Pen Test!” The PCI DSS Council has at least defined what they consider a penetration test. In section 11.3 the Council defines it to be: “vulnerability assessment simply identifies and reports noted vulnerabilities, whereas a penetration test attempts to exploit the vulnerabilities to determine whether unauthorized access or other malicious activity is possible.” Even the EC Council states that, “Penetration testing simulates methods that intruders use to gain unauthorized access to an organization’s networked systems and then compromise them. Penetration testers may use proprietary and/or open source tools to test known technical vulnerabilities in networked systems. Apart from automated techniques, penetration testing involves manual techniques for conducting targeted testing on specific systems to ensure that there are no security flaws that may have gone undetected earlier.”

The SecureState Profiling Team utilizes lower risk vulnerabilities in some systems with additional vulnerabilities in other systems and links them together into larger attacks. By pulling off an attack in this fashion, the Profiling Team utilizes what is called the Vulnerability Linkage Theory in which we can show why it's important to maintain system baselines and other security measures. The Vulnerability Linkage Theory shows how the attack was pulled off by coupling vulnerabilities in many systems to result in the end compromise. For instance, username enumeration from a website, coupled with a brute force attack on the mail system, could allow SecureState to access mail from a company. From here we can email the tech support team and social engineer them into divulging information on how to access the corporate VPN and voila: access to the internal corporate network. There is no way a vulnerability scanner can do that.

Penetration tests zero in to specific systems in order to break in and see what information can be divulged. Pilfering computers and file shares will explain the benefits of Pen Tests by finding the important documents and unencrypted data. Even finding password protected Microsoft Office files can be cracked to release potentially serious data about a company we’re hacking into. Pen Tests can also be used by Security Departments to show why things need to be fixed and get budget to move forward.

There are conflicting views on Pen Tests and Vulnerability Scans. Pen Tests aren’t performed to find vulnerabilities; they are done in order to compromise systems and networks. The main difference between the two is that in a pen test the attackers are actually exploiting vulnerabilities in systems, adding user accounts, and compromising machines across the network. A full or total compromise, which means total control over the entire network, is the end goal of a pen test. Throughout a pen test, the attackers will inevitably generate a list of findings. Many of these findings may be the same as what a vulnerability assessment will also come up with, but there are many vulnerabilities that scanners just can’t find, which leads to the fact that tools can’t think; consultants can. Consultants are able to interpret results and decide on how to use them in order to leverage certain attack vectors against machines and networks.

Don’t get me wrong: I am not discounting the need, want, or value of a vulnerability assessment. These assessments, as well as pen tests, have their place and need. What I am saying is that these assessments need to be better understood in order to know how and when they should be performed. Additionally, there have been companies that run regular vulnerability assessments and the same vulnerabilities keep coming up every single scan. These companies are either overwhelmed with the amount of vulnerabilities present in their networks and don’t know how to fix them, or they don’t see the value or need in fixing them. Penetration tests can enforce the reasoning. In turn, by better understanding what the difference is, the clients will understand what to expect as a final product and won’t be dissatisfied with the results of each test.

Read more!

Penetration Tests: Not just the normal cup of joe

Well, last week celebrates the ever so fun Information Security Summit in Independence Ohio. I usually try to sit in a few topics here and there to see what people are talking about. I must have sat in three different presentations where they preached that only high level risk assessments could find the core deficiencies in a security program. While I tend to agree to an extent on this, they also made the bold claim that penetration testing cannot accomplish this and is only used for "technical" findings.

Every time I hear this at presentations I wish Bobby Bouchey (pronounced "Boo-SHAY") from Waterboy would come out and pummel the guy on stage.



Unfortunately, nothing happens and I have to continue to hope one day Bobby or Terry Tate the office linebacker answers my prayers.





I'm not discrediting risk assessments at all, in fact they are a necessity, however, a penetration test can absolutely bridge technical and high level findings and identify core deficiencies in current process.

Lets take an example we see all the time during a penetration test. Lets say we do a buffer overflow on a third party application, lets pick on HP's OpenView. We see that OpenView has continuous buffer overflows all across the clients network, we also see that Veritas NetBackup has a ton of buffer overflows, lets see, so does VNC, and the list goes on.

One could logically deduce that third party patch management in the organization is failing and there is a process breakdown for the inventory and maintenance of these third party applications.

Lets take this a step further, a client has multiple domains, Domain A and Domain B. Domain A has a 10 character password lockout, 4 invalid attempts, logs everything, uses IPSEC for all communications, using NTLMv2 hashes, and all that other great security stuff you can do through group policy. Domain B has a 3 character password, unlimited attempts, no encryption, LM hashes, etc. etc. etc. We can assume that hardening techniques are not uniformly applied across the entire organization.

My point to all of this is penetration testing has evolved far beyond the "fix this patch and your golden" because we all now realize that if we fix this patch, there's another patch six months down the road that isn't patched and we have the same risk. Instead of doing the "fix this patch, fix this patch" we can say "fix your patch management process". Not many pentesting companies out there are doing this nor even understand what I'm talking about here. It's always been challenging to teach the nerds how to understand higher level functions of security.

Risk Assessments and Penetration Testing compliment each other extremely well. Where organizations say their patch management is an A+, when the pentesters come and rip open all third party applications and their grade is a D-, validation and testing offers and excellent understanding of gaps within current policies, procedures, and standards.

To close this whirlpool of thoughts, penetration testing isn't just for the techno weenies to plug holes, its to understand where the core deficiencies are within current process within the organization. Don't get me wrong, patching those holes are important and winning the battle, but identifying the root cause of those problems will ultimately help reach your secured state and win the war.

Read more!

Defcon – "And this is very illegal! So the following material is for educational use only."

I’m not a hacker, but I live with them. I took the pilgrimage to Defcon, attended by many of the world's best-known security experts, and felt much like the kid reporter in the movie “Almost Famous.” Among other (sometimes bewildering) presentations, Defcon showcases demonstrations of the latest discovered weaknesses in computer systems.

The big brew-haha this year was “The Anatomy of a Subway Hack” of the Boston T that got blocked. A federal judge ordered three college students to cancel a Sunday presentation where they planned to show security flaws in the automated fare system used by Boston's subway. I wouldn’t have thought this was any different than the presentation the SecureState team gave where we released various new tools, including SA Exploiter. However I guess when one of your slides proclaims: "And this is very illegal! So the following material is for educational use only," it draws attention to you.

At SecureState, we believe everyone (most especially those organizations trying to protect themselves) should have access to all information available. The belief is if you hide the findings (zero-day exploits) it’s not going to stop the bad guys who have the time and incentive to find the vulnerabilities themselves. It just keeps the good guys on the forefront.

Many organizations without the resources to properly research the latest and greatest vulnerabilities use penetrations tests to get the results of the research with the ability to see how it affects them specifically. Penetration tests are the foundation of security since you don’t know what you don’t know. Thus, keeping security problems secret, or the “Security through obscurity” idea, doesn’t protect the businesses relying on those systems.

In short, our goal at SecureState is to make security better. We don’t look to disclose things that can hurt people. That’s especially true if there is nothing they can do about it. Releasing exploits and tools gives researchers and ethical hackers the opportunity to learn from the experience we have, gives organizations a better idea about the attacks that are possible, and the steps they need to take to prevent them. The bottom line is that while there are risks, the public good is better served by having knowledge freely available. Besides, H4CK3RS are people too.

Read more!

PCI Compliance: Close but no cigar.

PCI Compliance is on almost everyone's mind and it will continue to grow in popularity as time progresses. Before I start, this is by no means a rant or flaming of the PCI DSS 1.1 standard, in fact, to date the PCI DSS is the most technical security framework really out there to date. There are a lot of positives toward the standard; however it has many shortcomings. I'm not going to list every one, but focus on some of the most pressing issues that we've seen as penetration testers and where the standard really is failing.

First, let’s start with: the standard should never be fully relied upon for organizations overall security posture. A matured security organization typically pulls from many standards like the ISO 27001/27002, NSA IAM, NIST, PCI, etc. One thing that this standard has really pulled through was a start to security in organizations and a start for security to be taken seriously. Generally, banks were the forefront for protecting data due to the sensitive nature of information, and most other organizations security fell in the wind and has since begun to shift in a different direction.

As a penetration tester, and running a team of gifted hackers, we get to see every environment and configuration known to man. Due to PCI's 11.3 "Performing penetration testing at least one a year", we do a variety of penetration testing assessments against PCI Compliant organizations. One of the most alarming statistics is our 63 percent success rate for breaching systems in PCI Compliant organizations. By breaching systems, we're talking about full access to the underlying operating system and potential to further penetrate into the network. The 63 percent doesn't even include access to the back-end databases, login bypasses, and various other issues we find during a pentest.

PCI's 1.1 states in 6.5 to use the OWASP guidelines for securing their systems. PCI 1.1 uses the OWASP 2004 framework for vulnerabilities, which is missing the good ol' malicious file execution amongst others. In addition to this, the ASV scans that need to be performed only check for XSS and SQL Injection. Vulnerability scanners, in general, are pretty rudimentary and basic in vulnerability identification, but only detecting two of the overall top ten is a major issue. In 6.6 a code review or WAF has to be in place, which should hopefully stop SOME of these attacks, but the alarming issue here is we've done successful penetration tests against sites that have undergone a code review, or have a WAF in place and do minor protection against attacks.

WAF are great (if properly configured), don't get me wrong, but they should not be the only layer of defense. Creating a security systems development life-cycle (SSDLC) from the beginning of development and through development and establishing code freezes for security testing before going into production is vital and the preferred method. Web applications have been getting a bum rap and are the major points of entry for external breaches to date. The standard simply states "Installing an application-layer firewall in front of web-facing applications", it does not state anything about allowing exceptions for functionality of the site (and introducing exposures) or hardening techniques. Again, the purpose of the standard was not to secure your systems for you, but at least give you a framework for organizations to implement security.

My next question on the DSS standard is who performs the code reviews looking for these common vulnerabilities? Most VARs have a penetration testing wing, generally consisting of two people that implement product and do penetration testing as a side job (oh and hey, if you buy this product it'll fix your vulnerabilities). These guys generally use tools that don't even touch the web application layer in depth and give false satisfactions and assurance to organizations. There’s no certification process for web applications assessors, it only states "an organization that specializes in application security", if I have Nessus does that make me a specialist in web application security? If I have AppScan, WebInspect, Ounce, Fortify, or any of the others in there, does running a tool make me a specialist?

My last main issue covers the ASV guidelines, quoted directly from the ASV scanning standard:

"Merchants and service providers have the ultimate responsibility for defining the scope of their PCI Security Scan, though they may seek expertise from ASVs for help. If an account data compromise occurs via an IP address or component not included in the scan, the merchant or service provider is responsible. "

We've run into many scenarios where organizations are using this loophole to take systems out of scope for the PCI assessment. While the organization is "responsible" if it becomes compromised, generally the companies we're seeing this as a way to pass the test without ever having any of the security restrictions on the system in place. So as per this statement, a main e-commerce site handling all credit card transactions for the organization can be taken out of scope by the merchant or service provider if they choose. At this point, what is the point in becoming compliant at all? Why study hard for a “C” when you can get an “A” every time without ever studying? “Sure if you get caught cheating its bad, but what’s the payoff, we're never going to get breached!”

To finalize the point I was trying to get out here is organizations are really using this to become "secure", when that really isn't the impact intended. In order for organizations to incorporate security, it has to be a widespread adoption within the organization, pull multiple frameworks and standards, and incorporate them into a regularly tested and updated programs. Relying solely on of PCI compliance and going through the checklist is not going to protect you from a breach by any means.

Read more!