What to Do When Automated Application Security Testing Falls Short

As the number of applications developed explodes, the prospect of performing Application Penetration Testing on each application, with limited budgets and scarce resources, becomes increasingly daunting and seemingly impossible. Application risks will never be sufficiently mitigated by relying on automated scanning alone. Typically, there are three main dynamic options practiced today, and they vary in coverage, accuracy, and cost.

The volume and sophistication of cyber attacks continues to increase at an alarming rate. To ensure sufficient protection, organizations must answer these questions:

  • Do we have vulnerabilities that an attacker could find?
  • If an attacker found them, could they be exploited?
  • If exploited, what damage could they do to our business
  • What should be done to fix the vulnerability?

What about Automated Scans?
Application risks will never be sufficiently mitigated by relying on automated scanning alone. Typically, there are three main dynamic options practiced today, and they vary in coverage, accuracy, and cost.

  1. Automated Scanning
  2. Automated Scanning with Manual Validation
  3. Application Penetration Testing (Automated Scanning with Manual Validation and Manual Testing)

To determine which option is right for each application, it is common to take a risk-based approach to classify applications.
This risk-based approach is used to influence the type of assessment that each application requires. Although risk-based classification is an effective way to prioritize limited resources, it leads to the conclusion that automated scanning alone is acceptable for some applications, when in fact, this is rarely the case.

Automated Scans consistently miss high-risk vulnerabilities
At TUV Rheinland OpenSky, we have hundreds of examples that form a clear pattern: automated scans, even with manual validation, consistently miss high-risk vulnerabilities that leave organizations exposed.

Automated scans are well suited to efficiently find certain types of application vulnerabilities, including Cross-site Scripting, SQL Injection, and Server-Side Request Forgery. Automated scans also identify particular misconfigurations, including incorrectly implemented TLS or the absence of recommended security-focused HTTP headers and cookie attributes.

What Automated Scans miss
Automated scans fail to identify many complex vulnerabilities, including Authentication Bypasses, many types of Access Control Weaknesses, and flaws in business logic. Also, automated scans contain scores of false-positives, and use generic risk ratings that can lead to significant levels of wasted effort if not identified and addressed.

Manual validation of automated scan results by security professionals removes false-positives and adjusts risk ratings to an organization’s context; but manual validation will not improve coverage.

Automated Scans leave organizations exposed to risk of cyber attack
In the examples below, see how relying on automated scans with manual result validation would have left each organization with a false sense of security and exposed them to risk of cyber attack. The findings highlighted in red represent significant additional risk that would not have been addressed if the client had relied solely on Automated Scans.

Example 1: A pre-production public-facing customer application

In Example 1, we see that the automated scans missed five significant vulnerabilities, including a critical vulnerability that would have allowed any authenticated user to gain access to highly sensitive data from other customer accounts.

Example 2: A third-party order verification application

In Example 2, we see that an automated scan without Application Penetration Testing would have left the organization unaware of multiple ways that any user could gain full read/write access to any other user’s data and access.

Example 3: A crypto currency user account management application

In Example 3, we see that while automated scanning identified multiple vulnerabilities, it missed that a third-party JavaScript file was sending valid JSON Web Tokens back to the third party, which meant that any active session could be hijacked by the third party. The manual testing also identified a stored XSS vulnerability that only presented itself when a user manually reviewed the malicious payload in a separate application (Second Order XSS). 

Test regularly, manually and adjust frequency based on risk
These examples demonstrate that, while it is necessary to find and fix the vulnerabilities identified by automated scans, it is critical to be aware that it is far from sufficient. Without also performing manual testing with experienced Penetration Testers you’ll miss dangerous and damaging vulnerabilities buried in the heart of your applications.

In today’s cyber threat landscape, it is essential that all applications be manually tested by experienced Penetration Testers. If they aren’t, many sophisticated and high-risk vulnerabilities will be missed. If an organization’s approach to Application Testing overlooks these vulnerabilities, it will leave sensitive corporate and customer information at risk.

Instead of asking “which of my applications require Manual Penetration Testing?” ask “How frequently does each application need Manual Penetration Testing?”
This approach enhances standard practices by using a risk-based classification to prioritize the order and frequency in which applications receive Manual Penetration Testing and ensures that all applications receive this level of screening at least once.Today, all organizations face the same reality: digital business are moving into online and mobile applications that need protection, despite scarce resources. It is critical to understand and recognize the limitations of Automated Scans and establish a risk-adjusted frequency for the manual testing of every application in your portfolio.

Back to Blog