What a penetration test is, and what it is not
A penetration test, or pen test, is carried out by a person with security skills who has written permission to attack a defined target. They look for weaknesses, try to exploit them to show the real impact, and report what they found. The value is in the evidence: a finding that shows an attacker reading another customer's records is much harder to ignore than a line in a scanner export.
Three activities are often confused, and each answers a different question.
| Activity | What it does | What it can miss |
|---|---|---|
| Vulnerability scan | A tool probes a running system or its libraries for known weaknesses and common configuration mistakes. | Business logic, anything that needs judgement, and chains of small issues. It also produces false alarms. |
| Penetration test | A person tests the running application, links findings together and tries to exploit them. | Code paths the tester never reaches. The time is limited, so it is not exhaustive. |
| Secure code review | A reviewer reads the source code, supported by static analysis tools. | Problems that only appear in a deployed environment, such as server configuration. |
They work best together. Our article on penetration tests versus vulnerability scans goes deeper on the first comparison, and the secure code review page covers the third. For a short definition, see the glossary entry.
A report with few findings does not prove the application is secure. It says that, within the time and scope agreed, the tester did not find a way in.
Scope and rules of engagement
Before testing starts, you and the tester agree in writing what may be tested and how. This protects both sides: attacking a system without permission is an offence in many places, and testing the wrong system can disrupt real users. The agreement is usually called the rules of engagement. It should cover:
- Targets: which applications, API endpoints, mobile builds, environments and user roles are in scope, and what is explicitly out of scope, such as third-party services you do not own.
- Environment: a staging copy that matches production is the safest choice. Testing production is possible with care and with limits.
- Timing: the test window, and any hours when testing must pause.
- Techniques: whether denial of service, social engineering and physical access are excluded. They usually are for an application test.
- Data handling: how test data, credentials and findings are stored, and when they are deleted.
- Contacts and stop conditions: who the tester calls on finding something critical, or if something breaks.
Check the rules of your hosting and cloud providers too. Many publish what customers may and may not test on their platforms.
Black box, grey box and white box
These terms describe how much the tester knows in advance.
| Approach | What the tester receives | Best for |
|---|---|---|
| Black box | Only the public address, as an outsider would see it. | Seeing what an unknown attacker could find from outside. Realistic, but time goes into discovery. |
| Grey box | Test accounts for each user role, API documentation and sometimes an architecture overview. | Most application tests. Time goes into the logic that matters, such as access control between users. |
| White box | Full access, including source code and design documents. | The deepest coverage. Best combined with a code review. |
For most products we suggest grey box. Test accounts for an ordinary user, an administrator and a user in a second organisation let the tester check whether one user can reach another's data, which is one of the most common serious flaws in applications.
Web, API and mobile application testing
Web applications
Testers work through authentication and session handling, access control, input handling (injection and cross-site scripting), file upload, business logic and browser security headers. The OWASP Web Security Testing Guide (WSTG) is the standard catalogue of test cases. The OWASP Application Security Verification Standard (ASVS) lists the requirements a secure application should meet, at three levels of rigour. The OWASP Top 10 is a useful awareness list, but it is not a test plan.
APIs
APIs deserve their own testing because they expose functions that the interface may hide. Testers check authorisation for each object and each function, mass assignment, rate limiting, and responses that return more data than the screen shows. The OWASP API Security Top 10 is the usual reference. Give the tester an API specification, such as an OpenAPI file, so every endpoint is covered.
Mobile apps
Mobile testing covers the app on the device and its traffic to the back end: data stored insecurely on the phone, secrets inside the app package, weak certificate validation and exposed components. The OWASP Mobile Application Security Testing Guide (MASTG) describes the tests, and the Mobile Application Security Verification Standard (MASVS) the requirements. The back-end API is tested alongside the app.
Broader frameworks such as the Penetration Testing Execution Standard (PTES) and NIST SP 800-115 describe the whole process of planning, testing and reporting.
How findings are reported and prioritised
A good report has two layers: a short summary that a non-technical decision maker can act on, and a detailed entry for every finding. Each entry should include:
- A clear title and the affected component.
- A severity with its reasoning, based on likelihood and impact, often alongside a CVSS score.
- Steps to reproduce and evidence, such as requests and screenshots, so a developer can see the problem first-hand.
- The impact in plain language.
- A specific recommended fix.
Severity should reflect your system, not only a generic score. A medium-rated weakness in a payment flow can deserve more urgency than a high-rated one in a page nobody uses. A sensible order is critical and high findings first, then quick wins, then the rest scheduled into normal work. A good report also says what held up well and what was not tested.
Retesting
A fix is only a claim until someone checks it. A retest repeats the original steps for each finding and marks it fixed, partly fixed or still open. Agree in advance whether a retest is included and how long after the report it can be requested. It also catches fixes that close one route and leave a similar one open, such as blocking an attack through one input field while three others stay unprotected.
How to prepare
- Fix the scope: the applications, user roles and environments to test.
- Prepare a staging environment that mirrors production, with realistic but non-sensitive data.
- Create test accounts for every role, including two accounts in separate organisations if your product serves several customers in one system.
- Share documentation: the API specification, an architecture diagram and any earlier threat model.
- Decide who needs to know the test dates, such as your hosting provider, monitoring team and support, so nobody mistakes it for a real attack.
- Take a backup and name one person who can respond during the test window.
- Fix known issues first, so the tester's time goes to what you have not found yet.
How often to test
There is no single schedule. Common moments are before the first launch, after major changes to architecture, authentication or payment flows, on a regular cycle for important systems (many teams choose once a year), when a customer or partner asks for evidence, and after a security incident. Between tests, habits such as dependency scanning and code review reduce what a test will find, so the tester's time goes to harder problems. If a contract or standard sets a testing interval, confirm the exact requirement with the party that sets it.
What to ask a provider
- Who will do the testing, and what is their experience with your technology and type of application?
- Which methodology do they follow: the WSTG, ASVS or MASTG?
- Can you see an anonymised sample report?
- How do they report a critical finding during the test, rather than at the end?
- Is a retest included, and for how long after the report?
- How do they handle your data, credentials and any access they are given, and how is it deleted afterwards?
- How do they limit risk when testing production?
- Do they test business logic by hand, or mostly run scanners?
Be cautious of any provider that promises to find every vulnerability, or that quotes without asking about scope.
At Asphaleia, every project begins with a threat review and has a security review before launch that gives you a prioritised fix list. Web and mobile application testing is part of our cybersecurity service. If we also built your product, say whether you want a separate, independent tester. That is a fair thing to ask for. Businesses in Qatar can see how this fits local needs on our Qatar cybersecurity page.