CSO and Pen Tester: A Perspective From Both Sides


With over 25 years of experience working across Cybersecurity and IT disciplines in both offensive and defensive roles, I have enjoyed wearing many hats for both small and global organizations.  Recently, I made the decision to leave a challenging role as a CSO for a SaaS company in order to focus exclusively on work as a Penetration Tester.  Combined with software, systems and network infrastructure ownership experience, earning certifications such as the OSCE, OSCP, CRTE, CISSP, and GCIH has given me a unique perspective.  With this experience, I have learned, not only how to identify vulnerabilities in a company and its network, but how to effectively mitigate them.  After all, the real fun is in cybersecurity is finding possible vulnerabilities and preventing them from being exploited.

But to achieve a comprehensive penetration test of a company’s systems, a cooperative partnership is needed between the owner of the network and the pen testing team trusted to challenge it.  Though it is human nature to want to protect and defend your systems yourself, having a partnership approach to working with your pen test team will not only yield the most comprehensive assessment, but more efficiently position you to keep your system’s and their data out of the hands of the nefarious.

However, there is more to effective penetration testing than just hiring an external team and turning them loose on your network.  With that in mind, I am going to share my experiences and best practices for conducting penetration testing of corporate networks from both sides of the coin, as a CSO (or any systems owner role) and that of the penetration tester.  Several attack vectors will be discussed, as will what a collaborative approach with the pen test team looks like and how it can enhance the outcome of a critical need for a company.

Attack Vector:  External Penetration Tests

The first area of focus is the external network boundary, and to have the most comprehensive engagement, a bit of legwork here can go a long way.  Before a pen test engagement, scan all addresses associated with the company and evaluate all the firewall rules—that’s what I did when I joined a new organization. And I did it while I still had fresh eyes, so if there is a relatively new member of your IT/Security team, this is a great task for them as they are free of the endowment effect.  The other members of the team can help make sure all upgrades are current and all patches have been deployed.  If reboots are required, make sure they have been completed.  You’ll want the pen test team to invest their efforts with the IP addresses and hosts that are valid and are working to the latest and cleanest configurations and builds whenever possible so as to ensure they are focused on the attack vectors, rather than background research.  However, if resource constraints prevent the polishing of these areas ahead of time, it might then be worth scoping additional hours with the pen test team to ensure the engagement has the time for a thorough scan of the external boundary and inspection of each of the services found. 

"...open dialog with your pen tester sets the stage for a cooperative engagement..."

When reviewing this information with the tester, ask them what IP address ranges they will be approaching from, and agree upon a timeframe for when the test is going to occur.  This open dialog with your pen tester sets the stage for a cooperative engagement and ensures they are first focusing on the known entry points to search for vulnerabilities.  Then as time permits, to look for additional areas of exploitation.  While this testing is going on, your security team will be monitoring your systems, in particular the IP addresses the tester is approaching from, which allows for real-time monitoring of the outcome of these attacks.  The results will guide the company on where they can further improve their defenses.

Attack Vector:  Phishing

As a defender, you can have the highest rated spam filter, sandbox checks, antivirus, and proper domain security controls in place (i.e. DMARC, DKIM, etc.), but at the end of the day an attacker with enough time will get the spear phish to that end user.  Once the email reaches the end user’s inbox, what happens when that employee clicks that link or opens the attachment?  How does the employee react when they see a dialog asking for credentials?  Do the monitoring controls alert the security team to a malicious file being dropped on the computer or that a blacklisted web page has been opened?  With all these unknowns, where does a security team focus their defenses?

As a CSO, I provided the penetration tester a list of employees to include in the campaign, as sending the phishing campaign to the entire company is not always feasible.  I would then work with the tester to verify the campaign they came up with would get past my security controls and provide final feedback and approval on the phishing emails to be sent.  If the email pretense was one I knew no one would fall for, I would communicate that to the pen tester.

With this approach, it allows companies to see how the employees would respond, how would the secondary security controls react, how long would it take for the security team to see the alerts, and finally (and often the most insightful finding) which employees would report the phish to the security team, and which would not.  On any penetration test engagement there is a limited set of hours to work on each vector.  With cooperation between the client and security consultant, this test will be more successful in assessing your overall security controls and employee training. 

Attack Vector:  Assumed Breach

Assumed breach testing, aka internal testing, is to me one of the more exciting vectors.  As a penetration tester, you often start with crafting a payload that will bypass the clients’ known defenses, the details of which may be communicated ahead of time by the client.  Anti-virus (AV) and Endpoint Threat Detection and Response (EDR) products have become quick to respond to new attack techniques, which makes this step challenging but well worth the extra effort.

While there is a high likelihood that a penetration tester will be able to craft a working payload, there are times when this will not be the case.  In those cases, the tester may ask to whitelist the payload itself (not disable AV/EDR completely) so that they can proceed forward.  When they ask that this be done, it is mainly because they want to ensure they are spending their time on the most valuable work – testing the companies internal controls.

As a pen tester, this vector is where our clients are the most uncomfortable.  I often see clients push back on allowing a payload, accepting a virtual machine, permitting a small form-factor device to be deployed, or whitelisting a payload.  Then, once this hurdle is addressed, I often find that we’re only given access to a segmented portion of the network, or worse, the payload is placed on a freshly imaged computer that does not represent an actual user who has been breached. 

Cooperation on this vector is very important.  If an attacker makes it through the external defenses, or an employee launches a malicious file from a phishing attack, the need to see how far an intruder can get is critical, and more importantly, how long does it take the security team to find out that they are on the network.  As a CSO, I too was uncomfortable allowing broad access to my network.  However, it was necessary to allow them access to the internal network from either a physical device or virtual machine we place on the network for them, or even from an actual employee workstation, which would achieve the most realistic approach to an assumed breach vector.  Then, the tester would attempt to escalate privileges and move laterally throughout the network, as we would monitor our security systems, like usual, and see if we are alerted.  It is incredibly valuable to know how far a pen tester can get in a limited timeframe, as a truly malicious actor would likely be on the network for a much longer period.

Internal Communications

Though communicating with leadership about a scheduled pen test is absolutely necessary, it is important to be as stealthy as possible, and limit those who know about the engagement to as need-to-know as possible.  This is essential to obtain the most real-world test as possible, and truly see where the IT team and security members/vendors respond to all events as they typically would.  This gives a better view into where the blind spots are in the environment.  This approach also allows for the testing of the security vendors responses to attacks and evaluate their performance, and in some cases, even reveal that they do not have the right security vendor on your team.


As a CSO, I have participated in many penetration tests of my systems, and it was always an exciting and nervous time.  You are optimistic to see how well your employee training and defenses protect your company from the penetration tester, while also wondering what security gaps you missed, or what will be exposed when a phishing attack works.

"The companies I hired to perform our penetration tests were always treated as an extension of our Security team."

The companies I hired to perform our penetration tests were always treated as an extension of our Security team.  I would provide the testers as much detail as possible, to help them perform the best they can, and also make the most of our investment.  With that in mind, a penetration test is limited on time, so if I help the tester by ensuring they had what they needed up front, they would have more time to spend on the higher complexity systems.  In return, we would receive a more comprehensive test and insights into our systems

In talking with fellow penetration testers, they too have experienced working with clients that look at them as the enemy and did not want to help-us-help-them.  We are not your enemy, nor are we trying to get people in trouble or fired.  Our goal is to find as many security holes as we can, in the short period of time that we have, so you are better prepared to protect your company.  As someone who has lost many nights of sleep over the possibility of a breach occurring on my watch, and the impact a breach would have on our Company, our Employees, and our Customers, I would have rather a professional penetration tester find and inform on our vulnerabilities as opposed to being a casualty of a truly malicious actor.

About the Author

John Bullinger

John Bullinger is a Senior Associate with Schellman & Company. John brings 25 years of experience in Cybersecurity and IT, including areas such as: network, data, and physical security design, management and assessment, web application security assessments, PCI and SOC compliance, and project management. His work has covered the healthcare, industrial, academic, energy, SaaS, and retail sectors. He holds a B.S. in Management of Information Systems from Bellevue University and has earned multiple certifications, including OSCE, OSCP, GCIH, CISSP, and PMP.

More Content by John Bullinger
Previous Article
Deterring Attackers with Low Effort in Active Directory
Deterring Attackers with Low Effort in Active Directory

Schellman Penetration Tester Wes Dorman shares techniques for slowing down an adversary's attacks with acti...

Next Article
OSWE Review and Exam Preparation Guide
OSWE Review and Exam Preparation Guide

For seasoned penetration testers who want to become a true web app exploit guru, OSWE certification deliver...