How to Add Cybersecurity as a Criterion in a Product Bake-off

Organizations increasingly rely on intertwining technology to power their operations. The importance of validating the lack of negative security impact of adding a new software supplier has never been higher.

As amply demonstrated in 2022 trends in third-party supplier attacks, a breach in a single supplier’s infrastructure could have severe consequences for the organization. 

Securing the supply chain starts at the product bake-off stage. A bake-off, or proof of concept process, consists of comparing competing technologies and selecting the most aligned with the list of requirements and budget considerations. Often forgotten in the process is the cybersecurity impact. This oversight potentially catastrophic consequences even led MITRE ATT&CK to draft a supply chain security framework. 

Yet, that framework relies heavily on the supplier's Software Bill of Material (SBOM). Though an essential factor in evaluating the security aspect of a potential software supplier, a product bake-off should include a direct measurement of the potential risk introduced by each candidate supplier. 

This requires actually measuring that impact, or, ideally, lack thereof, directly in the organization’s environment. 

How to Quantify the Cybersecurity Risk Impact of Connecting with a Candidate Third-Party Service   

In a B2B environment, offering a free trial for connected services is the norm. By definition, running a product bake-off implies connecting the candidate product to the organization’s infrastructure. This is a golden opportunity to evaluate the cybersecurity implications by comparing the security score before and during the bake-off trial. 

What is a Security Score? 

Exposure management software - such as Breach and Attack Simulation (BAS), Attack Surface Management (ASM), or Virtual Red Teamer (VRT) - typically provide a security score based on the success rate of attack simulations in the validated environment. Cymulate’s security score is calculated by correlating the attack simulations success rate with CVSS scores, MITRE ATT&CK factors, and DREAD (Damage potential, Reproducibility, Exploitability, Affected users, Discoverability) risk assessment factors. 

This security score is easily viewed from the dashboard. 

Quantifying Additional Third-Party Exposure During Product Bake-off 

 Evaluating the risk for additional exposure from each competing third-party provider is a straightforward three-step procedure:

• Step 1 - Run a full assessment without any connected bake-off product and keep the generated report and security scores. The assessment runs automated attack simulations that evaluate the organization's security without any of the candidate suppliers.  
• Step 2 – Connect the product and rerun the assessment. 
• Step 3 – Compare the security scores and keep track of the delta between the Cymulate score before and after trying the connecting service. That delta quantifies the added risk posed by the candidate supplier product or service. 

After repeating the procedure for each competing product, compare their respective delta. The difference in delta value reliably appraises the added risk, if any, each prospective product poses to the organization’s infrastructure. 

More details about the potential added risk specificity are available as well. The assessment measures the risk granularly, scoring attack vectors individually, making it easy for all stakeholders to decide whether to include or exclude prospective solutions.    

Incorporating the Resulting Score in the Product Bake-off Selection Process 

There are two main ways to incorporate those results into the selection process. 

• As a disqualifying filter: Security stakeholders in the selection process can determine a tolerance factor for added exposure that matches the organization’s risk baselines for each environment potentially linked with the solution. Any candidate solution that creates an added risk superior to the tolerance level could be automatically excluded. 
• As a bargaining element: for the solutions that include an added risk factor within the risk tolerance level, uncovered vulnerabilities can serve as a bargaining chip in the negotiation process with the third party. 

Some suppliers, ideally all of them, when informed about the uncovered security gaps, might offer to address the issue. They might even request an opportunity to be reconsidered once the problem is fixed. 

When relevant, the trial assessment process should be rerun in full. This is crucial to validate that the fixes implemented by the supplier are also effective in preventing the creation of security gaps within the organization’s infrastructure. 

How to Apply this Quantification Process to Services Provided by Existing Third-Party Providers

The only way to apply this process with existing suppliers is to interrupt their service and follow the inverted test. That is, interrupting the service and comparing the security scores with and without the service running. 

However, this could be unpractical as it might disrupt operations. 

An alternative option would be to condition service renewal to a third-party security score validating the third-party security posture on an ongoing basis. 

Though this would not provide an exact evaluation of the security impact on site, it would give a better evaluation of the cybersecurity reliability of the provider. 

How deep should such a product bake-off test be? 

The extent of the security side of the bake-off test can vary based on: 

1. The sensitivity of the connected assets 

2. The type of continuous security posture validation used by the organization

Depending on internal factors ranging from size, compliance requirements, and others, the extent of security validation technological coverage varies between organizations. This will directly influence the potential scope for the product bake-off test. It cannot exceed the organization’s available security validation active technologies. These technologies are: 

• Breach and Attack Simulation (BAS): a core security validation technology, BAS assesses the efficiency of security controls. This consists in running attack simulations in the organization’s environment and verifying which ones are detected and, ideally, stopped.
  Connecting an external service to an internal environment opens new potential penetration routes detectable through the product bake-off process described above. 
• Lateral Movement: Lateral movement technologies assess a system’s defense ability to prevent an attack lateral or vertical progression across known attack routes. The Cymulate Lateral Movement module includes the attack simulation assessment ability to modify its initial intended route when encountering an obstacle and attempts to find an alternative attack path. 
• Purple Teaming: Complementing BAS set of off-the-shelf attack simulations scenarios, a purple teaming framework allows cybersecurity teams to accelerate the creation of environment-specific attack scenarios. The Cymulate advanced scenarios module provides a framework with a bank of attack executions that can be chained together or enriched with custom code to match the desired assessment scope. These bespoke customized attack templates can be stored and rerun at will. 

3. Obtaining consent from the third-party 

4. Running an Attack Surface Management (ASM) assessment test

Run the ASM test on the prospective suppliers consists of simulating cyber-attackers' reconnaissance attack phase procedure. Practically, it entails scanning the Internet to uncover exposed digital assets potentially exploitable by adversaries, such as domains, subdomains, ports, and other Internet-facing vulnerabilities and Open-Source Intelligence (OSINT). This gives an indication of the third-party security consciousness but no information about the direct security impact of connecting the organization’s infrastructure with the third party.

Additionally, when new intelligence about threats involving an organization's supplier or other third parties emerges, Cymulate provides the ability to assess the impact on the organization and assists in planning defensive and remediation efforts.