Vulnerability Scanning

Intended Audience: Cloud Service Provider (CSP) technical teams, Information System Security Officers (ISSOs), Third Party Assessment Organizations (3PAOs), Agency Reviewers

Published: August 2024

Goal: Provide implementation best practices for vulnerability scanning in a FedRAMP Authorized (or prospective) Cloud Service Offering (CSO)

GitHub Issue: https://github.com/GSA/fedramp-ops/issues/91

Overview

Vulnerability scanning is a key part of FedRAMP Continuous Monitoring (ConMon). Vulnerability scans are delivered by CSOs to FedRAMP monthly. These scans augment the FedRAMP Integrated Inventory Workbook Template and FedRAMP Plan of Action and Milestones (POA&M) to provide the designated agency ConMon lead and consuming agencies monthly insights into the risk posture of the CSO. This article does not define additional requirements; rather, it clarifies existing program requirements and provides best practices for implementing FedRAMP vulnerability scanning. Applying the information in this article will help improve the identification,prioritization, and remediation of vulnerabilities, which will enable more effective risk mitigation related to those vulnerabilities.

FedRAMP Requirements

FedRAMP’s vulnerability scanning requirements for CSOs are documented in the FedRAMP Vulnerability Scanning Requirements and the appropriate FedRAMP Low, Moderate, and High security control baselines. Vulnerability scan deliverables fall under the ConMon program, which is documented in the FedRAMP Continuous Monitoring Strategy Guide.

The following summarizes the types of vulnerability scanning required for meeting FedRAMP controls RA-5: Vulnerability Monitoring and Scanning and CM-6: Configuration Settings that apply to all baselines:

1. Component Discovery: Conduct scans on all subnet ranges to identify and enumerate components that are running within the authorization boundary, and map those components to IP addresses, open ports, protocols, and services. These scans should cover all of the components listed in the FedRAMP Inventory, and non-ephemeral ports should also match the Services, Ports and Protocols (Table 9-1) in the FedRAMP System Security Plan (SSP). Occasionally, CSOs leveraging an underlying cloud-based infrastructure discover network components that reside in the authorization boundary of the underlying infrastructure. These components should be noted in the comments section of the FedRAMP Inventory as such.
2. Network Infrastructure: Conduct scans with elevated privileges on network infrastructure components to identify vulnerabilities, including on management interfaces. This may be accomplished by configuring the scanner to SSH into the component via a dedicated service account and execute commands to determine if the device is vulnerable. For CSOs leveraging managed services from an underlying cloud infrastructure/platform, review the FedRAMP Customer Responsibility Matrix (CRM) to understand whether the provider or the customer is responsible for scanning specific components.
3. Operating Systems (OS): Conduct scans with elevated privileges on all operating systems running within the authorization boundary to identify vulnerabilities. Where images are built as part of a continuous integration and continuous deployment/delivery (CI/CD) pipeline, FedRAMP encourages cloud providers to accomplish this by an agentless approach at build time if possible. Alternatively, OS vulnerability scanning may be accomplished by deploying individual scan agents on production hosts, along with network-deployed scanning appliances. If an agent-based approach is used, the cloud provider should take extra care to limit and monitor the attack surface introduced by the agent or appliance itself. Scanning operating systems via credentialed SSH access should be avoided unless there is no other option for achieving a privileged scan.  
4. Component Configurations: Conduct configuration scans to identify misconfigured operating systems, containers, network components, databases, or platform services running within the authorization boundary based on Department of Defense (DoD) Security Technical Implementation Guides (STIGs) (or Center for Internet Security [CIS] benchmarks if STIGs are not available).
5. Web Interfaces: Conduct privileged scans on all web applications and APIs running within the authorization boundary, including internally accessible web interfaces. This is generally performed using web application scanners (WAS) or Dynamic Application Security Testing (DAST) tools.
6. Databases: Conduct privileged scans on all databases running within the authorization boundary, including databases leveraged via a platform service offering (i.e., database “as a service”) scoped to the customer-managed configuration. 
7. Containers: Conduct vulnerability scans on container images. This may be achieved via container registry scanning in CI/CD, via agent or “sidecar” based container runtime monitoring, or a combination of both. However, registry scanning in CI/CD should be supplemented by tracking runtime containers to ensure that only scanned and hardened containers are deployed to production.
8. Public Disclosure: Establish a public reporting channel for receiving reports of vulnerabilities in organizational systems and system components per RA-5 (11).  FedRAMP encourages organizations to participate in disclosure programs, such as the Department of Defense Vulnerability Disclosure Program. Additionally, FedRAMP encourages organizations to promote periodic or continuous bug bounty programs.

For the Moderate baseline and above, the following summarizes the types of vulnerability scanning required for meeting FedRAMP controls SA-11: Developer Testing and Evaluation and SA-11(1): Developer Testing and Evaluation | Static Code Analysis in addition to the items listed above:

9. Codebase: Conduct codebase scans using Static Application Security Testing (SAST) and/or Software Composition Analysis (SCA), as applicable, in CI/CD. SAST analyzes an application's source code and compiled binaries (e.g., .exe, .dll, .class files) to identify security vulnerabilities and other defects, while SCA focuses on analyzing the application's dependencies, which include open-source and third-party libraries, frameworks, and components. Both types of scans detect CVEs and CWEs.  

For the High baseline, the following summarizes additional actions required for meeting FedRAMP controls RA-5 (4): Vulnerability Monitoring and Scanning | Discoverable Information and RA-5 (8): Vulnerability Monitoring and Scanning | Review Historic Audit Logs in addition to the items listed above:

10. Discoverable Information: Review for publicly discoverable security-relevant system information, and define and implement procedures for notifying relevant personnel and making system changes if necessary.
11. Security Log and Incident Report Review: Establish a recurring review of historic audit logs, incident reports, and other relevant after action artifacts to determine if CVEs with critical or high CVSS scores have been exploited. This practice may be incorporated into broader system monitoring activities. Relevant indicators of compromise (IoCs) should be defined per SI-4.

How Do Security Teams Meet These Requirements?

Best Practices for General Scanning Tools

Regardless of the type of scanner being used, some best practices apply across the board. The following practices are recommended:

• Ensure that scanners are patched with the latest updates and follow least privilege (per AC-6) and separation of duties (per AC-5).
• Use unique credentials/roles with scanners to allow traceability of actions performed by scanners.
• Ensure that detection identifiers to be scanned are up to date within 24 hours prior to running scans (per RA-5 (2)).  A detection is defined in the FedRAMP Vulnerability Scanning Requirements as an individual program within the scanning tool that checks for a given vulnerability or other data point (authentication, etc.) that is flagged as a finding, and is identified by a unique identifier.
• Ensure that scanners are configured to scan for all non-destructive detection identifiers, including relevant detections for auditing STIG (or CIS Benchmark) hardening implementations.
• Ensure that machine-readable outputs (XML, JSON, CSV, etc.) are exported from the vulnerability scanning tool at least every 30 days. FedRAMP recommends exporting outputs more frequently in case of any issues. 

The outputs should contain the following data types, as available:

• IP Address
• FQDN/hostname
• Unique detection identifier (tool-specific)
• CVE identifier
• Relevant CVSS score
  • Critical/High: 7.0-10
  • Medium: 4.0-6.9
  • Low: 0.1-3.9 
  • Informational: 0.0
• CVSS temporal and environmental vectors
• Original detection date
• Authentication/privilege escalation failures

• Ensure that scan outputs are uploaded to the USDA Connect or CSO repository at least every 30 days, and ensure their availability to FedRAMP and agency customers.

Note: FedRAMP is working to eliminate the need for manual uploads by providing capabilities to ingest these scans on an automated and ongoing basis. For more information, see automate.fedramp.gov.

Best Practices for Non-Containerized Components Leveraging Agent-Based (or SSH Based) Scanning Tools 

To meet vulnerability scanning requirements for systems running traditional physical and/or virtual operating systems, databases, and network devices, the following practices are recommended:

• For network-based scans, ensure that vulnerability scan profiles are configured to include all public and private subnets (every CIDR block) within the authorization boundary. These scan profile targets should match the hosts listed in the FedRAMP Inventory. 
• At a minimum, perform scans at least every 15 days. Scans should be performed more often if necessary to validate remediation or in response to security alerts, advisories, and directives. For example, Cybersecurity and Infrastructure Security Agency (CISA) Binding Operational Directive (BOD) 19-02 requires that CVEs detected on internet-accessible components with a CVSS score of 9.0 or higher are validated as remediated within 15 days of detection. Where applicable, CVEs on the CISA Known Exploited Vulnerabilities (KEV) catalog need to be validated as remediated per the timeline prescribed by CISA BoD 22-01.
• Ensure any physical and/or virtual network appliances used by the vulnerability scanning tool are not restricted by security groups and network access control list (NACL) rules that prevent the scan from working properly. For systems leveraging an underlying infrastructure as a service (IaaS), a best practice is to deploy a scanner appliance within every virtualized network (VPC, VNet, etc.) to reduce the risk of firewall rules blocking access to all components. Take care to keep rules allowing scanner access narrowly scoped to the scanner to avoid weakening network security.
• Ensure that agents are properly configured to actively perform and report a full scan to the server-side application on an ongoing basis. CSOs may tailor the cadence based on available compute resources; however, a best practice is to conduct an agent-based scan and server sync somewhere between every 15 minutes and 24 hours. Organizations are encouraged to utilize blackout windows and CPU limits to further reduce compute costs and workload performance impacts as long as every component deployed to production (or approved sample set) is scanned at the time of deployment and at least once every 15 days.
• If an agent is used, ensure that it is configured to run on the OS with a dedicated privileged role.
• Where possible, for systems leveraging an underlying IaaS, integration with the cloud control plane (i.e., account level) should be achieved via connectors to augment visibility into asset discovery and scan agent uptime.

Best Practices for Containerized Components Leveraging Agentless Scanning Tools 

Systems may leverage automated container orchestration tools to build, test, and deploy containers to production environments. If this is the case, security activities should be accomplished as part of CI/CD. However, systems running container workloads are still responsible for performing web interface, database, and discovery/network infrastructure scans as applicable for the system. For systems running serverless applications, some of the following scanning requirements may be the responsibility of the underlying cloud infrastructure/platform provider. Such service or configuration specific responsibility designations should be defined in the provider’s CRM. To meet vulnerability scanning requirements for systems running container workloads:

• Ensure that container images are scanned for vulnerabilities and configuration findings as part of CI/CD prior to being deployed within the production environment. 
• Ensure that containers are minimized and hardened per DISA STIGs or CIS Level 2 benchmarks prior to deployment within the production environment. Hardening should be validated by scans, including validating that non-essential ports, protocols, and services are disabled in the production environment.
• Ensure that the container orchestration tooling itself is hardened and that container images are pulled from a trusted registry (i.e., controlled by the organization and/or part of a FedRAMP authorized platform service).
• Container integrity and tracking:
  • Consider using a collision-resistant checksum as a unique identifier for individual container images in the registry. The checksum may also be used to validate the integrity of container images. 
  • The checksum may also be used in the FedRAMP Inventory and POA&M as the Unique Asset Identifier to track associated vulnerability findings. 
  • While all containers actively running in production should be tracked internally by the CSO, they do not need to be listed as individual entries in the FedRAMP Inventory, unless they are specifically the target of a production scan performed by a security sensor.
• Ensure that containers actively running in production are being tracked for compliance drift, behavioral, and anomaly monitoring and that noncompliant containers are replaced at least every 30 days. This monitoring may be accomplished via agents running in or alongside runtime containers, the orchestration tool itself, third-party security monitoring tools, or all of these.
• Ensure that all container images that are in the production container registry and are actively running in production:
  • Are re-scanned in the container registry for vulnerabilities and compliance findings, or are replaced with fresh containers at least every 30 days, or 15 days for internet-accessible components with critical findings (or as defined by CISA KEV prescribed timelines). 
  • The scanning window begins as soon as the container is deployed to the production environment. 
  • Only containers derived from images that have been scanned within the scanning window can be actively deployed on the production environment.
• FedRAMP encourages leveraging serverless architectures where possible.  However, in the case of serverless architectures, FedRAMP strongly recommends conducting SAST in CI/CD to detect relevant CVEs (SAST is a requirement per SA-11 for the Moderate and High baselines). Refer to the infrastructure/platform provider’s CRM to understand specific scanning responsibilities. 

Best Practices for Web Applications and Interfaces

• For web application scanning tools (such as DAST), CWE identifiers are acceptable instead of CVE identifiers.
• Performing privileged web application scans on external web interfaces in a production environment may be destructive. In this case, FedRAMP encourages web application scanners to perform authenticated scans on web interface targets identical to production web interfaces hosted in a staging environment.
• Both public-facing and internal-facing web interfaces must be targeted for scans, including network components, APIs, web servers, and COTS application management consoles.
• Intracluster container communication is comparable to communication between processes on an OS. If multiple containers within a cluster are used to implement a web interface, a web interface scan (i.e. DAST scan) should focus solely on the HTTPS web interface exposed via port 443. Web scans of intracluster ports communicating between individual containers within a cluster are not required.

Best Practices for SAST and SCA Scanning

• For SAST and SCA tools, findings with an associated CVE or CWE are tracked and remediated based on FedRAMP remediation timelines. Findings without a CVE or CWE should also be reviewed and remediated where appropriate.
• Many SAST/SCA tools are capable of generating Software Bills of Materials (SBOMs).  FedRAMP authorized CSOs should be capable of generating an SBOM for 3rd party components in the production environment per EO 14028, M-22-18 and NIST SP 800-218, although ideally SBOMs should be supplied by the creator/publisher of the software. SBOMs may also supplement vulnerability triage, tracking, and remediation activities.

Best Practices for Applying and Validating Component Hardening

• DoD STIGs are product specific and should be applied for the following core DoD Security Requirement Guides (SRGs) and corresponding Technology SRGs:
  • Application SRG
  • Network SRG
  • Operating System SRG
  • Policy SRG
• Ensure that all relevant STIGs are applied to production components. Multiple STIGs may be applicable to a single component, especially in cases where the component operates on multiple planes (i.e. the data plane, control plane, and management plane).  For example, STIGs would be applied to both the operating system and to specific software applications running on the operating system.
• If DoD STIGs are not available for a specific component, CIS Benchmark Level 2 may be applied for Moderate and above, or Level 1 for Low authorized systems.
• FedRAMP encourages organizations to leverage pre-hardened operating systems where possible. There are many government and commercial organizations that provide pre-hardened images for this purpose (e.g., U.S. Air Force’s Iron Bank).
• Ensure that scan profiles include STIG audit detection identifiers for all relevant STIGs applicable to the component. 

Best Practices for Vulnerability Tracking and Remediation

Scanning tools are augmented by ticketing and Governance, Risk and Compliance (GRC) tools for tracking and reporting activities and configuration management tools for remediation activities. 

The ability to meet FedRAMP requirements around vulnerability remediation timelines is difficult without the use of system configuration management tools. These tools consist of command-and-control orchestration platforms, as well as an infrastructure as code (IaC) approach to build and deployment via CI/CD. 

To meet vulnerability tracking and remediation requirements:

• Automate the creation of tickets by integrating ticketing tools with vulnerability scanners via APIs wherever possible. This avoids unnecessary costs around repeated manual work and reduces human error. Most popular ticketing tools offer integrations with widely used vulnerability scanning tools. 
  • When configuring these integrations, consider the level of granularity of reporting that works best for your organization. It may make sense to organize vulnerability findings by specific engineering teams based on which assets the vulnerability is detected on, or to group multiple findings in a single ticket based on the overall patching cycle. Tickets do not necessarily need to be one-to-one with POA&M items; organizations are free to customize the approach based on what works best.
• Consider integrating the ticketing tool with a GRC tool that supports FedRAMP’S OSCAL formats. This approach allows organizations to automate the generation of the entire ConMon deliverable on a monthly basis.
• Consider leveraging configuration orchestration tools for deploying patches and making configuration changes at scale without having to apply them per individual host and for continually assessing the security posture of components. It is very difficult to meet the FedRAMP vulnerability remediation timelines without relying on automation.
• Consider adopting an IaC approach to deployment. IaC helps reduce human error (and cost) as systems scale, and enables security and compliance to be baked into the build rather than bolted on in production. 
• Scan IaC code itself prior to deployment to identify potential misconfigurations.
• Consider using standard formats for communicating vulnerability risk such as SBOM VEX and Vulnrichment.

How Do 3rd Party Independent Assessors (3PAOs) Validate Compliance for Vulnerability Scanning?

When responding to the RA-5(*) controls within their authorization package documentation, CSOs should be specific about: 

• The tools used and their applicability to the components that comprise the authorization boundary
• How they are configured
• Which components are scanned and when 
• Any sampling methodology if applicable 

3PAO Testing Guidance

For assessments, 3PAOs should review monthly vulnerability scan output provided by the CSO 60-90 days before the Security Assessment Report (SAR) is due. Important things to look for include: 

• Whether the scanners are up to date and configured to use all non-destructive detections. 
• Whether the scans encompass the components listed in the FedRAMP Inventory.
• Whether the scans detect components that are not referenced in the System Security Plan (SSP) architecture diagram and system description. All major technology components found in the scans should be documented in the architecture diagram.
• Whether the scans include all of the necessary data types described in the “Best Practices for General Scanning Tools” section of this article.
• The percentage of authentication failures.
• Whether ports and protocols discovered in the scans are documented in Section 9:  Services, Ports and Protocols table in the SSP:
  • Ports 0-1023 are rarely ephemeral and should be documented
  • Ports 1024-14191 may be ephemeral or static
  • Ports 14152-65535 are usually ephemeral 
• If detections are disabled, whether there is clarity as to the justification in RA-5 control write up of the SSP, including mitigation for what may be missed as a result. 
• Whether vulnerabilities are remediated or addressed via deviation request within the timelines specified in the FedRAMP Continuous Monitoring Performance Management Guide.
• Whether the scans demonstrate that STIGs (or CIS Level 2 Benchmarks) are applied to all relevant components per CM-6.
• Whether the scans indicate the use of any unsupported components per SA-22.
• Whether SAST scans are being conducted prior to deployment per SA-11.

During the initial or annual assessment, 3PAOs should validate CSO scanning tool configurations and perform their own vulnerability scans to compare them against the CSO's previously submitted scans to ensure consistency and completeness. Any vulnerabilities found are documented within the Risk Exposure Table (RET) or the Risks Corrected during Testing (RCDT) sections of the SAR. 

This annual scanning exercise should include:

• Validation of scan profile target ranges and configuration settings to ensure they match the inventory, and that all non-destructive vulnerability and configuration detections are enabled.
• Validation of configuration settings for container orchestration tooling and the production registry
• Validation of agent uptime and reporting cadence.
• If applicable, validation of SSH or another mechanism of authentication within the scan tool.
• Validation of adequate scope and granularity for assigning and tracking scan findings via ticketing tooling, to include network, OS, database, web, container, and SAST/SCA findings.
• Validation of proper integration into automated CI/CD pipelines for containerized workloads via pipeline hooks. 

How Do Agencies Validate Vulnerability Scan Deliverables?

Agencies should ensure that ConMon deliverables are being uploaded to USDA Connect or the CSO repository at least every 30 days. To validate the deliverable, agencies should perform completeness checks as well as validating that the risk posture remains acceptable. 

These checks may include:

• Checking the vulnerability scan outputs to ensure that they are well formed and valid in machine-readable formats. 
• Comparing the list of scanned components to the Inventory to validate acceptable scan coverage.
• Comparing the vulnerability findings to the POA&M to validate that all findings are represented appropriately.
• Checking whether the deliverable exceeds any late vulnerability or unauthenticated scanning thresholds listed in the FedRAMP Performance Management Guide, and taking appropriate escalation action if necessary.

How Do CSO ISSOs Write an Acceptable System Security Plan RA-5 Control Implementation Summary?

Below is an example of the information that should be included in an SSP RA-5 Control Implementation Summary:

Below are two examples of insufficient control implementation summaries:

How Do Vulnerability Scan Deliverables Impact the Initial FedRAMP Assessment?

CSOs and 3PAOs must ensure that all scan results obtained for a specific assessment/point in time (remediation scan) are in the same format for each deliverable (by scan type and scanner type) (e.g., file structure is unchanged and all fields covered in the original scans are accounted for in the remediation scans). If the scan results do not align, this can prolong the FedRAMP authorization due to the need for the 3PAO or CSO to revise automation.

The below table summarizes common vulnerability scanning deficiencies, and under what conditions they may be acceptable during the FedRAMP authorization process.

NIST SP 800-53 Revision 5 Vulnerability Scanning Related Controls:

• RA-5 Vulnerability Monitoring and Scanning (L)(M)(H)
• RA-5(2) Update Vulnerabilities to Be Scanned (L)(M)(H)
• RA-5(3) Breadth and Depth of Coverage (M)(H)
• RA-5(4) Vulnerability Monitoring and Scanning | Discoverable Information (H)
• RA-5(5) Vulnerability Monitoring and Scanning | Privileged Access (M)(H)
• RA-5(8) Review Historic Audit Logs (H)
• RA-5(11) Vulnerability Monitoring and Scanning | Public Disclosure Program (L)(M)(H)
• SI-2 Flaw Remediation (L)(M)(H)
• SI-2 (2) Flaw Remediation | Automated Flaw Remediation Status (M)(H)
• SI-2 (3) Flaw Remediation | Time to Remediate Flaws and Benchmarks for Corrective Actions (M)(H)
• SA-11 Developer Testing and Evaluation (M)(H)
• SA-11 (1) Developer Testing and Evaluation | Static Code Analysis (M)(H)
• CM-6 Configuration Management (L)(M)(H)

Additional Resources

1. DISA Container Hardening Process Guide Contains details for how organizations may harden containers
2. NIST SP 800-128 Guide for Security-Focused Configuration Management of Information Systems
3. NIST SP 800-218 Secure Software Development Framework (SSDF) Version 1.1: Recommendations for Mitigating the Risk of Software Vulnerabilities
4. NIST SP 800-204D Strategies for the Integration of Software Supply Chain Security in DevSecOps CI/CD Pipelines
5. NIST SP 800-40r4 Guide to Enterprise Patch Management Planning: Preventive Maintenance for Technology