Constellations FAQs

1 GB - Constellations relies on Glasswall Halo and shares the same maximum file size.

Yes

Constellations is designed to be run in a secure environment, therefore none of your data is sent to Glasswall. Glasswall reserves the right to request summary log information to verify license conformance but data does not leave the environment without intervention by the system owner.

In the future, most of the services will use a hardened Alpine base image. Where possible we implement CIS (Centre for Internet Security) guidelines for hardening. Alpine provides a small Linux distribution with a minimised attack surface which is attractive from both a performance and security perspective.

Please see Performance and Scaling.

Azure Blob Storage.

Logs for all of the Constellations services are written to `stdout` (Standard Output) for their respective docker containers.

Constellations deployments require two node pools in order to separate Glasswall Halo resources, from the scan management nodes. This separation allows for customisation of Node size/configuration.

For example, Glasswall Halo may require larger nodes with higher memory and cpu limits in order for the engine service to process complex files.

Constellations services communicate via RabbitMQ messages using the AMQP protocol. This data is not encrypted during transit. Messages are only ever sent inside the Kubernetes cluster.

Requests to the Scan Controller API and Event Projection API are received via TLS.

Requests from Constellations to Glasswall Halo are sent via TLS.

The hardening strategy for Constellations encompasses two aspects: 

• Hardening of the development process, which focuses on strengthening security measures during code creation, storage, and access.
• Hardening of the consumable components, used within the product.

The hardening process for the development lifecycle comprises multiple stages. We incorporate security gates at each stage of the development lifecycle, ensuring that only secure, vulnerability-free code is integrated into our code base. This process involves implementing Static Application Security Testing (SAST), Software Composition Analysis (SCA), Infrastructure as Code scanning (IaC), and Dynamic Application Security Testing (DAST) tools into our Continuous Integration/Continuous Deployment (CI/CD) pipeline.

Similarly, the hardening process for the consumable components is a crucial part of our overall approach. We diligently maintain hardened Docker images in-house, subjecting them to thorough scrutiny and ongoing monitoring. Consequently, we confidently state that the images used in Constellations undergo a hardening process that aligns with, at the very least, the minimum security standards set by the National Institute of Standards and Technology (NIST). 

Kubernetes Version: v1.25.6 or above.

The action of cancellation stops the cataloguing action of the scan from finding files in the specified container. Any files that have already been found and submitted for processing will continue to be processed.

Constellations uses the quarantine folder to store a JSON quarantine report for each file. This includes some of the original file metadata including: file size, file type, input and output hashes, and location in the source container.

A quarantined file will never be copied to an output container, this is to prevent duplication of potentially malicious files. The quarantine report allows a safe space to understand more about the file, allowing users to take further action.

Sometimes, Glasswall Halo fails to rebuild or analyse a file, resulting in an error status sent back to Constellations. If this happens, the quarantine report is populated with descriptions and error codes in the `errors` section. More information on these errors can be found in the Glasswall Halo API Documentation.

Non-successful status codes can occur for a number of reasons:

• The file exceeded the maximum file size limit.
• Processing took too long and the configurable timeout was reached.
• The file is masquerading as a supported file type but is actually a different file type.

Example: 

json 
"Errors": [
  {
    "ErrorCode": 5003,
    "ErrorDescription": "Failed to detect file type"
  }
]

When express mode is enabled, only supported file types will be sent to Glasswall Halo for processing.

Advantages

Unsupported files are not processed, saving the CPU and Memory required by the Glasswall engine to identify the file.

Disadvantages

Relies on the file extension of the file, potentially supported files could either be masquerading as other filetypes, or missing a file extension. This could cause supported files to be quarantined.

Constellations services use Kubernetes Event Drive Auto-Scaling (KEDA) to scale from a minimum number of replicas, to a maximum. Constellations expects a high volume of throughput and thus will require node pools scaling to meet the demands of the various services. 

A 'cold-start' has been observed with node scaling in AKS. This is the reason for a percieved ramp-up of throughput over time. 

In order to mitigate this cold-start, nodes can be pre-warmed:

• Set the CPU and Memory requests/limits to the same value for the `cdrplatform/engine` and `cdrplatform/api` services. Guaranteeing each pod has the same resources.
• Manually scale Glasswall Halo and Constellations services to the desired replica count, which will trigger node scaling to meet the CPU and Memory demands.

For storage accounts in Azure we recommend a minimum grade of `StorageV2 (general purpose v2)`. Increasing the grade shouldn't affect throughput.

For Cosmos instances - in order to reach a processing time of 500GB in 30 minutes, the total throughput limit should be set to 3000 RU/s. This is to prevent rate limiting and increasing this limit further has no effect on processing time.

Data Security 

• Connection strings for both source and destination storage accounts are provided by users on a per-scan basis. This security best practice is for the users to share connection strings that expire within a limited time period.
• No customer files are held within the product once the CDR processing is completed.

Network Security

• A Private AKS cluster is recommended with private endpoints on both Cosmos and Blob Storage.
• TLS certificates should be configured on the Ingress into the Scan Controller and Event Projection APIs.

Authentication and Authorisation

• The Scan Controller and Event Projection APIs support integration with Azure Active Directory (AAD). We recommend authentication is Enabled in both of these services.
• Kubernetes Secrets, along with Azure Key Vault can be used to store the credentials for Cosmos, RabbitMQ and AAD.

Every precaution is taken to ensure Constellations never executes malware. In the event that a malicious file brings down a pod, the damage is contained as docker containers are given restricted access to specific folders in specified areas of the filesystem and Constellations services are run in non-root user groups.

An alternative to using the Glasswall ACR in your production environment is to push the Constellations Helm charts and Docker images to a private ACR.

This can be achieved by pulling the desired images and charts to an intermediary location (for example: a local machine or a bastion server) and then pushing them into your private repo. Then, while installing the Helm charts, update the 'image.repository' tags to point to the correct repo.

Currently, only Azure Blob Storage is supported.

Provided the scan/job has completed, it is safe to delete the source and destination containers. The results of the scan will persist in the Cosmos Database.

If the source container is deleted, it will no longer be possible to perform a rescan.

The CDR Enabler service will re-create the quarantine/target containers as long as the shared access signature is still valid.

A maximum of 5 levels of nesting in archives is supported by Glasswall Halo and Constellations. Once Glasswall Halo reaches this nesting limit, no further processing takes place.

While 'basic' archive processing mode is enabled the entire archive will be marked as `Errored` and a quarantine report will be stored in the quarantine container.

With 'detailed' archive processing mode enabled the overall file will be marked as `Failed`. A clean file with child archives (up to the limit of 5) will be stored in the target folder. The first child archive past the nesting limit is replaced with a `.txt` placeholder and marked as `Errored` with a quarantine report stored in the quarantine container.

Azure - AKS

Single files (and archives while `basic` archive processing mode is enabled) that fail to be processed will not be stored in the target folder, instead a quarantine report is generated and stored in the specified quarantine folder.

In 'detailed' archive processing mode, partially rebuilt archives are stored in the target folder, but the quarantined files are replaced with `.txt` placeholders.

There is no silver bullet in security. Glasswall’s CDR technology doesn’t attempt to detect malware and therefore, unlike many other security tools we don’t have a false positive and false negative rate. Glasswall provides an important level of certainty to the system owner. If the file has been protected by our comprehensive level of CDR, malware does not have a hiding place within the file.  

Sometimes a file can deviate markedly from the safe specification for that file type, making the CDR process impossible. Whilst a parser might still be able to render the broken file, there is a significantly higher likelihood that a file which cannot undergo CDR may have been tampered with, to deliberately make it unsafe. Another explanation is that the authoring software is not respecting the industry specification for that file, and so may be saving in such a way to make other parsers unstable.  

Files which cannot be protected by other CDR, should be presented to other security filters such as next generation anti-virus, deep learning analyzers or sandboxes. None of these solutions can eradicate the risk completely but may be able to detect malicious signal in the file. These alternative solutions can be effective, but always have a false negative and false positive error rate. 

Having Glasswall CDR technology as the first in line of protection will ensure that less triage is necessary in relation to detection error rates that are associated with the other security filters.  

The size of the source file is certainly correlated with the processing time of a file. However, the nature of the file format and the data complexity of a specific file are more strongly correlated with the time it might take to analyze a file.

The CDR process builds a document object model of the underlying file to assist with the analysis. If this graph of that file has many thousands of nodes to inspect as the CDR Engine travels along that data representation, the computational effort is higher compared to a larger file which has a simpler structure. 

This is analogous to how compression rate varies between files, even though they are similar sizes before the compression step. Redundancy, encoding, and entropy are common factors which influence the speed and efficiency of the process. 

For AKS, the recommended approach is outlined in the Microsoft Documentation.

RabbitMQ Errors

When an error occurs in one of the Constellations services, the message is retried. Once the message hits a configured retry limit, it is sent to the dead-letter queue.

If a message has been sent to the dead-letter queue, the Scan Controller service will process it. For visibility the service logs a 'Critical' error.

Example:

json
{
   "Category":"Constellations.Scan.Controller.Business.Commands.FailScan",
   "EventId":"19",
   "LogLevel":"Critical",
   "Message":"A dead letter has been received when processing a message of type 'com.glasswall.cdr-enabler.process-file-item.v1' triggered at '05/24/2023 15:52:43 +00:00'. The Scan will now be failed"
}

Constellations uses the Glasswall Embedded Engine which does not currently support ARM based machines.

AKS

In AKS (Azure Kubernetes Service) Container Insights can be configured, which will include Log Analytics.

After you enable monitoring from Kubernetes clusters, metrics and Container logs are automatically collected for you through a containerized version of the Log Analytics agent for Linux.

Log Aggregation

A log aggregation tool such as DataDog can be deployed into the cluster.

We recommend installing the DataDog agent via Helm chart alongside the Constellations deployments.

When logs are generated by the system, they have associated data attached to them. This is achieved by using dotnets logging scopes.

Generally scopes are created when beginning a logical operation within the system, and end with the operation. Constellations will define data as key value pairs.

Messages sent from service to service include currently in-scope data within the RabbitMQ header 'trace-data'. 

Every scan constellations carries out is identified using a Globally Unique Identifier (GUID). This ID is defined within scopes to ensure that all logs will contain a reference to it.

When crossing domain boundaries from constellations to Glasswall Halo, the Scan ID needs to still be logged. The Synchronous API in Glasswall Halo provides a mechanism to do this.

Requests to the API include a header 'X-Trace-Data' which includes the scope data currently defined. At this point the Scan ID will be defined as scope data.

In order to trace a scan through the system, logs must be filtered by this scope value. Log statements are JSON formatted and contain a Scopes object which should be inspected.

Using Datadog as an example, this is achieved by using the following search:

@Scopes.scanId:374786e0-d740-4bc9-b13c-d47bb50ec72b

In the same way tracing a scan through the system, a file can be traced. A GUID is generated per file, and passed through the system as scope data. As with ScanId, FileId is also passed to Glasswall Halo. However Glasswall Halo also tracks internally a GUID for files it processes. For this reason SessionId is used instead to store Constellations idea of FileID.

An example using Datadog:

@Scopes.sessionId:374786e0-d740-4bc9-b13c-d47bb50ec72b
@Scopes.sessionId:374786e0-d740-4bc9-b13c-d47bb50ec72b

See the Upgrading Constellations.

Please see Third Party Libraries.