Fixed Resources

Overview

In this approach, one may set fixed parameters for service count, node count, and resources. This method is suitable if you are running a pre-defined solution and wish to avoid any wait time during the infrastructure's scaling process.

The file set used in testing comprises a variety of files, some of which are supported while others are not. As a result, performance may vary when processing different containers, which is influenced by the specific file set being handled.

The Setup

In this mode we have all our resources set:

• 2 Node Pools: 1 Constellations, 1 Glasswall Halo
• Node Spec: 8CPUs, 28GB RAM

Glasswall Halo

• 135 cdrplatform Nodes
• 375 engine pods
  • Fixed cpu request/limit of 2
  • Fixed memory limit of 5900Mi
• 75 api pods
  • Fixed cpu request/limit of 2
  • Fixed memory limit of 5900Mi
• 15 report-aggregator pods

Constellations

• 75 Nodes
• 20 scan-controller pods
• 20 page-scanner pods
• 20 scan-preprocessor pods
• 400 cdr-enabler pods
• 10 event-projection pods
• 10 event-collation pods

Storage

• File Share: The Azure file share incorporates rate limiting, which is determined by the size of the file share. To achieve optimal throughput, it is recommended to set the file share size to a minimum of 10,000GB. The Azure file share's throughput is calculated based on the allocated storage. It is important to avoid reducing this number, as it may lead to throttling at the Azure level. In case you encounter throttling issues, it is advisable to increase the allocated storage to alleviate the situation.

• Cosmos: Cosmos DB operates with allocated Request Units (RUs) for each individual instance. By enabling the auto-scale feature, Cosmos DB dynamically assigns RUs to containers, with a maximum quota of 24,000 RUs. Constellations heavily relies on substantial throughput, considering the significant number of read and write operations it performs. The required throughput is directly proportional to the volume of files being processed. If you encounter Cosmos DB throttling issues seen in the Azure portal, it is recommended to increase the number of allocated RUs to ensure smooth operations.

Modes

Express Mode: If the run is executed in Express Mode, this entails checking the file extensions. If a file's extension is not found in the supported list, the solution does not process the file. It is important to note that when Express Mode is disabled, the outcomes may differ, as all files are sent to the Glasswall Engine for processing, regardless of their extension.

Archive Mode: With a 'Basic' setting, archives are treated as individual files, and if any file within an archive fails, the entire archive is marked as a failure. It is important to be aware that when Archive Mode is switched to 'Advanced,' the outcomes may differ, as additional processing time is required to handle the archives effectively.

Guaranteed Mode: In Glasswall Halo, the API and Engine service may be executed in Guaranteed Mode to ensure that all pods assigned to a node were equipped with the necessary resources. Given that the Glasswall engine requires varying amounts of RAM and CPU, depending on the file's complexity, running in guaranteed mode guarantees sufficient memory for each file's processing. This approach minimizes the risk of a pod running out of memory during execution.

If you are encountering Out-of-Memory (OOM) exceptions when running Constellation, it is recommended to increase the resource limits. This adjustment will provide the necessary headroom for seamless processing and reduce the likelihood of memory-related issues.

File Set

• Input size: 501GB
• Input count: 46,694 blobs
• Total duration of run: 29mins
• Throughput = 17.28 GB/min or 0.29 GB/s