# Glasswall Documentation > Complete documentation for Glasswall CDR (Content Disarm and Reconstruction) technology and products. > Generated from https://docs.glasswall.com import ThemedImage from '@theme/ThemedImage'; import useBaseUrl from '@docusaurus/useBaseUrl'; ## Glasswall CDR: Understanding Content Disarm and Reconstruction CDR (Content Disarm and Reconstruction) uses a unique method to keep organizations and individuals safe from file-based threats. Instead of trying to detect threats like most cybersecurity solutions, Glasswall CDR follows a Zero Trust approach. This means that only files that have been checked and cleaned by Glasswall are considered safe, because all potential threats have been removed. We do not attempt to identify malicious code; instead, we remove its ability to affect the document. Every file that goes through the Glasswall Embedded Engine is treated as potentially harmful. The engine examines the file and rebuilds it according to the original specifications from its manufacturer, removing any dangers hidden in the file's structure. Fixing deeper problems within the file's structure is called remediation. We also remove content that could be harmful according to policy, such as hyperlinks in Office documents, which is referred to as sanitization. ## Why is CDR better than regular antivirus and sandboxing techniques? According to recent [reports](https://www.verizon.com/business/en-gb/resources/reports/dbir/), ransomware has been a significant factor in data breaches, with 24% involving ransomware attacks, as found in the Verizon 2023 Data Breach Investigations Report (DBIR). The Sophos report *The State of Ransomware 2023* further highlights the prevalence of ransomware incidents, with 66% of organizations experiencing them in 2023. Most ransomware samples are detected in Windows-based files, accounting for approximately 95% of all detected ransomware. Deploying CDR protection means organizations do not have to rely solely on next-generation antivirus software or threat intelligence databases, which can take an average of 18 days to detect new threats. While sandbox technology can help identify previously unknown malware, it often introduces delays that negatively impact productivity. Sandboxes rely on attackers executing malicious behavior while the file is being analyzed, which is not always guaranteed. Waiting days to identify new threats presents significant risks to enterprise IT environments. Next-generation antivirus software and sandboxes require an understanding of a threat in order to defend against it. Glasswall CDR rebuilds every file to its manufacturer's 'known-good' specification without requiring prior threat knowledge, eliminating the risk of malware being hidden within a file’s structure. ## The Glasswall CDR Process Glasswall’s CDR technology uses a patented four-step process to rebuild each file back to its manufacturer’s 'known-good' specification. Each phase analyzes the file independently, with results passed into the next stage. Once processing is complete, the Glasswall Embedded Engine produces a clean, usable file and a detailed report explaining what risks were identified and how they were addressed. ### Step 1 – Inspect The Glasswall Embedded Engine inspects the file to understand its structure and component sizes. It builds a tree-like structure, similar to a document object model (DOM) in HTML, automatically expanding compressed elements. If embedded file types are discovered, they are processed independently. For example, if a Microsoft Word document contains an embedded JPEG image, each file is analyzed separately. This phase generates a detailed structural analysis report. ### Step 2 – Rebuild During the rebuild phase, the engine iterates over the structures identified during inspection. Each structure is validated against the manufacturer’s published specifications, such as ISO 32000 for PDF files. If a structure does not conform, the engine attempts to repair it. If repair is not possible, the issue is recorded and the file may be deemed non-regenerable. Documents can contain thousands of individual structures. For example, the *SttbTtmbd* byte-level structure contains embedded font information within a Microsoft Word `.doc` file. The *SttbW6* structure defines the font type, location, and storage method. If structures do not meet specification requirements, default values may be inserted to allow validation. Structures that are hidden, unused, or not part of the known-good specification — such as caches, part-saves, or unreferenced data blocks — are not regenerated in the final output. ### Step 3 – Clean The cleaning phase applies configurable content management policies using [Policy Management](/docs/policy-management). Because all structures were identified during inspection, policies can be enforced deterministically. Each structure is handled in one of the following ways: #### Allow The structure is retained in the regenerated file and logged as allowed in the analysis report. #### Disallow The structure is logged as an issue. The document is marked non-conforming and is not regenerated. #### Sanitize The structure is surgically removed from the document and logged in the analysis report. This is achieved by excluding the structure from regeneration. The Glasswall Embedded Engine applies these policies to PDF, Microsoft Office, SVG, WebP, and GeoTIFF files. ### Step 4 – Deliver In the final phase, semantic checks ensure the document remains visually accurate and fully usable. Structural relationships are validated against the manufacturer’s specification, and any broken references introduced during remediation or sanitization are repaired. Once validation is complete, the engine writes the regenerated file by traversing the validated structure tree. The clean, usable file is delivered to the end user along with a detailed report describing all changes made. If a file cannot be safely processed, the reason is clearly reported. This entire process typically completes in under a second. ## Learn more - [Supported file types](/docs/supported-file-types) - [Policy management](/docs/policy-management) - [Glasswall CDR FAQs](/docs/glasswall-cdr-faqs) - [About Embedded Engine](/embedded-engine/about-embedded-engine) - [About Glasswall APIs](/rest-api/about-glasswall-apis) The following file types and extensions are currently supported by Glasswall. For more information about our supported file types, please [contact us](https://www.glasswall.com/contact). ## Supported file types | File type | Embedded Engine | Halo | Meteor | |:----------|:----------------|:-----|:--------| | Word binary (.doc, .dot) | ✔ | ✔ | ✔ | | Excel binary (.xls, .xlt) | ✔ | ✔ | ✔ | | PowerPoint binary (.ppt, .pps, .pot) | ✔ | ✔ | ✔ | | Word OOXML (.docx, .docm, .dotx, .dotm) | ✔ | ✔ | ✔ | | Excel OOXML (.xlsx, .xlsm, .xltx, .xltm) | ✔ | ✔ | ✔ | | PowerPoint OOXML (.pptx, .pptm, .ppsx, .ppam, .potm, .ppsm, .potx) | ✔ | ✔ | ✔ | | PDF (.pdf) | ✔ | ✔ | ✔ | | JPEG (.jpeg, .jpg, .jpe, .jif, .jfif, .jfi) | ✔ | ✔ | ✔ | | PNG (.png) | ✔ | ✔ | ✔ | | GIF (.gif) | ✔ | ✔ | ✔ | | BMP (.bmp) | ✔ | ✔ | ✔ | | SVG (.svg) | ✔ | ✔ | ✔ | | EMF (.emf) | ✔ | ✔ | ✔ | | WMF (.wmf) | ✔ | ✔ | ✔ | | TIFF (.tif, .tiff, .geotiff) | ✔ | ✔ | ✔ | | WebP (.webp) | ✔ | ✔ | ✔ | | WAV (.wav, .wave) | ✔ | ✔ | ✔ | | MP3 (.mp3) | ✔ | ✔ | ✔ | | MP4 family (.mp4, .m4a, .m4p, .m4b, .m4r, .m4v) | ✔ | ✔ | ✔ | | NITF/NSIF (.ntf, .nitf, .nsif) | – | ✔ | – | | Archive formats (.zip, .7z, .rar, .tar, .gz, .bz2, .xz, .iso) | – | ✔ | Pro only | | XML (.xml) | – | Schema validation | – | | Text (.txt) | ASCII / UTF-8 validation | ASCII / UTF-8 validation | – | --- ## Supported file sizes | Product | Maximum supported file size | |:--------|:----------------------------| | Embedded Engine | 2 GiB | | Halo | 1 GiB | | Meteor Essentials | 25 MB | | Meteor Pro | 1 GiB | --- ## Special cases and limitations ### Binary Office #### Limitations Office 95 and earlier versions are not supported. If these file types are processed, they will cause an Issue to be reported in the XML report (in Analysis Mode) and the file will not be regenerated in Protect Mode. Binary Office files containing Apple Mac or LibreOffice specific file features may be marked as non-conforming. ### Office Open XML #### Limitations Apple Mac or LibreOffice specific file features may be marked as non-conforming or the components may be removed from the file. ### Encrypted files #### Limitations Glasswall does not currently provide the ability to decrypt files, therefore **encrypted files are not supported** and are marked as non-conforming and blocked by default. ### NITF and NSIF #### Limitations Analysis reports for NITF (National Imagery Transmission Format) and NSIF (NATO Secondary Imagery Format) files are available in JSON format only. Requesting an XML report for these files is rejected with error code 4033, so set `format=JSON` when submitting them. Protecting NITF and NSIF files requires the relevant entitlement on your Glasswall Halo licence. Without it, a submitted file is rejected with a `403 Forbidden` response and the file is not processed. Protected NITF and NSIF files are listed in the Glasswall Halo portal but have no File Analysis view; remediation detail is in the JSON report returned through the API. --- ## Software executable formats Software executables can introduce significant security risk, so they should only be processed with strong safeguards in place. Additional security controls are recommended wherever possible. Glasswall inspects these formats at the **container level only**. This includes validation of headers, sections, and metadata against the expected standard. It does not perform full decompilation or source-level analysis, so executables may still contain harmful functionality even when the container structure is valid. The table below lists the software executable formats that are inspected and validated at the container level. | File type | Extensions | |:----------|:-----------| | Windows Portable Executable format | .acm, .ax, .cpl, .dll, .drv, .mui, .exe, .ocx, .scr, .sys, .efi, .tsp | | Mach Object format (Mach-O) | .o, .dylib | | Common Object File format | .o, .obj | | Executable and Linkable format (ELF) | .elf, .o, .so, .axf, .bin, .ko, .mod, .out | When using Glasswall's CDR technology, you have the option to set your own sanitization preferences by changing the *[policy settings](/docs/policy-management#policy-settings)* for each of the [supported file types](/docs/supported-file-types). This means you can shape your organization’s security policy in accordance with its risk tolerance. ## Choose the right policy settings for your organization Glasswall allows you to configure content management policies to match your organization's risk appetite, so you control the sanitisation settings of files and their contents during the CDR process. Initially, all default settings are set to *Sanitise* to give you the best balance between security and usability. Allowing certain types of content presents a risk if an attacker has placed malware within a file; however, your organisation may be more willing to accept and manage that risk for certain file types. Each type of [risky content](/docs/policy-management#risky-content-types) has been assigned a risk level (High risk, Medium risk, Low risk) to help you make an informed decision. ## Policy settings The following *policy settings* can be applied to risky content: - **Sanitise**: We'll analyse the file and remove this type of risky content, then rebuild the file. - Example: You can sanitise (remove) macros from Word files. - **Allow**: We'll analyse and rebuild the file, but we won't remove this type of risky active content. - Example: You can allow macros for Word files (this presents a risk if an attacker has placed malware within a file). - **Disallow**: We'll analyse the file, but if we find the risky content, we won't process the file at all. - Example: You can specify that Word files with macros aren't processed at all. ## Risky content types You're able to set preferences for the following active content depending on your file type: | Risky content type | Risk level | Description | | --- | --- | --- | | **Acroform** | High risk | An AcroForm is a PDF that contains form fields ("Acrobat form"). In addition to looking like a form, it may also contain active code (e.g., JavaScript) that could be malicious. AcroForms can also be used to hide objects inside other objects. | | **Connections** | High risk | The Excel connections feature controls connections to external data sources. This may present a risk if the external data source is compromised with malware or lead to SQL injection attacks, where the external data source is an SQL database. | | **Digital signatures** | Low risk | The source document may have been signed with a digital signature. While the signing may not represent a threat, if the ownership and trust of the certificate chain has been compromised, it could trick a user into viewing a document containing something malicious. | | **DDE** | High risk | Dynamic Data Exchange (DDE) allows data to be shared between applications in some older versions of Windows. DDE within a Microsoft document presents risk, as it may be used to execute malicious code on the recipient's computer. | | **Embedded files** | High risk | Embedded objects are files, items, or entities that have been added into a different program or object (for instance, a GIF in a Word document). Embedded objects may present risk if they provide a way for active code to be triggered or hide data within a document. | | **Embedded images** | Medium risk | Embedded images are pictures where data has been incorporated into the file. They may present risk if they provide a way for malicious content to be hidden inside the image. | | **External hyperlinks** | Medium risk | External hyperlinks redirect you to open something outside the file. A hyperlink may seem innocent, but it could route to a different destination than the link that’s displayed. Caution is advised when clicking links in documents. | | **Foreign objects** | Medium risk | Foreign objects may include items like HTML code, images, or other media types. They allow the author to include elements from other file formats or programming languages within the file that could be malicious. | | **GeoTIFF** | Medium risk | GeoTIFF is an extension to the TIFF file format and can include metadata that describes the coordinate system, projection, and other spatial properties of an image that the owner may not intend to disclose. | | **Internal hyperlinks** | Medium risk | Internal hyperlinks route you to a different place within a file. An internal hyperlink may seem innocent, but it could route you to a different destination than the link that’s displayed. Caution is advised when clicking links in documents. | | **JavaScript** | High risk | JavaScript is a form of active code that may be benign in nature, but is often used by bad actors to mount an attack against the user or receiving system in a business document. | | **Macros** | High risk | Macros are sequences of events (including keystrokes or clicks) that can be automated. Macros may seem benign in nature, but are often used by bad actors to mount an attack against the user or receiving system in a business document. | | **Metadata** | Medium risk | Metadata is data about other data. Metadata may reveal information the owner may not intend to disclose, such as what computer was used to create the document or the original author's name. | | **PDF actions** | High risk | PDF actions are built-in functionality in PDF documents. PDFs could include buttons that execute JavaScript or links to untrustworthy external resources. They could be used to launch a phishing attack, download malware, or steal sensitive information. | | **Review comments** | Medium risk | Review comments are comments that have been added to Microsoft files. They may reveal information the owner may not intend to disclose, such as the original author's name. | | **Scripts** | High risk | A script is a form of active code that may be benign in nature, but is often used by bad actors to mount an attack against the user or receiving system in a business document. | | **Tracked changes** | Medium risk | Tracked changes convey the history of edits made within a file. They may reveal information the owner didn't intend to disclose. | For more information, please [contact us](https://www.glasswall.com/support). ## Glasswall FAQs What is CDR? CDR (Content Disarm and Reconstruction) is a cybersecurity technology that instantly removes potential file-based threats from incoming files and documents before they have a chance to enter a corporate network. CDR, also referred to as *content sanitization*, breaks files down into their discrete components and removes anything that does not comply with the manufacturer’s *known-good* specification. This includes potentially malicious content such as macros, hyperlinks, and embedded objects. The file is then rebuilt into a safe, trusted version. - - - What can Glasswall CDR offer? Glasswall CDR processes and cleans supported file types by: * Automatically removing unrecognised objects hidden within file structures if they are not defined in the official specification * Automatically correcting components that deviate from the manufacturer’s specification * Enabling sanitisation of defined content using configurable content management policies * Accurately determining file type regardless of file extension presence or manipulation *The Glasswall Embedded Engine uses multiple techniques to provide a reliable determination of file type.* - - - What is the Glasswall security promise? File-based security threats are growing rapidly, with approximately 1 in every 100,000 files containing potential malware. Most of these threats are unknown to traditional antivirus solutions. Glasswall CDR protects against threats *before* they enter file servers via email, web traffic, and cloud services. Our file sanitisation technology ensures files are safe at the point of delivery, across endpoints where file sharing is most common. - - - What threat vectors does Glasswall CDR target? A major advantage of CDR over conventional threat detection is that it does not rely on identifying *known bad* threats. By focusing on *known good* specifications, Glasswall safeguards organisations without needing to identify attack vectors. Glasswall CDR protects against: * **Data attack risks** * Risky content attacks (macros, JavaScript, etc.) * Binary exploits (stack overflows, heap sprays, etc.) * **Data disclosure risks** * Content removal * Metadata removal - - - How does Glasswall CDR conform to ISG? The NSA’s Inspection and Sanitization Guidance (ISG) defines requirements for file inspection and sanitization software. The Glasswall Embedded Engine achieves a high level of compliance by addressing ISG concerns through sanitization and remediation during CDR processing. Further work is underway to achieve full compliance and enable mapping between ISG sections and detected file content. - - - Are files modified during the CDR process? While file content remains unchanged, the underlying file structure is often modified during processing. This can result in a different checksum (hash) compared to the original file. Glasswall records the SHA-256 hash of both the original and regenerated file, ensuring a verifiable chain of custody for compliance and provenance use cases. - - - What assurance is there around file integrity? During final processing, semantic checks ensure the visual integrity and usability of the reconstructed document. Each release of the Embedded Engine undergoes extensive in-house testing to verify file integrity. - - - What file types are supported? [View the list of supported file types](/docs/supported-file-types) - - - What happens if vendors update file specifications? Glasswall continuously updates the Embedded Engine to support the latest file specifications. If specifications change and are not supported, some files may fail reconstruction. The Glasswall team proactively schedules updates, and customers are encouraged to remain on the latest product versions to benefit from ongoing improvements. Glasswall products may include open-source or third-party software components. For more details about the libraries used in each product, please refer to the following pages: - [Glasswall Embedded Engine](/embedded-engine/embedded-engine-third-party-libraries) --- These references outline the open-source and third-party components incorporated in each respective product, including their licenses and attributions. For more information, please contact [Glasswall Support](https://www.glasswall.com/support). ## Introduction These Customer Support Service Standards set out how Glasswall will provide the Support Services to the Customer and should be read in conjunction with the applicable General Terms. The Service Standards are not solution specific and apply to all solutions for which the Customer has purchased Support Services. These Service Standards may be modified or updated from time to time. It is the Customer’s responsibility to ensure that they are familiar with the most up to date version of these Service Standards, as that is what will apply. --- ## Accessing support services To obtain Support Services, the Customer must raise a service request ticket (Ticket) with Glasswall’s customer support team (Support Team). * Tickets can be raised 24 hours a day, 7 days a week. * The number of Tickets that can be raised by the Customer is not limited. The Customer must raise a Ticket using one of the following channels: ### By email [support@glasswall.com](mailto:support@glasswall.com) ### By telephone +44 (0)203 868 6305 (UK) +1 844 527 7925 (USA) The standard support language is English. When raising a Ticket, the Customer must follow the process and provide the information set out in Customer Responsibilities. Glasswall recommends raising a Ticket by email for tracking purposes. --- ## Customer responsibilities Prior to raising a Ticket, the Customer is encouraged to read the support documentation for their product, as many support issues can be resolved by following this information: [https://docs.glasswall.com/](https://docs.glasswall.com/) Each Ticket raised by the Customer should include at least the following information: * Subject * Brief description * Priority * Impact to the Customer’s business * Triage already attempted * Steps to reproduce * Other relevant information * Glasswall Software Version * Operating System * Configuration and environment details * Tool or example code * Test file or data * Designated Customer contact(s) and availability including time zone To facilitate Ticket diagnosis and resolution, Glasswall Support Team agents may be required to view a Customer’s remote server or desktop environment via a web conferencing solution such as Teams or Zoom. The Customer will make every effort to identify a suitable method of communication that enables collaboration and provide all information requested by the Support Team for Ticket resolution. Glasswall shall have no responsibility or liability for any delay or failure in responding to or resolving Tickets where the Customer fails or delays in providing required information, remote access, or communication. --- ## Service level agreement / Ticket priority When a Ticket is raised: 1. A record of the Ticket is created and logged in the case management system and associated with the account in CRM. 2. The Support Team determines the relevant severity level of the Ticket in accordance with the definitions below. All response times are business hours and business days local to the UK and US Eastern Time. Business hours are defined as **9:00am to 5:00pm**. --- ## Severity levels and first response times Glasswall will use reasonable efforts to provide an initial response in accordance with the First Response Time listed below. The applicable response time depends on whether the Customer has purchased Standard Support Services or Premium Support Services. ### Severity and response table | Severity Level | Definition | Standard Support First Response | Premium Support First Response | | ------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------- | ------------------------------ | | **Severity Level 1 – Critical** | A critical functionality severely impacts use of the Glasswall Solution, halts business operations, and no procedural workaround exists. | Within 4 business hours | Within 1 business hour | | **Severity Level 2 – High** | Major functionality is impacted or significant performance degradation occurs with high impact to business operations and no reasonable workaround exists. | Within 1 business day | Within 2 business hours | | **Severity Level 3 – Medium** | Partial, non critical loss of use with medium to low business impact. Business continues and a short term workaround is available but not scalable. | Within 2 business days | Within 4 business hours | | **Severity Level 4 – Low** | Low or no impact to use of the Glasswall Solution. Does not impede business operations. Includes user questions and enhancement requests. | Within 3 business days | Within 1 business day | --- ## Out of scope services Glasswall provides Support Services only where the Customer has a valid Agreement that expressly includes Support Services and only for the Solutions specified in the applicable Order Form. All Support Services are provided remotely unless otherwise agreed in writing and subject to additional fees. Support Services do not include: * Step by step installation of the Solution or New Releases * On site services unless expressly agreed in writing and subject to additional fees * Modification of the Solution’s code * Security policy configuration * Audits * Security design The Support Team has no obligation to provide Support Services where Tickets arise from: * Customer misuse, misapplication, or use outside Documentation or instructions * Causes beyond Glasswall’s control * Releases past their notified end of support date unless affecting supported New Releases * Damage not caused by Glasswall including power failure, communications network failure, negligence, accident, or ultrahazardous use * Failure to maintain required environmental conditions * Use with equipment or software not supported by Glasswall * Unauthorized repair, maintenance, or modification * Software licenses not purchased from Glasswall or authorized partners ## Introduction These Customer Support Premium Standards indicate the additional services provided by our Premium Support package. These services are supplementary to the **Customer Support Service Standards**, which covers the standard support offering and all support procedures. --- ## Premium support benefits ### Improved Response Time Premium support provides a faster response time in comparison to the standard support package. Response time commitments are detailed in our [Customer Support Service Standards](/docs/support-service-standards) guide. --- ### Dedicated support Customers enrolled in the Premium Support package are assigned a dedicated support agent. This representative will: * Become familiar with the specific needs of your business * Act as the primary contact for support queries *In the case of annual leave or staff turnover, the dedicated support agent may change.* --- ### Additional expertise and direct access to support teams The Premium Support package provides customers with access to in house communication channels to raise issue queries. Through a private Slack channel, customers will have direct access to: * Their dedicated support agent * Their account executive * Members of the Glasswall engineering team * Members of the technical leadership team **Important:** * This Slack channel may be used for issue discussion and queries. * It cannot be used to raise formal support cases. * Formal support cases must still be raised through the standard support procedure. * This Slack channel is not covered by service level agreements. --- ### Alternative escalation pathways Premium Support offers alternative escalation pathways for support cases that have already been raised through the standard procedure. Initial issue raising must follow the standard process. After that, escalation can occur through: * A dedicated Slack channel * A dedicated Sales Account Executive * The VP of Operations * A dedicated Customer Support Agent These pathways increase communication and feedback options when discussing support cases. --- ### Software updates and regular meetings Premium Support customers may request recurring meetings with a Glasswall Product Manager. These meetings enable: * Greater visibility into future product changes * Opportunity to provide structured user feedback * Improved understanding of customer environments and challenges * Stronger alignment between product direction and customer needs Customers will automatically be informed when a new software release becomes available. To request delivery of a release, email: [support@glasswall.com](mailto:support@glasswall.com) import ThemedImage from '@theme/ThemedImage'; import useBaseUrl from '@docusaurus/useBaseUrl'; Glasswall Halo is built upon powerful cloud-native architecture using scalable Kubernetes. Our CDR technology removes potential threats from unknown files in less than a second, without compromising the file’s contents. Traditional antivirus solutions don't provide complete protection because newer threats still evade detection. Sandboxes are often better at addressing malware compared to antivirus solutions, but this often affects user experience by interrupting productivity. Glasswall can enable organizations to deploy our solution into their own environments. The Glasswall API endpoints are provided as standard, to ensure that integration is as friction-less as possible. ## What is CDR? CDR stands for Content Disarm & Reconstruction. Glasswall’s CDR technology works by: * Inspecting a file’s digital DNA * Cleaning risky content according to a set policy * Rebuilding to ‘known good’ standard * Delivering a safe & visually identical file Compared to traditional sandboxing and antivirus solutions which fail to detect zero-day threats, our CDR technology provides an on average 18-day protection advantage. ## Key features * Kubernetes-based architecture deployed via managed services such as AKS, EKS, and OKE. * Terraform deployment scripts and Helm charts to support managed Kubernetes services. * Content management policies controlled dynamically via the API. * Intuitive [REST API](/rest-api/about-glasswall-apis) endpoints which help developers devise the most effective integration approach to meet their needs. * Cloud APIs compliant with OpenAPI v3.x specifications, allowing development teams to rapidly create client software integrations using Codegen tools. * Over 85+ supported file formats and counting. ## Glasswall Halo API's ### [Synchronous API](/rest-api/about-synchronous-rest-api) Glasswall's synchronous REST API endpoints enable security teams to present file analysis and rebuild requests to Glasswall Halo – securing files in under a second. In addition, the REST API facilitates policy management actions – enabling security teams to match the processing of file components with their organizations risk appetite. ### [Asynchronous API](/rest-api/about-asynchronous-api) While the Synchronous API effectively meets the need for handling substantial volumes of data, certain client architecture patterns will require the incorporation of an asynchronous mode. With Glasswall Halo's Asynchronous API, requests can be handled via a two-stage process. Initially, a UUID is generated for a request, which can subsequently be used to retrieve the processing outcome. This enables the client to continue at its own convenience, eliminating the requirement to wait for file processing. ### [Prompt injection detection API](/rest-api/prompt-injection-detection-api) The Prompt injection detection API detects prompt injection in a text string before it reaches a large language model. Security teams submit the text in one request and Glasswall Halo returns a verdict in the response. Glasswall licenses this capability separately. ### [Policy management API](/rest-api/about-policy-management-api) The Policy management API gives security teams the capability to configure content management policies to match their organizations risk appetite, allowing them to govern the treatment of files and their contents during the CDR process. Users are able to create, update, delete or reset content management policies for Glasswall Halo. ### [License management API](/rest-api/about-license-management-api) This is an API used to manage active licenses and can be accessed via the main entry point into the cluster. The license management service focuses solely on license storage and does not impact the application of licenses within the cluster. ### [ICAP profile management API](/rest-api/icap-profile-management-api) The [Glasswall ICAP](/halo/about-glasswall-icap) integration provides users with a robust tool for customizing file processing by an ICAP server and managing CDR. The ICAP profile management API allows for the creation and modification of these profiles, streamlining the management and optimization of file processing workflows. Additionally, the API supports the simultaneous activation of multiple ICAP Profiles, offering flexible and precise control over file security measures. ### [XML validation API](https://api.docs.glasswall.com/xml-validation-v1#XML-validation-API) The XML validation API is a tool that streamlines XML validation, ensuring your XML markup is secure and well-formed according to the specified schema. ### [Storage monitoring API](https://api.docs.glasswall.com/storage-monitoring-V1#storage-monitoring-API) Storage monitoring API (v1.0) is a secure REST API for managing automated sanitization of files stored in SharePoint and OneDrive. It allows developers to list relevant storage resources, configure monitoring rules through creation, updates, and deletion of monitors, and retrieve active monitors, all over authenticated HTTP connections with structured JSON response schemas. ## Kubernetes Glasswall Halo is built using scalable Kubernetes (k8s). This is a technology that automates deployment, scaling, and management of containerized applications. Containers within an application are grouped into logical units for easy management, and Kubernetes provides a way to schedule and deploy those containers. It also scales them to your desired state and manages their life cycle. Learn more about [Kubernetes](https://kubernetes.io). When submitting a request to Glasswall Halo, what steps does a file go through? 1. The file is sent to the REST API. 2. The file is stored on a persistent volume within the cluster. 3. A message is sent to the Engine service to process the file. 4. The Engine service receives the message to process a file. 5. That file is read from the persistent volume and is then protected by the Glasswall Engine. 6. The protected file and analysis report are then stored in the persistent volume. 7. A message is sent to the report extractor so the Engine report can be generated. 8. A message is then sent back to the API from the Engine service. 9. The API receives the message from the Engine and reads either the protected file, report or both from the persistent volume and generates the correct response to the client. 10. A message is then sent to the report extractor from the API. 11. The report extractor receives this message and generates a report from the analysis file and the API. 12. Metric data is then generated. 13. Finally the original file, protected file and analysis report are deleted from the persistent volume. What types of files can Glasswall Halo process? Glasswall Halo supports a wide range of file types including PDFs, Office documents (Word, Excel, PowerPoint), images, and more. It ensures that all processed files are safe and retain their original functionality. How are files stored? Original files, protected files and analysis reports are temporarily stored on a persistent volume during processing. The persistent volume is backed by the appropriate storage provider for the deployment platform: - Azure - Azure File Share - AWS - Amazon Elastic File System (EFS) - GCP - Google Cloud Filestore - Oracle - OCI File Storage Service - On-premise (RKE2/OVA) - local disk Transaction reports, file details and processing summaries displayed in the Portal are stored in MongoDB. How is the system configured and deployed? Glasswall Halo can be deployed in a Kubernetes environment, leveraging Helm charts for easy setup and configuration. Detailed deployment instructions are provided in the documentation. You may also deploy via an OVA. How does Glasswall Halo handle file types that it cannot process? An error report is returned to the user. How long does the file persist in shared storage? Original files, protected files and analysis reports are removed from the persistent volume according to the [file retention policy](/halo/halo-file-retention). The retention behavior varies by submission method. What are the maximum nested levels of archives? A maximum of 5 levels of nested archives are supported by Glasswall Halo. What are the archive types? The following archive types are supported in Glasswall Halo: Zip, Tar, 7Zip and Rar. Is Zipx (.zipx) supported? No. The supported archive types are **Zip, Tar, 7Zip and Rar**. The extended `.zipx` format is not a supported archive type, so its contents are not unpacked and processed as an archive. Files in an unsupported archive format are handled the same way as any other file type Halo cannot process—see *How does Glasswall Halo handle file types that it cannot process?* above. Can I retrieve the original file after processing? No. Glasswall Halo returns the **rebuilt (protected) file** and the **analysis report**. The original submitted file is held only temporarily on the persistent volume during processing and is then removed according to the [file retention policy](/halo/halo-file-retention)—it is not retained for later retrieval through the API or portal. If you need to keep a copy of an original (for example, a suspicious file for investigation), retain it in your own system **before** submitting it to Halo. Halo is a Content Disarm and Reconstruction (CDR) service, not an original-file archive. How do I process large numbers of files (for example, batch uploads)? Files are submitted to the REST API per request. To process large volumes: - Use the **Asynchronous API**, which is designed for higher-throughput, non-blocking submission, and poll for results. The Synchronous API is better suited to lower-volume, request/response use. - Submit requests **concurrently** from your client rather than strictly one-at-a-time, and apply your own retry/back-off. If the system is saturated you may receive `429` responses (see the 429 entry below)—back off and retry rather than dropping work. - Be aware of your **daily license entitlement**. High-volume jobs can reach the per-day file or data limit, after which requests return `401` until the daily reset—see [Retrieving license usage data](/halo/retrieving-license-usage-data). Refer to the API documentation for the available endpoints and request formats. What are the Content Management Flags? - **Allow** – 0 - **Sanitise** – 1 - **Disallow** – 2 What does the code mean in the error response? Within the error response, Glasswall Halo returns a "code" which indicates a particular scenario. Error codes start with the HTTP response code and include a number to indicate the specific error. For more information please refer to our API documentation. Can I amend the replica count of the rebuild pods to increase performance? Yes, the current configuration includes contingency for workloads that are more challenging than the average file. Whilst it may be possible to increase the replicas count and observe improved performance, the consequence might be increased errors if the concurrent load depleted the available memory. Can I use a larger node size to improve performance? This is possible and would therefore allow the replicas count per node to be amended without necessarily experiencing any scarcity of compute resources. However, assigning more workloads to a single node increases the blast radius should a node fail. Increasing the node count may achieve the same objective with less risk. Why are you allocating memory larger than 1 GB if that’s the maximum file size? The CDR process creates an intermediate representation of the original file, which may be much larger than the original file. Reserve values ensure that multiple large files can be processed. How does license management work? Glasswall provides customers with entitlement to process a number of files or a volume of data each day. Currently Glasswall does not impose a technical limitation on over-consumption, but over-consumption is contractually prohibited. A license-management layer will be introduced to Glasswall Halo, enforcing limits based on the entitlement. Does Glasswall limit the number of engine calls? Currently there is no hard limit on Engine calls beyond those implied by the base Helm Chart configuration. Future versions may throttle throughput to align hourly and daily licensed capacity. What is the maximum file size that you support? 1 GB. Some files may require much more memory to CDR due to their structure or model representation requirements. Why does Glasswall use some 3rd party services? Glasswall is the world's premier CDR solution provider and uses select third-party components. Rigorous static analysis, SCA, and IaC scanning ensure these components do not introduce vulnerabilities. Glasswall is transparent about dependencies and enforces strict security standards. Does Glasswall provide a software bill of materials (SBOM) for the deployment? Glasswall generates SBOMs for the services that comprise the Halo deployment. These can be made available on request. Do you only support v3 of Helm? Yes, that is correct. What information is sent back to Glasswall from my deployment? Glasswall Halo is designed to run in a secure environment — **no data is sent back to Glasswall**. Glasswall may request summary logs to verify license conformance, but **no customer data ever leaves the environment** without explicit administrator action. Why do some of your services use Alpine as the base OS? Most services will move to a hardened Alpine base image following CIS guidelines. Alpine provides a minimal, security-hardened Linux distribution with a reduced attack surface. Why am I seeing a 429 status code when Halo hasn't been used for a while? What should I do? When Halo is overloaded, the request queue may build up and return 429 errors. The system will eventually consume the backlog automatically. If this takes too long — and you are comfortable losing queued messages — you can purge the queue: ```shell kubectl exec -ncdrplatform -it RabbitMQ-server-0 -c RabbitMQ -- RabbitMQctl purge_queue engine-request-queue ``` How is the solution monitored and maintained? Kubernetes-integrated monitoring tools provide insights into performance, resource usage, and health. Maintenance and updates are handled through Kubernetes to maintain security and stability. What are the security measures in place to protect the data processed by Glasswall Halo? Glasswall Halo employs encryption, secure communication protocols, strict access controls, and the CDR process itself, ensuring no malicious content passes through. What support and resources are available? Glasswall provides documentation, support services, and training resources for implementing and managing the solution. Support includes online resources, customer support, and community forums. What is the expected latency for file processing? Latency depends on file size, type, and complexity. Glasswall Halo is optimized to process files quickly to minimise delay. How is performance and scalability managed? Kubernetes allows Glasswall Halo to scale automatically based on load. Using **KEDA**, Halo dynamically adjusts CDR service instances based on file traffic. [Learn more about Keda](https://keda.sh/) # CDR file formats Glasswall Halo supports all file types that are supported under the [Glasswall Embedded Engine](/docs/supported-file-types), along with NITF and NSIF. All file types supported: File Type| Extension(s) ---|--- Word Binary File| .doc, .dot Excel Binary File| .xls, .xlt PowerPoint Binary File| .ppt, .pps, .pot PDF Document| .pdf Office Open XML Document| .docx, .docm, .dotx, .dotm Office Open XML Workbook| .xlsx, .xltx, .xltm, .xlsm Office Open XML Presentation| .pptx, .pptm, .ppsx, .ppam, .potm, .ppsm, .potx TIFF Image| .tif, .tiff, .geotiff SVG Image| .svg Enhanced Meta File| .emf Windows Metafile Format| .wmf JPEG Image| .jpeg, .jpg, .jpe, .jif, .jfif, .jfi PNG Image| .png GIF Image| .gif BMP Image| .bmp, .dib WebP Image| .webp NITF 2.1| .ntf, .nitf NSIF 1.00, 1.01| .nsif Waveform Audio File| .wav, .wave MPEG-2 Media File| .mpg, .mpeg MP3 Audio File| .mp1, .mp2, .mp3 MP4 Media File| .mp4, .m4a, .m4p, .m4b, .m4r, .m4v Windows Portable Executable Format| .acm, .ax, .cpl, .dll, .drv, .mui, .exe, .ocx, .scr, .sys, .efi, .tsp Mach Object Format (Mach-O)| .o, .dylib Common Object File Format| .o, .obj Executable & Linked Format| .elf, .o, .so, .axf, .bin, .ko, .mod, .out ## NITF and NSIF versions Halo protects the NITF and NSIF versions below. Glasswall Genesis processes the imagery and the content embedded in it. - **NITF 2.1** — file header `NITF02.10`, standard MIL-STD-2500C - **NSIF 1.01** — file header `NSIF01.01`, standard STANAG 4545 (current, harmonized with NITF 2.1) - **NSIF 1.00** — file header `NSIF01.00`, standard STANAG 4545 (legacy); ISG validation flags these files as non-conformant **NITF 2.0 (`NITF02.00`) is not supported.** Halo does not identify these files as NITF, so Genesis does not protect them. ## Non-CDR formats Halo also supports archive files as [detailed in the archive support documentation](/halo/glasswall-halo-archive-support). Archive Type| Extension(s) ---|--- Zip (and compression type Bzip2)| .zip, .bz2 Tar| .tar Gzip| .gz 7Zip| .7z, .7zip Rar| .rar The [validation of XML (.xml) files](/halo/glasswall-xml-schema-validation-overview) is also supported. ## Overview Glasswall Halo offers advanced protection for archive files. This powerful feature utilises the state-of-the-art Glasswall Engine to safeguard every single file within an archive. It not only shields the files but also intelligently recompresses the entire archive into a supported format. With this enhanced functionality, your archive files are effectively secured and optimized for seamless use. ## Supported input types ### Unprotected - Zip - (supports the compressions type of bzip2) - 7-Zip - Gzip - Rar - Tar ### Protected - Zip ## How does it work Upon receiving an archive, Halo undertakes a comprehensive processing workflow. Firstly, it decompresses the archive, delving up to five nested levels of archives. At each level, the platform individually processes every non-archive file encountered. Each file is subjected to the powerful Glasswall engine for processing. Once all files within the archive have been processed, the platform recompiles the archive, maintaining the original structure. The expected outcomes of this process are as follows: 1. Protection and sanitisation: the Glasswall Engine ensures that each file is thoroughly protected and sanitised, eliminating potential threats and vulnerabilities. 1. Compliance with policies: Glasswall Halo adheres to the specified policies, ensuring that the resulting archive aligns with the designated security and content management configurations. 1. Format preservation: the archive is recompiled in a manner that preserves the original structure, maintaining the hierarchical organization and integrity of the files. By following this robust processing workflow, Halo ensures the highest level of file protection, policy compliance, and integrity preservation for archives. ## Expected outcomes ### File types While Glasswall Halo is highly capable, there are instances where certain archive types cannot be supported for recompression due to licensing restrictions. In such cases, the platform applies an alternative approach: it recompresses the unsupported archive types into the universally compatible Zip format. Importantly, during this process, the platform ensures that all file types and folder structures within the resulting Zip archive remain completely unchanged. By implementing this solution, Glasswall Halo guarantees seamless compatibility and preserves the integrity of the original files and folder organization, even when faced with unsupported archive types. This ensures that data remains accessible and unaltered, despite any limitations posed by licensing constraints. | Input File | Input File Example | Output File | Output File Example |------------|-------------------|-------------|-------------------- | Zip | file.zip | Zip | file.zip | Tar | file.tar | Tar | file.tar | GZip | file.gzip | GZip | file.gzip | 7Zip | file.7zip | Zip | file.7zip.zip | Rar | file.rar | Zip | file.rar.zip ### File contents In most scenarios, when processing an archive, Halo replaces a file within the archive with a clean version before returning it via the API. However, there are situations where the platform may encounter difficulties processing a specific file within the archive. In such cases, if some files are successfully processed while others are not, the unprocessable file will be substituted with a `.txt` file. The contents of this replacement file will offer an explanation as to why the file cannot be processed. By employing this approach, Halo ensures that the majority of files within an archive are effectively processed and returned in a clean state. For any files that encounter processing issues, clear and informative explanations are provided, allowing users to understand the reasons behind the un-processable files. The following scenarios are to be expected: #### Rebuild - Archive is being rebuilt and entry is allowed - uses the original file - Archive is being rebuilt and entry is disallowed - replace with a text file (same name) saying "file disallowed" - Archive is being rebuilt and entry is unsupported file type - replace with a text file (same name) saying "unsupported file type" - Archive is being rebuilt and Engine fails while rebuilding entry - replace with a text file (same name) saying "unable to rebuild file" - Archive is being rebuilt and entry is successfully rebuilt - use rebuilt file #### Analysis - Archive is being analysed and entry is allowed - replace with a text file (same name) saying "file allowed by policy, no analyse needed" - Archive is being analysed and entry is disallowed - replace with a text file (same name) saying "file disallowed" - Archive is being analysed and entry is unsupported file type - replace with a text file (same name) saying "unsupported file type" - Archive is being analysed and Engine fails while analysing entry - replace with a text file (same name) saying "unable to analyse file" - Archive is being analysed and entry is successfully rebuilt - use analysis of file In all cases of archives when using the V3 endpoint a `manifest.json` is also outputted which details the processing result of all the files inside the archive and can be used to help understand the archive without having to unpack it. ## Compression types When processing files through Halo, it treats `Bzip` and `Gzip` compression types as individual files. As a result, no `manifest.json` is generated during the processing. However, the processing of these compressed files still follows the rules of archive processing. If a compressed file contains an archive within it, a `manifest.json` is created and outputted at the highest level of the archive structure. The file type response header for a compressed file is set to `compressed_file`, indicating its compressed nature. When an analysis report is generated for a compressed file, it undergoes a renaming process to ensure proper decompression of the inner file. For instance, if the original file is named `test.pdf.bz2`, the report file is renamed as `test.pdf.report.xml.bz2`. This naming convention ensures a clear association between the report and the corresponding compressed file, facilitating correct decompression and subsequent analysis. By implementing these measures, Halo maintains consistent handling of compressed files, produces accurate reports, and preserves the integrity of the processed files throughout the analysis workflow. ## Policy config Glasswall API endpoints provide wide support for Content Management Flags, enabling granular control over individual files within an archive. This configuration empowers you to define specific actions for each file type within the archive. The default value for each file type in this configuration is set to `allow - 0`. However, you also have the flexibility to choose alternative supported actions, such as `sanitise - 1` or `disallow - 2`. Using the Content Management Flags, you can precisely control the handling of different file types within archives, ensuring that each file receives the appropriate level of management and security based on your desired configuration. This feature enhances the overall control and customization options available to effectively manage files within archives through Glasswall Halo API. The following file types are supported under archive configuration: ```json "ArchiveConfig": { "bmp": 1, "doc": 1, "docx": 1, "emf": 1, "gif": 1, "jpg": 1, "wav": 1, "elf": 1, "pe": 1, "mp4": 1, "mpg": 1, "pdf": 1, "png": 1, "ppt": 1, "pptx": 1, "tif": 1, "wmf": 1, "xls": 1, "xlsx": 1, "mp3": 1, "rtf": 1, "coff": 1, "macho": 1, "unknown": 1 } ``` `extendedFormatSettings` controls embedded content within NITF (National Imagery Transmission Format) and NSIF (NATO Secondary Imagery Format) files in archives, not `ArchiveConfig`. See [Define content management policies](/rest-api/define-content-management-policies). Two behaviours differ from a standalone NITF or NSIF file: * Requesting an XML report does not reject the submission. The archive is accepted, its NITF and NSIF members are errored individually, and every other member is processed as normal. * A member that cannot be protected is replaced by a `.txt` placeholder, its report still ships alongside it, and the archive is returned with a `206` response. ### How is the policy applied within archives Archives processed through Glasswall Halo adhere to the provided policy in a hierarchical manner. The fundamental principle guiding the application of policies is to select the most stringent outcome based on the given configuration whilst trying to fulfil the initial request. This means that in some cases the `ContentManagementFlags` specified in the request take precedence. Additionally, when the `return-executable-file` parameter is set to false, it serves as a blanket restriction for all executables within archives, irrespective of the archive configuration supplied. Furthermore, the `ArchiveConfig` section is also considered, taking into account the values within it. If the values within this section represent the strictest outcome, they will be utilised to determine the resulting outcome of the archives. By following this hierarchical approach and incorporating the provided parameters, Halo ensures that policies are enforced effectively, providing maximum security and control over the archive processing. ### Outcome matrix This example is based on a Word document which contains an internal hyperlink: CMF = `WordContentManagement:InternalHyperlinks` value AC = `ArchiveConfig:doc` value | | **CMF = 0** | **CMF = 1** | **CMF = 2** |--|--|--|-- | **AC = 0** | File Allowed | File Allowed | File Allowed | **AC = 1** | File is Sanitised | File is Sanitised | File is Disallowed | **AC = 2** | File is Disallowed | File is Disallowed | File is Disallowed This example is for a pe file with `return-executable-file` set: REF = `return-executable-file` value | | **REF = true** | **REF = false** |--|--|-- | **AC = 0** | File Allowed | File is Disallowed | **AC = 1** | File is Sanitised | File is Disallowed | **AC = 2** | File is Disallowed | File is Disallowed ### Limits To ensure the system operates smoothly and avoids overloading, specific limits have been set for processing archives. These limitations are as follows: - A max file count of 500. This means any archives that contain more than 500 files when unpacked will not process. - A max file size of 500 MB. Any archives that contain more than 500 MB of files inside will fail to process. - A max archive count of 50. This tracks any nested archives (archives inside other archives). If more than 50 nested archives are found the overall file will fail. - A max nesting count of 5. This refers to how many layers of nesting can occur in a file. This limit does not cause the entire file to fail to process but any files wrapped in more than 5 layers of archives will not be processed and instead a placeholder text file will be placed at the 5th level. This nesting only refers to archives inside archives, folders do not count. These limits are all configurable via service configuration. Halo offers a configurable file retention policy that allows users to specify the duration after which original files, protected files and analysis reports will be automatically removed from internal storage. The retention behavior varies depending on how files are submitted to Halo. ## Cleanup service The Halo cleanup service is responsible for the automatic deletion of files from internal storage. It accomplishes this through two methods. Firstly, the services will notify cleanup to remove files once they are no longer required. Secondly, a scheduled Cron Job is executed periodically to remove any extraneous files from storage. The Cron Job can be configured to modify the quantity of files that are cleaned up, the maximum age for files stored, and the schedule in which it is executed. For more information on configuration of the Cleanup Service please refer to [Configuration Changes](glasswall-halo-configuration-changes). ## Retention by submission method ### Sync API Files submitted via the Sync API are removed from internal storage immediately after the response is returned to the caller. The cleanup service cron job acts as a safety net to remove any orphaned files that were not cleaned up during processing. ### Async API Due to the asynchronous nature of the API, files are retained for a configurable period to allow users time to retrieve results. This retention window can be configured through the `QUEUE__MessageDelayInMs` setting of the Async API (default: 1 hour). After this period, the cleanup service removes the files. The cron job `maxAge` setting provides an additional safety net for any files that were not cleaned up by the message-based mechanism. Ensure `maxAge` is greater than or equal to `QUEUE__MessageDelayInMs`. For more information on configuration of the Async API please see [Async API Configuration](glasswall-halo-configuration-changes). ### ICAP (Internet Content Adaptation Protocol) The ICAP server forwards files to the Sync API for protection. The Sync API retention behavior described above applies to these files. The ICAP server maintains its own local cache of protected files, which is governed by a cache size limit rather than a time-based retention policy. ### Storage Monitor Storage Monitor protects files in place within the connected cloud storage (e.g. OneDrive, SharePoint, Outlook). Files are removed from Halo's internal storage immediately after Storage Monitor replaces the original file in cloud storage. The Halo file retention policy does not apply to files in the customer's cloud storage - retention of those files is governed by the customer's own cloud storage policies. ### Portal The Portal's "Clean a File" feature submits files to the Sync API for protection. The Sync API retention behavior described above applies - files are removed from internal storage immediately after the response is returned. ## Industry-leading performance ### Overview - Test suite of 20 business files ranging from 0.05 MB to 17 MB in size - Mean file size = 3.74 MB - File types include: Excel, Word, Powerpoint, video, image, PDF, audio - The test was run against v2.6.1 of Halo ### AKS Cluster configuration assumes specific memory and compute allocations for containers. Production performance will always depend on size and complexities of real world files. Configurations can be optimised to favour throughput or file processing speeds. For these tests the following requests and limits were set on the pods. | Service | Requested CPU | Requested memory | CPU Limit | Memory limit |--------|--------|--------|--------|-------- | Engine | 200m | 500Mi (524 MB) | 1 core | 1Gi (1.07 GB) | API | 1500m | 4Gi (4.29 GB)| 1500m | 4Gi (4.29 GB) #### Small Based on a single node (1 x [DS5_v2](https://learn.microsoft.com/en-us/azure/virtual-machines/sizes/general-purpose/dsv2-series?tabs=sizebasic#sizes-in-series)) in a Kubernetes cluster comprising of: - 10 Engines - 16 virtual cores - 56 GB memory | Throughput | Files processed | Data processed |--------|--------|-------- | Per hour | 26,500 | 96 GB | Per day | 635,000 | 2300 GB | Mean processing speed | 136ms | | API Success rate | 99.999% | #### Medium Based on 5 node (5 x [DS4_v2](https://learn.microsoft.com/en-us/azure/virtual-machines/sizes/general-purpose/dsv2-series?tabs=sizebasic#sizes-in-series)) in a Kubernetes cluster comprising of: - 25 Engines - 8 virtual cores per node 40 total - 28 GB memory per node 140 total | Throughput | Files processed | Data processed |--------|--------|-------- | Per hour | 54,800 | 200 GB | Per day | 1,300,000 | 4800 GB | Mean processing speed | 65ms | | API Success rate | 99.999% | #### Large Based on 5 nodes (5 x [DS5_v2](https://learn.microsoft.com/en-us/azure/virtual-machines/sizes/general-purpose/dsv2-series?tabs=sizebasic#sizes-in-series)) in a Kubernetes cluster comprising of: - 50 Engines - 16 virtual cores per node 80 total - 56 GB memory per node 280 total | Throughput | Files processed | Data processed |--------|--------|-------- | Per hour | 93,800 | 343 GB | Per day | 2,250,000 | 8230 GB | Mean processing speed | 38ms | | API Success rate | 99.999% | ### OVA For these tests the following requests and limits were set on the pods. | Service | Requested CPU | Requested Memory | CPU Limit | Memory Limit |--------|--------|--------|--------|-------- | Engine | 1300m | 3Gi (3.22 GB)| 1300m | 3Gi (3.22 GB) | Api | 1 core | 3Gi (3.22 GB) | 1 core| 3Gi (3.22 GB) #### Small Based on a small VM ([F16s_v2](https://learn.microsoft.com/en-us/azure/virtual-machines/sizes/compute-optimized/fsv2-series?tabs=sizebasic#sizes-in-series)) running the OVA. comprising of: | Throughput | Files processed | Data processed |--------|--------|-------- | Per hour | 9,150 | 33 GB | Per day | 220,000 | 800 GB | Mean processing speed | 392ms | | API Success rate | 99.999% | - 3 Engines - 16 virtual cores - 32 GB memory #### Large Based on a large VM ([F32s_v2](https://learn.microsoft.com/en-us/azure/virtual-machines/sizes/compute-optimized/fsv2-series?tabs=sizebasic#sizes-in-series)) running the OVA. | Throughput | Files processed | Data processed |--------|--------|-------- | Per hour | 13,800 | 50 GB | Per day | 331,400 | 1200 GB | Mean processing speed | 260ms | | API Success rate | 99.999% | - 13 Engines - 32 virtual cores - 64 GB memory ### OVA vs Kubernetes performance differences - The OVA uses RKE with the Kubernetes control plane also using resources on the machine. - Multi node clusters need to use a file share for storing files. io with this share will incur an overhead. - While it limits the scalability, the OVA will inherently have faster network speeds between services due to all being on the same machine. Glasswall Halo is accessible via AWS Marketplace with your existing AWS account, allowing you to discover, try, and deploy Halo instantly and globally. Get started by visiting the [AWS Marketplace](https://aws.amazon.com/marketplace/pp/prodview-twapwbt4zhfx4?sr=0-2&ref_=beagle&applicationid=awsmpcontessa) listing and following the deployment steps in our [Amazon EC2 Setup Guide](/halo/aws-ec2-setup-guide). import ThemedImage from '@theme/ThemedImage'; import useBaseUrl from '@docusaurus/useBaseUrl'; ## Supported platforms Glasswall Halo currently supports deployment into the following Kubernetes and virtualised environments. --- ### **Managed Kubernetes** {/* | Icon | Platform | |------|----------| | | [Azure AKS](/halo/aks-setup-guide) | | | [AWS EKS](/halo/eks-setup-guide) | | | [Oracle OKE](/halo/oke-setup-guide) | | | [Google GKE](/halo/gke-setup-guide) | */} | Icon | Platform | |------|----------| | | [Azure AKS](/halo/aks-setup-guide) | | | [AWS EKS](/halo/eks-setup-guide) | | | [Oracle OKE](/halo/oke-setup-guide) | | | [Google GKE](/halo/gke-setup-guide) | --- ### **Virtual servers** {/* | Icon | Platform | |------|----------| | | [Amazon EC2](/halo/aws-ec2-setup-guide) | | | [Microsoft Hyper-V](/halo/hyperv-prerequisites) | | | Proxmox | | | [VMware vSphere / ESXi](/halo/vmware-prerequisites) | | */} | Icon | Platform | |------|----------| | | [Amazon EC2](/halo/aws-ec2-setup-guide) | | | [Microsoft Hyper-V](/halo/hyperv-prerequisites) | | | [Proxmox](/halo/proxmox-prerequisites) | | | [VMware vSphere / ESXi](/halo/vmware-prerequisites) | --- ### **Evaluation** | Icon | Platform | |------|----------| | | [Evaluation via AWS Marketplace](/halo/evaluation-via-aws-marketplace) | --- ### Related links - [Shared responsibility model](/halo/glasswall-shared-responsibility-model) - [API documentation](https://api.docs.glasswall.com/) Glasswall Halo deployment and upgrades are carried out via the use of Helm charts. Glasswall provides customers with a secure token to access the Helm charts and all associated container images. You can follow the steps below to complete the OKE deployment and setup, but please [contact us](https://www.glasswall.com/contact/) if you have any questions or require more support. ## Glasswall Halo installation via Oracle Kubernetes Engine (OKE) * [**Prerequisites**](/halo/oke-prerequisites) * [**Step 1 - Login to OCI CLI**](/halo/oke-step-1) * [**Step 2 - Create Glasswall Halo namespace**](/halo/oke-step-2) * [**Step 3 - Add secrets in vault**](/halo/oke-step-3) * [**Step 4 - Enable access to Glasswall's artifact registry**](/halo/oke-step-4) * [**Step 5 - Pull Helm charts** ](/halo/oke-step-5) * [**Step 6 - Install and configure prerequisite components**](/halo/oke-step-6) * [**Step 7 - Install CDR components**](/halo/oke-step-7) * * * Let's get started → Before you begin the process of deploying Glasswall Halo, check that you have the following tools and resources installed and setup. ## Required tools * [Helm](https://helm.sh/docs/intro/install/) * [Kubectl](https://kubernetes.io/docs/tasks/tools/) * [OCI CLI](https://docs.oracle.com/en-us/iaas/Content/API/SDKDocs/cliinstall.htm#Quickstart) ## Required OCI resources ### 1. OKE cluster * Recommended total of at least 8 vCPU and 32 GB RAM. * Minimum node size is 4 vCPU and 16 GB RAM. * For production workloads a minimum of 2 nodes is recommended. * Only a 'Basic' cluster is required. >**Note:** Glasswall Halo does not support ARM64 node VMs. For guidance on creating an OKE cluster please refer to: * [Container Engine and Kubernetes concepts](https://docs.oracle.com/en-us/iaas/Content/ContEng/Concepts/contengclustersnodes.htm) * [Creating Kubernetes clusters using console workflows](https://docs.oracle.com/en-us/iaas/Content/ContEng/Tasks/contengcreatingclusterusingoke.htm) * [Managing Kubernetes clusters](https://docs.oracle.com/en-us/iaas/Content/ContEng/Tasks/contengmanagingclusters_topic.htm) ### 2. File storage and mount target The OCI File Storage Service (FSS) is used to create persistent volumes in OKE. Create the file system instance in the same VCN where OKE cluster is created. For guidance on creating the FSS please refer to: * [Overview of File Storage](https://docs.oracle.com/en-us/iaas/Content/File/Concepts/filestorageoverview.htm) * [Managing mount targets](https://docs.oracle.com/en-us/iaas/Content/File/Tasks/managingmounttargets.htm) ### 3. Vault The OCI vault is used to store sensitive data, such as connection strings and secrets. For guidance on creating the FSS please refer to: * [Overview of vault](https://docs.oracle.com/en-us/iaas/Content/KeyManagement/Concepts/keyoverview.htm) * [Managing vaults](https://docs.oracle.com/en-us/iaas/Content/KeyManagement/Tasks/managingvaults.htm) * [Create vault secret](https://docs.oracle.com/en-us/iaas/Content/KeyManagement/Tasks/managingsecrets_topic-To_create_a_new_secret.htm) ## Access to Glasswall Artifact Registry * You are provided with a token & token ID to access [Glasswall's Artifact Registry](glasswall-artifact-registry). * This allows you to directly pull container images and Helm charts from your AKS cluster. * In subsequent steps, the token and token ID will be referred to as: **token** and **token_id**. ## Assigning variables ### Variables assigned before you begin Variables for resources mentioned in [prerequisites](/halo/oke-prerequisites#required-oci-resources) above need to be assigned before you can begin Glasswall Halo installation. you'll need to replace **"..."** with your own values. **Note:** the steps below assume each resource is in the same OCI compartment referred to as: **compartment_ocid** * The [OCI region](https://docs.oracle.com/en-us/iaas/Content/General/Concepts/regions.htm) where resources are created: **oracle_region="..."** * The OCI compartment ID: **compartment_ocid="..."** * The OKE cluster's OCID: **oke_cluster_ocid="..."** * The FSS mount target's OCID: **mount_target_ocid="..."** * The [availability domain](https://docs.oracle.com/en-us/iaas/Content/General/Concepts/regions.htm) of the FSS mount target: **availability_domain="..."** * The vault's OCID: **vault_ocid="..."** * The vault secret name: **vault_secret_name="..."** * * * Continue Need help? Once the [OCI CLI](https://docs.oracle.com/en-us/iaas/Content/API/SDKDocs/cliinstall.htm#Quickstart) is installed, there are two primary authentication methods available. - **Key pair based authentication** - Recommended for automation scenarios such as CI and CD pipelines - [Documentation](https://docs.oracle.com/en-us/iaas/Content/API/Concepts/apisigningkey.htm#Required_Keys_and_OCIDs) - **Token based authentication** - Recommended for local development - [Documentation](https://docs.oracle.com/en-us/iaas/Content/API/SDKDocs/clitoken.htm#Tokenbased_Authentication_for_the_CLI) >The recommended approach is to use token based authentication for local development and key pair based authentication for automation scenarios such as CI and CD pipelines. ## Authentication 1. To configure the OCI CLI for token based authentication, run the commands below. If you encounter any issues, refer to the linked documentation for additional guidance. Note that `gworacle` is the tenant name used in Oracle Cloud. 2. After running the `oci session authenticate` command, you will be prompted to enter a profile name. Use the same profile name specified in the default profile configuration, `oci_auth`. >```bash >export OCI_CLI_AUTH=security_token >mkdir -p $$HOME/.oci >cat $$HOME/.oci/oci_cli_rc >[OCI_CLI_SETTINGS] >default_profile=oci_auth >EOF >oci session authenticate >``` >After authenticating your session, you should get an output similar to the one below: >```shell ># Try out your newly created session credentials with the following example command: >oci iam region list --config-file $$HOME\.oci\config --profile oci_auth --auth >security_token >``` >However, you should able to run just `oci iam region list` without the `--profile` and `--auth` parameters since they were configured prior in the step above. 3. Once logged in, set your Kubernetes context by running the following command: >```bash ># As our dev cluster is in uk-london-1 region >oracle_region=uk-london-1 >oci ce cluster create-kubeconfig --cluster-id ${oke_cluster_ocid} --region >${oracle_region} --token-version 2.0.0 >``` >This will merge your OKE settings into your Kubernetes configuration file (typically stored in the`~/.kube/config` file). * * * Continue Need help? All services associated with Glasswall Halo are deployed to a dedicated Kubernetes namespace named **cdrplatform**. Run the following commands to create the namespace and set it as the default for subsequent steps. ```sh kubectl create namespace "cdrplatform" kubectl create namespace "license-management" kubectl config set-context --current --namespace="cdrplatform" ``` * * * Continue Need help? ## 3.1 Create Dynamic Group and IAM Policy The Dynamic Group and IAM Policy together grant the OKE cluster nodes permission to sync Vault secrets to Kubernetes. Replace `` and `` with appropriate values for your environment. ```bash oci iam dynamic-group create --name "" \ --description "Dynamic Group for OKE cluster nodes" \ --matching-rule "instance.compartment.id='${compartment_ocid}'" oci iam policy create -c ${compartment_ocid} --name '' \ --description "Oracle vault secrets access for OKE" \ --statements "[\"Allow dynamic-group to read secret-family in compartment where target.vault.id='${vault_ocid}'\"]" ``` ## 3.2 Add MongoDB passwords to the Vault secret **Note:** if you do not plan to manage policies or use the Asynchronous API, you can skip this step. To allow the Glasswall Halo Policy Management API to create and manage policies in MongoDB, and the Asynchronous API to process requests, MongoDB must be deployed using Helm charts. The MongoDB Helm chart creates two users. The passwords for these users must be stored in a Vault secret. Sensitive data, such as passwords, must be stored as key value pairs in JSON format within the Vault secret. If passwords from multiple steps are required, create a single JSON object containing all passwords or update the existing JSON object with the additional values. ```json { "mongodb-cdrp-password": "", "mongodb-admin-password": "" } ``` ## 3.3 [Optional] Add ReversingLabs credentials To integrate Halo with ReversingLabs, store the ReversingLabs credentials in Key Vault. ```json { "reversinglabs_username": "", "reversinglabs_password": "" } ``` ## 3.4 [Optional] Add ICAP mTLS certificates ICAP servers can be configured for mutual client authentication using mTLS certificates. The certificates are mounted to the ICAP server pods as Kubernetes secrets. Add the server certificates and certificate authority certificates to Oracle Vault so they can be synced to Kubernetes secrets. ```json { "tls-server-cert": "", "tls-server-key": "", "tls-cafile": "" } ``` * * * Continue Need help? Glasswall’s Artifact Registry stores the container images and Helm charts for Glasswall Halo. To pull container images and Helm charts from the Artifact Registry, you must use your `token` and `token_id`. - First, create a Kubernetes secret to allow the cluster to access the registry. >```sh >kubectl create secret docker-registry acr-secret -n cdrplatform \ > --docker-server="glasswallhub.azurecr.io" \ > --docker-username="${token_id}" \ > --docker-password="${token}" >kubectl create secret docker-registry acr-secret -n license-management \ > --docker-server="glasswallhub.azurecr.io" \ > --docker-username="${token_id}" \ > --docker-password="${token}" >``` - Then, login to the Helm registry: >```sh >helm registry login "glasswallhub.azurecr.io" -u "${token_id}" -p "${token}" >``` * * * Continue Need help? Glasswall Halo consists of Glasswall Helm charts hosted in our authenticated Artifact Registry, alongside a few industry recognized third-party charts. - Before the charts can be installed, the right versions need to be pulled by Helm. The required versions are listed in the corresponding release notes. - An example pull command is shown below; this needs to be executed for each Helm chart listed. >```sh >helm pull oci://glasswallhub.azurecr.io/ --version --untar --untardir . >``` - The following example is pre-populated with the versions for [v2.16.0](/halo/v2160): ```sh helm pull oci://glasswallhub.azurecr.io/helm/cdrplatform-rabbitmq --version 0.8.0 --untar --untardir . helm pull oci://glasswallhub.azurecr.io/helm/cdrplatform-external-secrets --version 0.5.0 --untar --untardir . helm pull oci://glasswallhub.azurecr.io/helm/cdrplatform-storage --version 0.7.0 --untar --untardir . helm pull oci://glasswallhub.azurecr.io/helm/cdrplatform-engine --version 0.4.0 --untar --untardir . helm pull oci://glasswallhub.azurecr.io/helm/cdrplatform-sync-api --version 0.3.0 --untar --untardir . helm pull oci://glasswallhub.azurecr.io/helm/cdrplatform-portal --version 0.3.0 --untar --untardir . helm pull oci://glasswallhub.azurecr.io/helm/cdrplatform-policy-api --version 0.2.0 --untar --untardir . helm pull oci://glasswallhub.azurecr.io/helm/cdrplatform-api-access --version 1.1.0 --untar --untardir . helm pull oci://glasswallhub.azurecr.io/helm/cdrplatform-portal-access --version 0.1.1 --untar --untardir . helm pull oci://glasswallhub.azurecr.io/helm/cdrplatform-license-management --version 0.2.0 --untar --untardir . helm pull oci://glasswallhub.azurecr.io/helm/cdrplatform-cleanup --version 0.2.0 --untar --untardir . helm pull oci://glasswallhub.azurecr.io/helm/cdrplatform-async-api --version 0.1.0 --untar --untardir . helm pull oci://glasswallhub.azurecr.io/helm/cdrplatform-metrics-collation --version 0.0.4 --untar --untardir . helm pull oci://glasswallhub.azurecr.io/helm/cdrplatform-metrics-projection --version 0.1.0 --untar --untardir . helm pull oci://glasswallhub.azurecr.io/helm/cdrplatform-report-extractor --version 0.1.0 --untar --untardir . helm pull oci://glasswallhub.azurecr.io/helm/cdrplatform-tally-accumulator --version 0.1.0 --untar --untardir . helm pull oci://glasswallhub.azurecr.io/helm/cdrplatform-storage-monitor --version 0.1.2 --untar --untardir . helm pull oci://glasswallhub.azurecr.io/helm/cdrplatform-mongodb --version 0.2.0 --untar --untardir . ``` In order to gain access to the repositories from which our prerequisite components are installed, they must first be added: ```sh helm repo add mongodb https://mongodb.github.io/helm-charts helm repo update ``` * * * Continue Need help? Now that you have access to the Helm charts, they can be deployed. - First install the prerequisite components: ```sh # Install rabbitMQ cluster operator helm upgrade --install rabbitmq-cluster-operator oci://glasswallhub.azurecr.io/docker/bitnamicharts/rabbitmq-cluster-operator \ --atomic \ --version 4.4.23 \ --set global.imageRegistry=glasswallhub.azurecr.io \ --set global.imagePullSecrets[0]=acr-secret \ --set global.security.allowInsecureImages=true \ --set msgTopologyOperator.fullnameOverride=rabbitmq-messaging-topology-operator \ --set clusterOperator.image.tag=2.15.0-debian-12-r0 \ --set msgTopologyOperator.image.tag=1.17.2-debian-12-r0 \ --set credentialUpdaterImage.tag=1.0.7-debian-12-r0 \ --set rabbitmqImage.tag=4.1.1-debian-12-r2 \ --set clusterOperator.watchAllNamespaces=false \ --set clusterOperator.watchNamespaces={cdrplatform} \ --set msgTopologyOperator.watchAllNamespaces=false \ --set msgTopologyOperator.watchNamespaces={cdrplatform} \ --set clusterOperator.resources.requests.cpu=100m \ --set clusterOperator.resources.requests.memory=256Mi \ --set clusterOperator.resources.limits.cpu=100m \ --set clusterOperator.resources.limits.memory=256Mi \ --set msgTopologyOperator.resources.requests.cpu=100m \ --set msgTopologyOperator.resources.requests.memory=256Mi \ --set msgTopologyOperator.resources.limits.cpu=100m \ --set msgTopologyOperator.resources.limits.memory=256Mi # Install keda helm upgrade --install keda "oci://glasswallhub.azurecr.io/ghcr/home-operations/charts-mirror/keda" --atomic \ --set global.image.registry="glasswallhub.azurecr.io/ghcr" \ --set imagePullSecrets[0].name=acr-secret \ --version 2.17.2 # Install nginx ingress controller helm upgrade --install nginx-ingress oci://glasswallhub.azurecr.io/k8s/ingress-nginx/charts/ingress-nginx --atomic \ --set imagePullSecrets[0].name=acr-secret \ --set global.image.registry="glasswallhub.azurecr.io" \ --set controller.image.image="k8s/ingress-nginx/controller" \ --set controller.admissionWebhooks.patch.image.image="k8s/ingress-nginx/kube-webhook-certgen" \ --set controller.image.digest=null \ --set controller.admissionWebhooks.patch.image.digest=null \ --set controller.service.annotations."service\.beta\.kubernetes\.io/azure-load-balancer-health-probe-request-path"=/healthz \ --version 4.12.8 # Install External Secrets Operator helm upgrade --install external-secrets oci://glasswallhub.azurecr.io/ghcr/external-secrets/charts/external-secrets \ --set imagePullSecrets[0].name=acr-secret \ --set webhook.imagePullSecrets[0].name=acr-secret \ --set certController.imagePullSecrets[0].name=acr-secret \ --set image.repository="glasswallhub.azurecr.io/ghcr/external-secrets/external-secrets" \ --set webhook.image.repository="glasswallhub.azurecr.io/ghcr/external-secrets/external-secrets" \ --set certController.image.repository="glasswallhub.azurecr.io/ghcr/external-secrets/external-secrets" \ --version 0.16.1 \ --set image.tag=v0.9.11 \ --set installCRDs=true \ --atomic # Install Glasswall Halo External Secrets helm upgrade --install cdrplatform-external-secrets cdrplatform-external-secrets -n cdrplatform --atomic --create-namespace \ --set cloud_providers.oracle.enabled=true \ --set cloud_providers.oracle.vault=${vault_ocid} \ --set cloud_providers.oracle.region=${oracle_region} \ --set cloud_providers.oracle.secretName="${vault_secret_name}" # Install MongoDB kubernetes operator helm install community-operator mongodb/community-operator --namespace ${operator_namespace} \ --set operator.version=0.13.0 \ --set agent.version=12.0.25.7724-1 \ --atomic ``` >**Note:** when upgrading the `rabbitmq-cluster-operator` Helm chart, CRDs will not be installed. If new CRDs are introduced in the new releases, the CRDs needs to be installed manually to avoid issues with running the RabbitMQ cluster operator pods. ```sh helm pull bitnami/rabbitmq-cluster-operator --untar kubectl apply -f rabbitmq-cluster-operator/crds/ ``` - Then, install the supporting components. > Due to an inconsistency in the OCI API, the prefix of the `availability_domain` value needs to be omitted for the command below. > E.g. For a value of `UyoZ:UK-LONDON-1-AD-1` only `UK-LONDON-1-AD-1` is needed. ```bash helm upgrade --install cdrplatform-storage cdrplatform-storage -n cdrplatform --wait --atomic \ --set cloud_provider=oracle \ --set oracle.fss.availability_domain=${availability_domain} \ --set oracle.fss.mount_target_ocid=${mount_target_ocid} helm upgrade --install cdrplatform-rabbitmq cdrplatform-rabbitmq -n cdrplatform \ --set image.registry=glasswallhub.azurecr.io \ --set image.tag=160215 \ --set cloud_provider=oracle ``` * * * Continue Need help? Finally, install the Glasswall Halo services. For each Helm chart, ensure that the image tag is set to the corresponding version listed in the release notes. The examples below are pre populated with values for [v2.16.0](/halo/v2160). ## 7.1 - Engine ```sh helm upgrade --install cdrplatform-engine cdrplatform-engine -n cdrplatform \ --set image.tag=165275 \ --set image.registry=glasswallhub.azurecr.io \ --set cloud_provider=oracle \ --atomic ``` ### 7.1A - Enable ReversingLabs (optional) To enable ReversingLabs file reputation: 1. Store the credentials in Vault under `halo-reversinglabs-username` and `halo-reversinglabs-password`, as described in [Step 3](/halo/oke-step-3). external-secrets syncs them into `cdrplatform-secrets`. 2. Set the non-secret fields on the Engine-owned `engine-reversinglabs` ConfigMap. The ReversingLabs endpoint is fixed by the Engine and is not configurable. ```sh kubectl patch configmap engine-reversinglabs -n cdrplatform --type merge \ -p '{"data":{"enabled":"true","timeoutSeconds":"5"}}' ``` `timeoutSeconds` accepts 1–120 and defaults to 5. No restart is required: the Engine picks up the change once the updated ConfigMap is projected into the pod, typically within a minute. ## 7.2 - Synchronous API ```sh helm upgrade --install cdrplatform-sync-api cdrplatform-sync-api -n cdrplatform \ --set image.tag=166415 \ --set image.registry=glasswallhub.azurecr.io \ --set cloud_provider=oracle ``` ## 7.3 - API Access service The API Access service acts as a gateway to the Glasswall Halo Synchronous API and the Policy Management API. It exposes CDR functionality over HTTP. If your environment requires HTTPS with TLS or SSL, follow the instructions to install the service with TLS or SSL enabled. Otherwise, follow the instructions for installation without TLS or SSL. ### 7.3A - For deployments without TLS/SSl ```sh helm upgrade --install cdrplatform-api-access -n cdrplatform cdrplatform-api-access \ --set image.registry=glasswallhub.azurecr.io \ --atomic --set image.tag=165925 ``` ### 7.3B - To use SSL certificates on CDR API, create a private key and certificate for the domain to be used. Create a Kubernetes secret using the key and CRT files using the command below. With this command we create a secret with the name "tls-secret" from the files server.key (private key) and server.crt (certificate). The key should not be passphrase protected in this example. ``` kubectl create secret tls tls-secret --key server.key --cert server.crt ``` This secret can then be used to enable TLS on the ingress ensuring that the domain name is set on the command below: ```sh helm upgrade --install cdrplatform-api-access -n cdrplatform cdrplatform-api-access \ --set image.registry=glasswallhub.azurecr.io \ --set image.tag=165925 \ --set cloud_provider=oracle \ --set ingress.tls.enabled=true \ --set ingress.tls.domain= \ --set ingress.tls.secretName=tls-secret \ --atomic ``` ### 7.3C - Enable API authentication Authentication for the Glasswall Halo API is disabled by default. To enable authentication, complete the steps below. - Add two new entries to the Vault secret `vault_secret_name`. - One entry for the organisation ID - One entry for the organisation tokens When specifying multiple tokens, separate them with commas. Ensure that individual tokens do not contain commas. - Vault secret entries must follow the naming conventions below. - **Organisation ID** - Must start with `organisation` - Must end with `-id` - Include a numeric index between the prefix and suffix - Examples: `organisation1-id`, `organisation2-id`, `organisation3-id` - **Organisation tokens** - Must start with `organisation` - Must end with `-tokens` - Include a numeric index between the prefix and suffix - Examples: `organisation1-tokens`, `organisation2-tokens`, `organisation3-tokens` ```json { "organisation1-id": "org-1", "organisation1-tokens": "org-1-token-1,org-1-token-2" } ``` - Set `configuration.AuthenticationScheme=Basic` while deploying the helm chart. For example, ```sh helm upgrade --install cdrplatform-api-access -n cdrplatform cdrplatform-api-access \ --set image.registry=glasswallhub.azurecr.io \ --set image.tag=165925 \ --set configuration.AuthenticationScheme=Basic \ --set cloud_provider=oracle \ --atomic ``` ## 7.4 - Portal ### 7.4A - Deployments without TLS or SSL To deploy the Portal service, run the commands below. **Note:** the `` referenced in these commands is the public IP address of the load balancer. This can be retrieved by following the **Portal and API Access** steps below. ```sh helm upgrade --install cdrplatform-portal cdrplatform-portal -n cdrplatform \ --set image.registry=glasswallhub.azurecr.io \ --set configuration.OIDC=null \ --set image.tag=166435 \ --set configuration.BackendUrl=http:// \ --set configuration.HaloVersion=2.16.0 \ --atomic ``` ### 7.4B - Deployments with TLS or SSL To enable TLS, add the `--set ingress.tls.enable_tls=true` and `--set ingress.tls.domain=` parameters. In the example below, the same Kubernetes secret created for the API Access service is used to retrieve the TLS certificates. ```sh helm upgrade --install cdrplatform-portal cdrplatform-portal -n cdrplatform \ --set image.tag=166435 \ --set image.registry=glasswallhub.azurecr.io \ --set ingress.tls.enabled=true \ --set ingress.tls.domain= \ --set ingress.tls.secretName=tls-secret \ --set configuration.BackendUrl=https:// \ --set configuration.OIDC=null \ --set configuration.HaloVersion=2.16.0 \ --atomic ``` ### 7.4C - Enable admin access without authentication When Single Sign On login is not enabled for the Portal, administrative features such as license management and policy configuration are not accessible. To enable access to these administrative features without configuring SSO, deploy the Helm chart with the following configuration. ```sh helm upgrade --install cdrplatform-portal cdrplatform-portal \ --set image.tag=166435 \ --set image.registry=glasswallhub.azurecr.io \ --set ingress.tls.enabled=true \ --set ingress.tls.domain=${portal_domain} \ --set ingress.tls.secretName=tls-secret \ --set configuration.AutoAdmin="true" \ --atomic ``` ## 7.5 - MongoDB MongoDB is a prerequisite for Policy Management API and Asynchronous API. ```sh helm upgrade -i cdrplatform-mongodb cdrplatform-mongodb -n cdrplatform --atomic \ --set cloud_provider=oracle ``` After MongoDB is deployed, get the connection string of the MongoDB and add it to the key value secret. * Retrieve the connection string from the Kubernetes secret: ```sh kubectl get secret mongodb-cdrplatform-cdrp-user -o jsonpath='{.data.connectionString\.standard}' | base64 -d ``` - Update the Oracle Vault secret by adding the key-value pair: ```json { "mongodb-connectionstring": "" } ``` - After updating Oracle Vault, run the following command to sync the secrets to Kubernetes. ```sh kubectl annotate externalsecret external-secret updated_at=$(date +%s) --overwrite -n cdrplatform ``` - Describe the `cdrplatform-secrets` Kubernetes secret and verify that `mongodb-connectionstring` is present and synced. Then proceed with deploying the next Helm charts. ```sh kubectl describe secret cdrplatform-secrets ``` ## 7.6 - Policy Management API The Policy Management API is used to manage Glasswall Halo content management flags. This service is optional and should be installed only if you plan to create and use custom policies. ```sh helm upgrade --install cdrplatform-policy-api -n cdrplatform cdrplatform-policy-api \ --set image.registry=glasswallhub.azurecr.io \ --set image.tag=165921 \ --set cloud_provider=oracle ``` ## 7.7 - Portal Access The Portal Access service acts as the backend for the Portal. It enables the Portal to access the Policy Management API and the Synchronous API. ### 7.7A - Deployments with TLS or SSL If TLS is required, add the `--set ingress.tls.enable_tls=true` and `--set ingress.tls.domain=` parameters. In the example below, the same Kubernetes secret created for the API Access service is used to retrieve the TLS certificates. ```sh helm upgrade --install cdrplatform-portal-access -n cdrplatform cdrplatform-portal-access \ --set image.registry=glasswallhub.azurecr.io \ --set image.tag=166410 \ --set ingress.tls.enabled=true \ --set ingress.tls.domain= \ --set ingress.tls.secretName=tls-secret \ --set configuration.AuthenticationScheme=None \ --atomic ``` ### 7.7B - For deployments without TLS/SSL ```sh helm upgrade --install cdrplatform-portal-access -n cdrplatform cdrplatform-portal-access \ --set image.registry=glasswallhub.azurecr.io \ --set configuration.AuthenticationScheme=None \ --set image.tag=166410 \ --atomic ``` ## 7.8 - License management The License management service is used to manage licenses in Glasswall Halo. ```sh helm upgrade --install cdrplatform-license-management cdrplatform-license-management \ --set image.registry=glasswallhub.azurecr.io \ --set image.tag=157724 \ -n license-management \ --atomic ``` ## 7.9 - Clean up service The Clean up service deletes the original and rebuilt files from the persistent storage after the files are processed. ```sh helm upgrade --install cdrplatform-cleanup cdrplatform-cleanup \ --set image.registry=glasswallhub.azurecr.io \ --set cloud_provider=oracle \ --set image.tag=165228 \ --atomic ``` ## 7.10 Asynchronous API The Asynchronous API can be deployed using the command below. A MongoDB database is required for the Asynchronous API. ```sh helm upgrade --install cdrplatform-async-api cdrplatform-async-api \ --set image.tag=166413 \ --set image.registry=glasswallhub.azurecr.io \ --atomic ``` ## 7.11 - Metrics collation (removed from version 2.6.2) The Metrics collation service captures events from CDRPlatform and stores them in MongoDB. Only deploy it when using Halo version 2.6.1 and below. ```sh helm upgrade --install cdrplatform-metrics-collation cdrplatform-metrics-collation \ --set image.tag=120018 \ --set image.registry=glasswallhub.azurecr.io \ --set cloud_provider=oracle \ --atomic ``` ## 7.12 - Metrics projection The Metrics projection service is used to pull reporting data from MongoDB to display in the Portal UI. ```sh helm upgrade --install cdrplatform-metrics-projection cdrplatform-metrics-projection \ --set image.tag=166407 \ --set image.registry=glasswallhub.azurecr.io \ --set cloud_provider=oracle \ --atomic ``` ## 7.13 - Report extractor The Report extractor service extracts analysis reports and publishes them for reporting. ```sh helm upgrade --install cdrplatform-report-extractor cdrplatform-report-extractor \ --set image.tag=166411 \ --set image.registry=glasswallhub.azurecr.io \ --wait --atomic ``` ## 7.11 - Tally accumulator The Tally accumulator service tracks and maintains a tally of usage statistics. ```sh helm upgrade --install cdrplatform-tally-accumulator cdrplatform-tally-accumulator \ --set image.tag=165227 \ --set image.registry=glasswallhub.azurecr.io \ --set cloud_provider=oracle \ --wait --atomic ``` ## 7.12 - Storage monitoring service The Storage Monitoring service provides an API to subscribe to and track changes made to documents in configured storage services such as SharePoint. TLS must be enabled for the webhook. Add the `--set ingress.tls.enable_tls=true` and `--set ingress.tls.domain=` parameters. In the example below, the same Kubernetes secret created for the API Access service is used to retrieve the TLS certificates. ```sh helm upgrade --install cdrplatform-storage-monitor cdrplatform-storage-monitor --wait --atomic \ --set image.tag="165098" \ --set image.repository="glasswallhub.azurecr.io/cdrplatform-storage-monitor" \ --set configuration.DATABASE__Provider="${database_provider}" \ --set configuration.WEBHOOKS__CallbackBaseAddress="https://" \ --set ingress.tls.enabled=true \ --set ingress.tls.domain= \ --set ingress.tls.secretName=tls-secret \ --atomic ``` ## 7.13 - Portal and API access Use the command below to determine the external IP address associated with your cluster. **Note:** the external IP is the same as the public IP address assigned to the Oracle Cloud load balancer. ```sh kubectl get services --namespace cdrplatform nginx-ingress-ingress-nginx-controller --output jsonpath='{.status.loadBalancer.ingress[0].ip}' ``` You can now use the IP returned above to navigate to the Portal and API documentation (use https if TLS enabled): ```sh Portal: http:// API Documentation: http:///swagger ``` * * * Need help? As an alternative to cloud managed Kubernetes deployments, Glasswall Halo can be deployed in a single node configuration on traditional virtual machine infrastructure. This option is intended for on premises environments where cloud based infrastructure is not available or permitted. Single node deployments consist of the following core components. For version details, refer to the Release Notes. - Red Hat Enterprise Linux base operating system - Rancher Kubernetes Engine Government (RKE2), a Cloud Native Computing Foundation certified Kubernetes distribution - Glasswall Halo Helm charts and container images To simplify deployment, preconfigured virtual machine images are available in OVA and VHD formats. ## Architecture ## Security posture The following security hardening procedures have been applied to each of the core components of the virtual machine deployments. - **Red Hat Enterprise Linux base operating system** - Hardened using the [DoD Security Technical Implementation Guides (STIGs)](https://public.cyber.mil/stigs/downloads/) for Red Hat Enterprise Linux - **Rancher Kubernetes Engine Government (RKE2)** - Hardened using the [Center for Internet Security (CIS) Kubernetes configuration benchmarks for RKE2](https://docs.rke2.io/security/hardening_guide) - **Glasswall Halo Helm charts and container images** - In line with standard Glasswall Halo deployments, all Helm charts and container images undergo continuous SCA, SAST, DAST, IaC, and container security scanning, along with associated patching Security reports generated from these hardening procedures are available on request. ## Current limitations of single node VM deployments In a single node deployment, scalability is limited by the capacity of the target virtual machine. The deployment dynamically allocates the appropriate number of engines based on the available resources of the underlying VM. ## Virtualisation ### Which hypervisors do you support, including the version? We support the following virtualisation platforms: - Microsoft Hyper-V: Windows Server 2019+ and Windows 10+ - VMware: ESXi 7.0.0+ - VirtualBox: 7.0.8+ ## Resources ### How is storage managed in the VM? A solid state drive (SSD) (for better I/O and performance) or a hard disk drive (HDD) can be attached to the VM for storage. ### What is the minimum number of virtual cores required? 16 ### What is the minimum disk size of the VM? 100 GB ### Does the VM self-rotate logs? Yes. Logs for services running in the OS follow the default Red Hat log rotation policy configured in `/etc/logrotate.conf`. This file can be customised as needed. Pod logs from RKE2 follow the default RKE2 log rotation: - Maximum log files per container: **3** - Maximum size per log file: **2 MB** ### Is there a benefit in assigning more vCPUs to increase performance? No. Increasing CPU allocation does **not** improve the performance of Glasswall Halo's synchronous API. ### What is the minimum memory required? 32 GB ### What is the size of the VM image? - VMware or VirtualBox: OVA approximately **5.5 GB** - Hyper-V: VHD **64 GB** ## Security ### Is an OS firewall active? No. The firewall is disabled to avoid conflicts with Kubernetes networking. ### What firewall rules are applied? None — firewall is disabled. ### What network ports remain open on the VM? - **80** - **443** - **22** (SSH) - **6443** (Kubernetes API server — restricted to cluster CIDR `10.42.0.0/16` and service CIDR `10.43.0.0/16`) ### Is there protection for brute-force SSH attempts? Yes. **Fail2Ban** monitors log files (e.g., `/var/log/auth.log`) and blocks malicious IP addresses after repeated failures. ### How do I keep the base OS patched? Glasswall provides updated VM images quarterly. To update directly from Red Hat mirrors: ```shell subscription-manager register --username --password --auto-attach ``` ### How do I configure my organisation's SSL certificate and preferred FQDN? Copy the private key and certificate of the domain to the VM and run `bash configure_tls.sh ` in the VM. ### How do I enforce certificate only authentication via SSH? Update `~/.ssh/authorized_keys` with your public key in the VM - disable password authentication: ```sh sudo sed -i "s/passwordauthentication yes/passwordauthentication no/g" /etc/ssh/sshd_config sudo service sshd restart ``` ### Does the default SSH username and password require immediate update? Yes, it is mandatory to update the password. Once you are logged into the VM, you will be prompted to update the password. ### What level of security hardening has taken place? * [STIG](https://public.cyber.mil/stigs) hardening is applied to the base OS. * CIS hardening is applied to the RKE2 (Kubernetes cluster). * All Glasswall containers utilise hardened container images. * All Glasswall software is hardened using SAST, DAST, SCA tooling. ### Do you provide hashes to authenticate the authenticity of the VM image? Yes, Glasswall provides base64 encoded MD5 hash value of the file. Verify if it matches by using the command by replacing `$ova_file_path` with the OVA file path: ```sh openssl dgst -md5 -binary < $ova_file_path | base64 ``` ### How do I enable an anti-virus solution in the VM? There is no anti-virus solution installed in the VM image. If needed, an anti-virus solution can be installed separately in the VM. ### Will an anti-virus solution impede or hang the CDR process? An anti-virus solution can interfere with the CDR process and hence the folders in the VM where files being processed should be excluded from the anti-virus scanning. ### Are there any folders which need to be out of scope for anti-virus protection? Yes, `/opt/local-path-provisioner` should be excluded from anti-virus scanning. ### Does RKE2 support FIPS 140-2 validated encryption? Yes, FIPS 140-2 support is built into RKE2 at the foundation level. Specifically, the functions used within RKE2 meet the stringent security requirements outlined in the FIPS 140-2 standard. This includes the algorithms used for encryption and decryption, the methods used for key generation and management, and the protections in place to prevent unauthorized access or use of the cryptographic modules. Additional documentation regarding FIPS 140-2 enablement can be found [here](https://docs.rke2.io/security/fips_support). ### Will an anti-virus solution impede or hang the CDR process? An anti-virus solution can interfere with the CDR process and hence the folders in the VM where files being processed should be excluded from the anti-virus scanning. ## OS configuration ### How do I configure the system banner message when I log onto the VM? The banner message can be customized by updating `/etc/issue` file in the VM. ### What OS level services are running on the VM? These are the list of OS level services running on the VM: Unit| load| active| state| description ---|---|---|---|--- Atd.service| loaded| active| running| job spooling tools Auditd.service| loaded| active| running| security auditing service Chronyd.service| loaded| active| running| NTP client/server Dbus.service| loaded| active| running| D-Bus system message bus Getty@tty1.service| loaded| active| running| getty on tty1 Irqbalance.service| loaded| active| running| irqbalance daemon Libstoragemgmt.service| loaded| active| running| libstoragemgmt plug-in server daemon Mcelog.service| loaded| active| running| machine check exception logging daemon Networkmanager.service| loaded| active| running| network manager Polkit.service| loaded| active| running| authorization manager Rhsmcertd.service| loaded| active| running| enable periodic update of entitlement certificates. Rke2-server.service| loaded| active| running| Rancher Kubernetes Engine v2 (server) Rsyslog.service| loaded| active| running| system logging service Serial-getty@ttys0.service| loaded| active| running| serial getty on ttys0 Smartd.service| loaded| active| running| self monitoring and reporting technology (SMART) daemon Sshd.service| loaded| active| running| OpenSSH server daemon Systemd-journald.service| loaded| active| running| journal service Systemd-logind.service| loaded| active| running| login service Systemd-resolved.service| loaded| active| running| network name resolution Systemd-udevd.service| loaded| active| running| udev kernel device manager Usbguard.service| loaded| active| running| USBGuard daemon User@1000.service| loaded| active| running| user manager for UID 1000 Have unnecessary OS level services been deactivated? The Red Hat OS has been gone through the STIG hardening process and any unnecessary services have been removed and no unnecessary services have been installed. ### What is the base operating system, and what version? The base OS version can be found in the release notes. ### Does Kubernetes run within the VM? Yes, Kubernetes cluster runs with a Single Node. ### What version of Kubernetes is running? Kubernetes version can be found in the release notes. ## Monitoring ### What error messages should we actively monitor? Glasswall Halo error codes and API can be found via [Glasswall API documentation](https://api.docs.glasswall.com). Health of the Glasswall Halo cluster can be monitored using the API health endpoint. ## CPU arch support ### What CPU architecture is supported? Currently x86-64 CPU processors are supported. ARM support will be available in upcoming releases. ## Log rotation & storage management ### How do I offload/aggregate logs to my preferred network location? The VM comes with _syslog_ pre configured and this can be used to send the logs to a _syslog_ server. ## Upgrade path ### What happens if I replace this VM with another VM image from Glasswall, and what would be the full upgrade path to maintain live operations? * Create a new VM from the new image from Glasswall following the deployment steps. * Make sure the new VM is working as expected. * Switch the DNS record from old IP address to the new IP address of the VM. ## Patching ### How do I update the Kubernetes software from a security perspective? Kubernetes software updates will be done in the VM images provided by Glasswall. However, it is recommended to update the Kubernetes cluster whenever security advisories are published. When upgrading the Kubernetes version of a cluster, we recommend that you: * Take a snapshot. * Initiate a Kubernetes upgrade. * If the upgrade fails, revert the cluster to the pre-upgrade Kubernetes version. This is achieved by selecting the restore etcd and Kubernetes version option. This will return your cluster to the pre-upgrade Kubernetes version before restoring the etcd snapshot. * The restore operation will work on a cluster that is not in a healthy or active state. ## Configuration changes ### Does applying configuration changes cause downtime? Configuration changes made through the setup wizard or shell scripts (such as enabling authentication, TLS, or Storage Monitoring) briefly restart the affected services. This typically takes a few seconds per service. Complete initial configuration before the appliance is in production use, or schedule changes during a maintenance window. ## Networking ### How do I manage DNS / IP range settings? Use `nmcli` command line utility or `nmtui` utility to configure the IP address, gateway and DNS server. ### Is DHCP active in the VM? Yes, DHCP client is active in the VM and can allocate an IP address when deployed to a network with DHCP server. ### How can I SSH into the VM? Once an IP address is configured to the VM, SSH using the username and password/private key shared by Glasswall. ### What are the IP addresses or internet URLs that need to be allow listed? None. ### What communication protocols are supported with the VM? HTTPS and HTTP endpoints are provided within the VM. ## Licensing ### Do I need to purchase a Red Hat subscription separately for my Glasswall Halo virtual appliance? Yes. While Glasswall Halo virtual appliances run on Red Hat Enterprise Linux (RHEL), **a RHEL subscription is not included** with the appliance. Customers are responsible for obtaining a valid RHEL subscription **directly from Red Hat or an authorized partner** to ensure access to system updates, security patches, and support. Each deployed instance must be registered with Red Hat Subscription Management (RHSM) using your own Red Hat account credentials. Without a valid subscription, your appliance may not receive critical updates and will not be eligible for Red Hat support. For more information, visit our [RHEL licensing advisory](/halo/rhel-licensing). ### Is the Kubernetes software licensed by Glasswall? There is no need to purchase a license for the Kubernetes as it is open source with [Apache 2.0 license](https://github.com/rancher/rke2/blob/master/LICENSE). ### What are the main open source components that are comprised within the overall solution? RKE2 & RabbitMQ are the main open source components and the complete software bill of materials (SBOM) can be provided on request. ### How are the open source components licensed? * RKE2 Kubernetes cluster is licensed under [Apache 2.0 license](https://github.com/rancher/rke2/blob/master/LICENSE). * RabbitMQ cluster Kubernetes operator is licensed under [Mozilla public license 2.0](https://github.com/RabbitMQ/cluster-operator/blob/main/LICENSE.txt). * RabbitMQ server is licensed under [Apache 2.0 license](https://github.com/RabbitMQ/RabbitMQ-server/blob/main/LICENSE-APACHE2). * Additional licensing information can be provided by software bill of materials (SBOM) on request. ## Advisory: Red Hat Enterprise Linux RHEL licensing requirements for Glasswall Halo virtual appliance customers Glasswall Halo virtual appliances are built on **Red Hat Enterprise Linux RHEL** to provide a secure and stable foundation for the platform. **Important:** A RHEL subscription is **not included** with the Glasswall Halo virtual appliance. Glasswall does not resell or bundle RHEL licenses. ### Customer responsibility for RHEL licensing To receive RHEL updates, security patches, and support, customers must obtain a valid RHEL subscription directly from **Red Hat** or an **authorized Red Hat partner**. #### Subscription requirement An active RHEL subscription is required for **each instance** of the Glasswall Halo virtual appliance. #### System registration Each RHEL system must be registered with **Red Hat Subscription Management RHSM** using the customer’s own Red Hat account credentials. #### Updates and patches Only registered and subscribed systems are eligible to receive: * Critical security updates * Software patches * Official Red Hat support Unregistered or unlicensed systems may become vulnerable and are not eligible for support. ### Why this matters Red Hat’s subscription model is required to: * Ensure timely access to security updates and patches * Maintain system integrity and compliance * Access official Red Hat support services Operating without a valid subscription increases the risk of unpatched vulnerabilities and unsupported environments. ### Next steps * If you do not have a Red Hat account or subscription, visit the [Red Hat Customer Portal](https://access.redhat.com/) for information on purchasing and managing subscriptions. * For guidance on registering your RHEL system, refer to [Red Hat system registration documentation](https://docs.redhat.com/en/documentation/subscription_central/1-latest/html-single/getting_started_with_rhel_system_registration/index). Everything you need to install and begin using Halo as a Single Node (virtual appliance). ## System requirements Single node virtual appliances can be deployed as OVAs, compatible with VMware and VirtualBox, or as VHDs, compatible with Microsoft Hyper V. ### Minimum footprint - **CPU:** 1 GHz or faster with 16 or more virtual cores using an x86 64 architecture - **Memory:** 32 GB RAM recommended - **Storage:** 100 GB disk space for OVA or 500 GB disk space for VHD - **Usage guidance:** Halo with the ICAP server enabled is recommended only for light to moderate traffic volumes ### Comfortable footprint - **CPU:** 1 GHz or faster with 32 or more virtual cores using an x86 64 architecture - **Memory:** 64 GB RAM recommended - **Storage:** 200 GB disk space for OVA or 500 GB disk space for VHD - **Usage guidance:** This footprint is recommended when the ICAP server is enabled, particularly for protecting users browsing external networks or for workloads with variable demand ### With Foresight enabled To run [Foresight](/halo/deploying-foresight-single-node-vm) (AI-powered threat prediction), the appliance requires at least **24 CPU cores and 48 GB RAM**. ### Supported hypervisors The appliance is distributed in several image formats. Choose the format that matches your hypervisor: | Hypervisor | Image format | Minimum version | Deployment guide | | --- | --- | --- | --- | | **VMware ESXi** | OVA | ESXi 7.0.0 or later | [VMware deployment](/halo/vmware-deployment) | | **VirtualBox** | OVA | 7.0.8 or later | [VMware deployment](/halo/vmware-deployment) (OVA import) | | **Microsoft Hyper-V** | VHD | Windows Server 2019 / Windows 10 or later | [Hyper-V deployment](/halo/HyperV-deployment) | | **Proxmox VE** (KVM-based) | qcow2 | — | [Proxmox deployment](/halo/proxmox-deployment) | The OVA, VHD and qcow2 images are available in Kiteworks. #### KVM-based hypervisors (including RHEL KVM) A native **qcow2** image is provided for KVM-based platforms. qcow2 installs directly on KVM with no conversion: - For **Proxmox VE** (which is KVM-based), use the qcow2 image and follow the [Proxmox deployment guide](/halo/proxmox-deployment). - For **libvirt / QEMU-KVM** (for example, KVM on RHEL), attach the qcow2 image to a new VM that meets the requirements above. If you need a different image format or assistance with a specific KVM platform, contact [Glasswall Support](https://www.glasswall.com/support). ### Disk layout and provisioning The appliance image ships with a fixed virtual disk size (see **Storage** above). When deploying, keep the following in mind: - **Thin vs thick provisioning is chosen at import, not baked into the image.** Most hypervisors let you select the provisioning mode when you deploy the OVA/VHD. In VMware, for example, choose **Thin Provision** on the *Select storage* step of *Deploy OVF Template* if you want the disk to consume space on demand rather than allocating it all up front. - **Provision enough disk before first boot.** The simplest way to run with more capacity is to deploy onto a larger disk from the start, sized for your expected throughput and retention. - **Extending an existing appliance's disk** requires both growing the virtual disk at the hypervisor *and* extending the partition/filesystem inside the guest, which depends on the appliance's partition layout. If you need to resize an already-deployed appliance, contact [Glasswall Support](https://www.glasswall.com/support) for the supported procedure rather than repartitioning manually. ### Scaling #### Vertical scaling (scaling up) Halo can use additional virtual CPU cores and memory to support increased demand on the virtual appliance. - **Adding resources:** Vertical scaling involves increasing the resources of a single virtual appliance, such as adding CPU cores, memory, or storage. - **Single instance focus:** The workload is handled by a single, more powerful appliance rather than being distributed across multiple instances. - **Simpler management:** Managing one scaled up appliance is typically simpler, as there is no requirement for load balancing or state management across multiple nodes. - **Hardware constraints:** Vertical scaling is limited by the maximum CPU, memory, and storage supported by the underlying hardware or virtual environment. - **Reduced fault tolerance:** Vertical scaling does not inherently provide redundancy. If the appliance fails, there is no automatic failover unless a backup or high availability solution is in place. #### Horizontal scaling (scaling out) - **Adding instances:** Horizontal scaling increases capacity by deploying additional virtual appliances rather than increasing the size of a single instance. - **Load distribution:** A load balancer is typically used to distribute traffic evenly across appliances, preventing any single appliance from becoming a bottleneck. - **Fault tolerance and redundancy:** If one appliance fails, others can continue to handle traffic, improving availability and resilience. **Limitations:** While horizontal scaling is supported, each virtual appliance maintains its own database. Appliances operate independently, and content management policies and configuration settings cannot be managed centrally. **Licensing:** Licences must not be shared between virtual appliances. A separate licence is required for each virtual appliance. ### Image base OS - STIG hardened RHEL v9.4 (Sep 2024) ### Orchestration layer - RKE2 v1.28.10~rke2r1 (Sep 2024) The **Glasswall Halo Wizard** is a guided, terminal-based tool for configuring a freshly deployed Halo single-node virtual appliance. It walks you through the tasks you would otherwise perform by hand, including networking, TLS/SSL, API and Portal authentication, and the STIG banner, and applies each change to the on-appliance cluster for you. The wizard is the recommended way to configure a new appliance. The manual [Authentication](./3-Authentication.md) and [TLS/SSL configuration](./4-TLS-SSL-configuration.md) guides remain available for advanced or scripted deployments. :::note The wizard configures the appliance itself. It does **not** create the Azure Entra ID app registrations required for single sign-on (SSO); you create those beforehand (see [Authentication](./3-Authentication.md)) and supply the resulting values when the wizard asks for them. ::: ## What the wizard configures | Step | What it does | | --- | --- | | **Network** | Chooses whether the hypervisor (cloud-init) or the guest (`nmtui`) owns networking, and helps you set a static IP. | | **TLS/SSL** | Installs your certificate and key and enables HTTPS on the Portal and API. | | **API authentication** | Sets Basic, Bearer/SSO, both, or no authentication on the API. | | **Portal authentication** | Enables Azure Entra ID single sign-on for the Portal, or disables authentication. | | **STIG banner** | Enables or disables the DoD Message of the Day (MOTD) shown on SSH login. | ## Before you start - **SSH access** to the appliance. The default user on the Halo OVA is `glasswall`. - **Root privileges.** The wizard modifies system files and restarts services, so it must be run with `sudo`. - The on-appliance **RKE2 cluster must be running** (`kubectl` and `helm` are available on the appliance). - For **TLS**, a PEM-encoded X.509 certificate and its matching private key for your chosen domain. - For **SSO/Bearer**, the Azure Entra ID app registrations created in advance (see [Authentication](./3-Authentication.md)), and the Tenant ID, Client ID, audience and scope values they produce. ## Launching the wizard SSH into the appliance and run: ```sh sudo halotui ``` The main menu is displayed: ``` 🧙 Welcome to the Glasswall Halo™ setup wizard! Halo v2.19.0 · Wizard v1.0.0 [Recommended] ○ Run setup wizard (all steps) [Configure individual settings] ○ Network settings (hypervisor or guest) ○ TLS/SSL certificate ○ API authentication ○ Portal authentication ○ STIG banner (MOTD) ○ Licensing ○ Show version ○ Exit ``` You can either: - **Run setup wizard (all steps)**: steps through every configuration task in order (Network, then TLS/SSL, then API authentication, then Portal authentication, then STIG banner). Each step offers a **Skip** option so you can pass over anything you don't need. - **Configure individual settings**: jump straight to a single task. Use this to change one setting later without re-running the whole sequence. Navigate with the arrow keys and press **Enter** to select. Press **Ctrl+C** at any point to exit the wizard cleanly. :::tip Resume and reboot reminders If the setup wizard is interrupted, whether by a reboot, a dropped SSH session, or simply exiting, relaunching it and choosing **Run setup wizard** offers to **resume from the next unfinished step**. If a network change left a reboot outstanding, the wizard reminds you every time it starts until the appliance has actually rebooted. ::: ## Step 1 - Network The Network step decides who owns the appliance's networking: - **Hypervisor-managed (cloud-init)**: the default for a fresh OVA. Networking is owned by the platform's DHCP/cloud-init pipeline. Recommended for cloud and most on-premises deployments. - **Guest-managed (nmtui)**: networking is owned by NetworkManager and configured locally with `nmtui`. Choose this when you need to set a static IP inside the guest. The option matching the appliance's current mode is shown first, so pressing **Enter** is always the safe, least-surprising choice. - **Multiple interfaces:** if more than one connected interface is detected, you're asked to choose which one to configure first. - **Setting a static IP:** choose **Guest-managed**, then let the wizard launch `nmtui`. Inside `nmtui`, **Edit a connection**, set *IPv4 Configuration* to **Manual**, enter your address, gateway and DNS, then **Activate** the connection and quit. :::warning Cloud platforms On a detected cloud VM (Azure, AWS, Google Cloud, Oracle), guest-managed networking is withheld because the IP is owned by the cloud's network fabric, and switching can lose connectivity. Leave hypervisor-managed enabled and set a static/reserved IP at the cloud level (for Azure, the VM's NIC, under IP configurations). ::: :::note Reboot required After any network change, the appliance must **reboot** so the cluster picks up the new IP. The wizard sets a persistent *Reboot Required* flag and offers to reboot now. **The TLS/SSL step is blocked until this reboot has happened**, because the certificate binds to the appliance's final IP address. Note that `cloud-init clean` also resets the SSH host keys, so your SSH client may show a host-key-changed warning after the reboot. ::: ## Step 2 - TLS/SSL This step installs your certificate, creates a Kubernetes TLS secret and enables HTTPS on the Portal and API ingresses. TLS is a prerequisite for single sign-on, because identity providers only accept HTTPS redirect URIs. 1. If a TLS configuration already exists, the wizard asks whether to overwrite it. 2. Choose how to provide the certificate and key: - **Use files I have transferred (SCP)**: the wizard shows an `scp` command pre-filled with the appliance's IP and your home directory. Run it from another machine to copy `tls.crt` and `tls.key`, then press a key in the wizard to continue. - **Paste the certificate and key contents**: paste the certificate (or full chain) and press **Enter** on a blank line to finish, then paste the private key. 3. The wizard validates the certificate before applying anything: - It must be a valid, unexpired X.509 certificate with a usable domain (SAN or Common Name). - **The private key must match the certificate.** Mismatched, encrypted or unreadable keys are reported here, before any change is made. - If the certificate covers only a wildcard (for example, `*.cdr.example.com`), you're prompted for the concrete hostname clients will use. 4. The wizard creates the `tls-secret` secret and enables TLS on the Halo ingresses. The domain from your certificate is **remembered and reused** in the authentication steps, so you don't have to re-enter it. :::note If applying fails (for example, the cluster is briefly unreachable), the wizard explains the error and lets you **re-enter** the certificate, **retry** with the same files, or **cancel**. ::: ## Step 3 - API authentication Configures how the Halo API (`cdrplatform-api-access`) authenticates requests. Choose one of: - **Basic**: a username and password checked by the API. You enter the username and password (twice to confirm). - **Bearer/SSO**: JWT/SSO tokens from an identity provider (Azure Entra ID, Keycloak or Okta). **Requires TLS.** - **Basic and Bearer**: accept either. - **No authentication**: explicitly leaves the API open. Not recommended for production; you must confirm (the prompt defaults to *No*). For **Bearer/SSO**, the wizard asks for: - The **identity provider**: for Azure Entra ID you supply the **Tenant ID**; for Keycloak/Okta you supply the **authority (issuer URL)** directly. - The **valid audience** (defaults to `api://cdrplatform-api-access`). Each field's format is checked as you type. You can optionally **paste an access token** and the wizard validates its audience, issuer and expiry locally, with no call to the identity provider, so this works air-gapped. :::note Switching modes Applying a new mode clears the configuration the old mode used, so switching never leaves stale credentials behind. For example, after switching to Bearer/SSO only, the previous Basic username/password no longer exists in the cluster. ::: ## Step 4 - Portal authentication Configures authentication for the Halo web Portal. Two options: - **Configure Azure Entra ID single sign-on (SSO)**: users sign in with their Entra ID identity. **Requires TLS.** You provide: - **Tenant ID** (GUID): the authority and issuer default to the Microsoft endpoints for that tenant. - **Portal application (Client ID)** (GUID). - **Portal domain**: pre-filled from the domain captured during the TLS/SSL step; used to build the redirect URIs. - **Portal access audience** (defaults to `api://cdrplatform-portal-access`) and **scope** (defaults to `api://cdrplatform-portal-access/PortalUserScope`). As with API Bearer/SSO, you can optionally paste a token to validate it locally before applying. - **No authentication**: leaves the Portal open to unauthenticated users. Intended for demo/development only; you must confirm (the prompt defaults to *No*). This option does **not** require TLS. SSO spans two Helm releases, the Portal UI (`cdrplatform-portal`) and the Portal backend (`cdrplatform-portal-access`), which the wizard configures together. Once SSO is applied, open the Portal domain in a browser and click **Login with SSO**. ## Step 5 - STIG banner (MOTD) Enables or disables the DoD-required security warning shown when users connect over SSH. Enabling it is required for STIG compliance on government systems. 1. Choose **Enable** or **Disable**. 2. Review the banner preview (enable) or the compliance warning (disable), then confirm. The wizard writes (or removes) `/etc/motd`, restarts the SSH service and validates the result. To confirm, open a new SSH session; the banner appears before the login prompt. ## Other menu options **Licensing** displays the appliance's license information. ## Logging and diagnostics The wizard writes diagnostic logs (warnings and errors) to a file rather than the screen, so the terminal UI stays clean: ``` /tmp/cdrplatform-wizard.log ``` The file is appended to across runs, so you can follow it live in another terminal: ```sh tail -f /tmp/cdrplatform-wizard.log ``` When a step fails and something was recorded, the wizard points you at this file. When raising a support ticket for a failed step, attach `/tmp/cdrplatform-wizard.log` and note which step you were on. ## Troubleshooting | Symptom | What to check | | --- | --- | | **TLS/SSL step won't offer options, and asks you to reboot instead** | A network change left a reboot pending. Reboot the appliance, then reconfigure TLS/SSL. | | **Bearer/SSO shows "Requires TLS"** | Configure TLS/SSL (Step 2) first; single sign-on needs HTTPS redirect URIs. | | **"Private key does not match the certificate"** | The certificate and key weren't generated together, or the key is passphrase-protected. Supply the matching, decrypted key. | | **Token not accepted** | Paste a current token issued for the audience/tenant you entered, and confirm the authority and tenant match your app registration. | | **`kubectl`/`helm` errors, or `connection refused`** | The cluster must be running. The wizard auto-detects the RKE2 kubeconfig; for a non-standard layout, launch with `sudo KUBECONFIG=/etc/rancher/rke2/rke2.yaml halotui`. | | **"System not supported"** | The wizard runs only on the target Linux appliance, not from a Windows or macOS machine. | ## Prerequisites - A domain name must be configured for the Portal service to use SSO authentication. Identify the domain to be used for the Portal. - Identify the `tenant_id` for the Azure tenant. - Ensure the [Azure CLI](https://learn.microsoft.com/en-us/cli/azure/install-azure-cli) is installed on a local machine and sign in using `az login`. - Run the [provided shell script](/.attachments/create-azure-app-registrations.sh) to create three app registrations and corresponding enterprise applications. Record the script output, as it is required for subsequent steps. - `cdrplatform-api-access` - `cdrplatform-portal-access` - `cdrplatform-portal-client` ```sh bash create-azure-app-registrations.sh cleanroom.glasswall.com ``` By default the script creates all three registrations. To set up only one type of authentication, pass `--skip-api` to create just the Portal registrations, or `--skip-portal` to create just the API registration: ```sh # Portal SSO only (skips api-access) bash create-azure-app-registrations.sh --skip-api cleanroom.glasswall.com # API authentication only (skips portal-access and portal-client) bash create-azure-app-registrations.sh --skip-portal ``` - The enterprise application `ar-Halo-portal-client` needs to be granted admin consent. Prefer to create these registrations by hand? Follow [Creating the Azure app registrations manually](/halo/creating-azure-app-registrations-manually) for the equivalent steps in the Microsoft Entra admin center. :::note Applying authentication configuration briefly restarts the affected services (Portal, Portal Access, API Access). Complete these steps before the appliance is in production use, or during a maintenance window. ::: ## Portal authentication installation The Glasswall Halo Portal supports SSO authentication using Azure Entra ID. Follow the steps below to configure the integration. 1. SSH into the virtual machine to run the commands below. 2. The `cdrplatform-portal` and `cdrplatform-portal-access` Helm charts are located in the `/home/glasswall` directory. ```sh k get deploy portal -o json | jq -r '.spec.template.spec.containers[0].image' | cut -d":" -f2 ``` 3. Deploy portal with Azure AD configuration: ```sh tenant_id="" portal_domain="" portal_client_id="" portal_access_uri="" image_tag=$(kubectl get deploy portal -o json | jq -r '.spec.template.spec.containers[0].image' | cut -d":" -f2) enabled_pages="SystemSettings,PolicySettings" # if XML validation entitlement is enabled enabled_pages="SystemSettings,PolicySettings,ValidationSettings" # if ICAP server is enabled enabled_pages="SystemSettings,PolicySettings,IcapSettings,IcapRequests,IcapReporting" # if Storage monitoring is enabled enabled_pages="SystemSettings,PolicySettings,IcapSettings,IcapRequests,IcapReporting,SharePointMonitoring,OneDriveMonitoring" helm upgrade --install cdrplatform-portal cdrplatform-portal \ --set image.tag="${image_tag:?}" \ --set image.pullPolicy=IfNotPresent \ --set ingress.tls.enabled=true \ --set ingress.tls.domain=${portal_domain:?} \ --set ingress.tls.secretName=tls-secret \ --set cloud_provider=local \ --set resources.requests.cpu=500m \ --set resources.requests.memory=500Mi \ --set resources.limits.cpu=500m \ --set resources.limits.memory=500Mi \ --set securityContext.seccompProfile.type=RuntimeDefault \ --set configuration.BackendScope="${portal_access_uri}/PortalUserScope" \ --set configuration.BackendUrl="https://${portal_domain}" \ --set configuration.EnabledPages="${enabled_pages}" \ --set configuration.OIDC.ProviderOptions.Authority="https://login.microsoftonline.com/${tenant_id:?}/v2.0" \ --set configuration.OIDC.ProviderOptions.RedirectUri="https://${portal_domain}/authentication/login-callback" \ --set configuration.OIDC.ProviderOptions.ClientId="${portal_client_id}" \ --set configuration.OIDC.ProviderOptions.PostLogoutRedirectUri="https://${portal_domain}/authentication/logout-callback" \ --atomic ``` 4. Deploy portal access with Azure Entra ID configuration: ```sh tenant_id="" portal_domain="" portal_access_uri="" image_tag=$(kubectl get deploy portal-access -o json | jq -r '.spec.template.spec.containers[0].image' | cut -d":" -f2) helm upgrade --install cdrplatform-portal-access cdrplatform-portal-access \ --set image.tag="${image_tag:?}" \ --set image.pullPolicy=IfNotPresent \ --set ingress.tls.enabled=true \ --set ingress.tls.domain=${portal_domain:?} \ --set ingress.tls.secretName=tls-secret \ --set cloud_provider=local \ --set resources.requests.cpu=1 \ --set resources.requests.memory=2Gi \ --set resources.limits.cpu=1 \ --set resources.limits.memory=2Gi \ --set securityContext.seccompProfile.type=RuntimeDefault \ --set configuration.AuthenticationScheme=Bearer \ --set configuration.Authentication__Schemes__Bearer__ValidAudiences__0="${portal_access_uri}" \ --set configuration.Authentication__Schemes__Bearer__ValidIssuer=https://sts.windows.net/${tenant_id:?}/ \ --set configuration.Authentication__Schemes__Bearer__Authority=https://login.microsoftonline.com/${tenant_id:?}/v2.0/ \ --atomic ``` 5. Open the Portal domain in a browser and select **Login with SSO** in the lower left corner. 6. Follow the login process via Azure Entra ID and grant the app required permissions on behalf of the organisation for the first time. ## API Authentication API authentication can be configured in 2 ways - - Basic authentication - Bearer authentication ### Basic authentication installation 1. SSH into the virtual machine to run the commands below. 2. The `cdrplatform-api-access` Helm chart is located in the `/home/glasswall` directory. 3. Configure credentials in the cluster by setting a `username` and `password` using the commands below. Multiple passwords can be specified by separating them with commas. ```sh bash add_secrets.sh organisation0-id bash add_secrets.sh organisation0-tokens ``` 4. Deploy api-access with Basic authentication: ```sh image_tag="$(k get deploy api-access -o json | jq -r '.spec.template.spec.containers[0].image' | cut -d":" -f2)" enable_tls="true" or "false" api_domain="" # ignore if enable_tls is false helm upgrade --install cdrplatform-api-access cdrplatform-api-access \ --set image.tag="${image_tag:?}" \ --set image.pullPolicy=IfNotPresent \ --set ingress.tls.enabled="${enable_tls:?}" \ --set ingress.tls.domain="${api_domain}" \ --set configuration.CLIENTS__Policy__BaseAddress="http://policy-api:8080" \ --set configuration.CLIENTS__License__BaseAddress="http://license-management.license-management.svc.cluster.local:8080" \ --set configuration.CLIENTS__AsyncApi__BaseAddress="http://async-api:8080" \ --set configuration.AuthenticationScheme="Basic" \ --set cloud_provider=local \ --set resources.requests.cpu=1 \ --set resources.requests.memory=3Gi \ --set resources.limits.cpu=1 \ --set resources.limits.memory=3Gi \ --set securityContext.seccompProfile.type=RuntimeDefault \ --atomic ``` ### Bearer authentication installation 1. Identify the `tenant_id` for the Azure tenant. 2. SSH into the virtual machine to run the commands below. 3. Ensure the `cdrplatform-api-access` Helm chart is available in the `/home/glasswall` directory. 4. Deploy the API Access service with Azure Entra ID configuration: ```sh tenant_id="" api_valid_audience="api://cdrplatform-api-access" image_tag=$(kubectl get deploy api-access -o json | jq -r '.spec.template.spec.containers[0].image' | cut -d":" -f2) enable_tls="true" or "false" api_domain="" # ignore if enable_tls is false helm upgrade --install cdrplatform-api-access cdrplatform-api-access \ --set image.tag="${image_tag}" \ --set image.pullPolicy=IfNotPresent \ --set ingress.tls.enabled="${enable_tls}" \ --set ingress.tls.domain="${api_domain}" \ --set configuration.CLIENTS__Policy__BaseAddress="http://policy-api:8080" \ --set configuration.CLIENTS__License__BaseAddress="http://license-management.license-management.svc.cluster.local:8080" \ --set configuration.CLIENTS__AsyncApi__BaseAddress="http://async-api:8080" \ --set configuration.AuthenticationScheme="Bearer" \ --set configuration.Authentication__Schemes__Bearer__ValidAudiences__0="${api_valid_audience}" \ --set configuration.Authentication__Schemes__Bearer__ValidIssuer=https://sts.windows.net/${tenant_id:?}/ \ --set configuration.Authentication__Schemes__Bearer__Authority=https://login.microsoftonline.com/${tenant_id:?}/v2.0/ \ --set cloud_provider=local \ --set resources.requests.cpu=1 \ --set resources.requests.memory=3Gi \ --set resources.limits.cpu=1 \ --set resources.limits.memory=3Gi \ --set securityContext.seccompProfile.type=RuntimeDefault \ --atomic ``` Follow the steps below to enable TLS and SSL configuration. :::note Applying TLS configuration briefly restarts the affected services (API Access, Portal, Portal Access). Complete these steps before the appliance is in production use, or during a maintenance window. ::: ## Using SSL certificates To configure HTTPS access to the Glasswall Halo APIs using SSL certificates, complete the steps below. 1. Copy your `.key` and `.crt` files to the home directory of the virtual machine. 2. Run the `configure_tls.sh` script located in the same directory, passing the paths to the private key and certificate files, along with the domain name. >```sh >bash configure_tls.sh private_key_path certificate_path domain_name >``` >For example: >```sh >bash configure_tls my-key.key my-cert.crt mydomain.com >``` 3. If prompted to overwrite existing TLS secrets, enter `y` to continue. Glasswall also offers an additional Single Node native deployment of Glasswall Halo that enables virtualization on Microsoft Windows. Before you begin the process of deploying Glasswall Halo via Microsoft Hyper-V, check that you have the following tools and resources installed and setup. ## System requirements - Supported Hyper-V guest operating systems are listed [here](https://learn.microsoft.com/en-us/windows-server/virtualization/hyper-v/supported-windows-guest-operating-systems-for-hyper-v-on-windows). - Instructions for enabling Hyper-V on your system are available [here](https://learn.microsoft.com/en-us/virtualization/hyper-v-on-windows/quick-start/enable-hyper-v). - The CPU must support virtualization and virtualization must be enabled in the BIOS. More information is available [here](https://support.microsoft.com/en-us/windows/enable-virtualization-on-windows-c5578302-6e43-4b4b-a449-8ced115f58e1). - Create the virtual machine using the provided VHD with 16 CPU cores, 32 GB of memory, and 500 GB of disk space. - To run [Foresight](/halo/deploying-foresight-single-node-vm), provision the virtual machine with at least **24 CPU cores and 48 GB of memory**. **Note:** the following steps represent deployment on Hyper-V version: 10.1.02621.1. You can follow the steps below to complete the Microsoft Hyper-V deployment and setup, but please [contact us](https://www.glasswall.com/support) if you have any questions or require more support. **Note:** the following steps represent deployment via Hyper-V version: 10.0.42621.1 1. Launch *Hyper-V Manager* by searching via the search menu. 2. In the *Hyper-V Manager*, select **Actions** from the right hand menu then click **New** and then **Virtual machine**. This launches the *New virtual machine wizard*. 3. Click **Next >** to create a new custom configured virtual machine. 4. From the *Specify name and location* section, enter a **Name** for your new virtual machine and select a location where you want to store it (we left it as unchecked). **Note:** the location needs to have sufficient storage space for the virtual machine and its virtual hard disk. 5. Click **Next >**. 6. From the *Specify generation* section, select **Generation 1**. 7. From the *Assign memory* section, assign the sufficient **Startup memory** allocated to the virtual machine, suitable for Glasswall Halo. >**Note:** minimum: 32 GB (32768 MB) 8. Click **Next >**. 9. From the *Configure networking* section, select the appropriate **Connection**. 10. From the *Connect virtual hard disk* section, select the **Use an existing virtual hard disk** option and then browse to the location where you have saved your Glasswall Halo v2 VHD file. 11. Once you have selected the file, click **Next >**. 12. From the *Summary* section, review your settings and click **Finish** to complete the wizard setup. ## Launch virtual machine Before starting the machine make sure the VM has assigned 16 CPUs configured in order to be able to run the services In the Hyper-V Manager, under *Virtual machines*, your newly created *Glasswall Halo v2* virtual machine is displayed. 13. Right click the virtual machine and click **Start** or click **Start** from the right hand menu. 14. Your newly created virtual machine *state* should be *running* which indicates it's ready to connect. Right click and click **Connect** or click **Connect** from the right hand menu. 15. Login with your provided credentials. ## Password change 16. For security purposes, you are required to change the default password shared from Glasswall upon your first login. Once the shared credentials are added, add a new password for the virtual machine. **Note:** your new password should be at least 15 characters long, with a combination of uppercase letters, lowercase letters, numbers, and special characters. There also shouldn't be any repeating characters. ## Network manager configuration - For static IP: follow the Network Manager configuration guide to continue Halo setup. Return here when finished. - For DHCP: ensure your DHCP server is operational. ## Initiate Glasswall Halo services **Note:** if your network has a DHCP server, this step can be skipped as the RKE2 server will be started automatically by *cloud-init*. 1. Execute the following command to launch Glasswall Halo: >```shell >sudo bash start_halo.sh >``` 2. Confirm all of the Glasswall Halo services and pods are running with the following command: : >```shell >kubectl get pods >``` > **Note:** there might be some old pods in fail state present. This won't have any effect in the VMs functionality > ## Access Glasswall Halo Portal 1. Depending on your network configuration and/or domain configuration, you can now access the Halo Portal through the domain name and/or IP address assigned to your virtual machine. 2. Run the following command to confirm network status: >```sh >sudo netstat -tlnp >``` ## Access Glasswall Swagger API endpoints Navigate to the following to access the Glasswall Halo API endpoints: https://[**your domain or IP**]/swagger/index.html **Note:** don't forget to enter your own domain or IP address into the URL. ## Validating deployment You are able to validate the deployment by using our [API Health Check](/halo/glasswall-halo-health-checks) feature. ## Proxmox virtual environment deployment Before deploying Glasswall Halo via Proxmox, ensure you have the necessary tools and resources installed and set up. This deployment supports OVA, VMDK, backup, and disk file formats within Proxmox, along with a single-node native deployment option that enables virtualization on an open-source KVM. ### Proxmox VE - **Version**: 8.x or later. - Ensure the Proxmox VE host is completely up to date. ### VM configuration requirements - **BIOS**: **seaBIOS (legacy)** is recommended for compatibility with STIG-hardened RHEL images. - **Disk bus**: **IDE**, ensuring stable performance under RHEL-based Halo VMs. - **SCSI controller**: **Virtio SCSI**, recommended for efficient input/output. - **OS type**: Linux (l26: Linux 2.6/3.x/4.x/5.x). - **Display**: Spice (QXL). - **CPU**: at least **16 virtual cores** (32+ for heavy workloads). - **Memory**: **32 GB** minimum (64 GB recommended). - **Foresight**: to run [Foresight](/halo/deploying-foresight-single-node-vm), allow at least **24 virtual cores** and **48 GB** of memory. - **Storage**: **100–500 GB** of free space; SSD or RAID for optimal performance. - **Network**: a configured bridge in Proxmox (e.g., `vmbr0`) and either DHCP or static IP assignments. #### Example hardware setup - **Server model:** Dell PowerEdge R650 - **CPU:** 2 × Intel Xeon Silver 4310 - **Memory:** 128 GB RAM - **Storage:** - RAID 1 for boot - Additional RAID for virtual machines - **Networking:** 10 GbE and 1 GbE NICs To deploy **Glasswall Halo** in a Proxmox VM, you can choose from the following two options: - [Deploy using an **OVA/VMDK** file](proxmox-deployment#deploying-from-an-ovavmdk) - [Deploy from a **Glasswall backup** (`.vma.zst`)](proxmox-deployment#deploying-from-a-glasswall-backup-vmazst) Please refer to [troubleshooting & faqs](proxmox-deployment#troubleshooting-and-faqs) or the [Proxmox documentation](https://pve.proxmox.com/wiki/Main_Page) for further information. * * * ## Deploying from an OVA/VMDK ### Step 1 - Update Proxmox VE host Ensure your Proxmox host is up to date by running the following command to prevent compatibility issues. - Online connectivity update: >```bash >apt-get update >apt-get dist-upgrade >``` ### Step 2 - Create a new VM shell 1. Access the Proxmox web UI and click **Create Virtual Machine**. 2. Under the *General* tab, provide a VM name (e.g., `Glasswall-Halo`). 3. Under the *OS* tab, select a generic Linux type (e.g., `Linux 5.x/6.x`). You do not need an ISO. 4. **System**: - **BIOS**: set to **seaBIOS**. - **SCSI controller**: **VirtIO SCSI**. 5. **Hard disk**: remove the default disk (we’ll attach a converted disk later). 6. **CPU**: set type to *host* for maximum performance; allocate at least 16 cores. 7. **Memory**: allocate 32–64 GB. 8. **Network**: attach a NIC to the desired bridge (e.g., `vmbr0`), typically **VirtIO** model. 9. Under the *Confirm* tab, view the summary and click **Finish**. >**Note:** do not start the VM yet. ### Step 3 - Extract and transfer the OVA 1. Extract the OVA on your local system by running the following command: ```bash tar -xvf Halo-FULL-.ova ``` 2. Identify the `.vmdk` file (e.g., `Halo-full--disk1.vmdk`). 3. Upload the `.vmdk` to proxmox, e.g.: ```bash scp Halo-FULL-...disk1.vmdk root@:/root/ ``` ### Step 4 - Prepare storage (optional LVM-Thin) **Why LVM-Thin?** It provides thin provisioning and snapshot support, making it ideal for larger or frequently updated VM disks. However, you can also attach disks to directory-based or other storage types in Proxmox. **Example steps to create LVM-Thin**: ```bash # Identify a free disk, e.g. /dev/sdb pvcreate /dev/sdb vgcreate vg-halo /dev/sdb # Create a thin pool consuming all space lvcreate -l 100%FREE -T -n thin-halo vg-halo ``` After creating your thin pool, add it via **Datacenter -> Storage -> Add -> LVM-Thin**. Choose **vg-Halo** as the volume group and **Thin-Halo** as the thin pool. ### Step 5 - Convert and attach the Halo disk #### Option A: `qm importdisk` **SSH into your Proxmox host** If using storage that supports import (e.g., directory or LVM-Thin), run: ```bash qm importdisk /root/Halo-FULL-...disk1.vmdk --format qcow2 ``` - `` is the ID assigned to your VM (e.g., 100). - `` is the Proxmox storage target (e.g., `local-lvm`, `Halo-lvmthin`). - After completion, go to **vm -> hardware**, find the “unused disk,” then **edit** and select **ide0**. #### Option B (recommended): manual conversion (`qemu-img`) **SSH into your Proxmox host** If `qm importdisk` encounters errors (e.g., “zeroinit”), or if you prefer more direct management: 1. **Create a thin-provisioned volume** for the Halo disk: ```bash lvcreate -n vm--disk-0 -V 200G --thinpool thin-halo vg-halo ``` 2. **Convert the `.vmdk` to a raw disk**: ```bash qemu-img convert -f vmdk -O raw /root/Halo-FULL...disk1.vmdk /dev/vg-halo/vm--disk-0 ``` 3. **Attach disk to the VM**: ```bash qm set --ide0 halo-lvmthin:vm--disk-0 ``` ### Step 6 - Final VM configuration 1. In **VM -> Hardware**: - **IDE0** should be your main Halo disk. - Confirm “Virtio SCSI” is listed as the SCSI controller. 2. In **VM -> Options**: - Ensure **Boot order** sets IDE0 as first. - Verify *BIOS* is **seaBIOS**. - **KVM hardware virtualization** and **ACPI** should be enabled. 3. Optional: **QEMU guest agent** can be installed inside the Halo VM for advanced functionality. ### Step 7 - Initial boot 1. **Start the VM** in the Proxmox UI. 2. Watch the console; it may take up to a minute to pass “Probing EDD…” on first boot. 3. **Configure network** inside the Halo VM: - For static IP: follow the Network Manager configuration guide to continue Halo setup. Return here when finished. - For DHCP: ensure your DHCP server is operational. 4. **Finalize** application-level configurations (i.e. start_halo.sh, syslog server, tls). ## Deploying from a Glasswall backup (`.vma.zst`) If our Glasswall team has provided you with a pre-configured Proxmox backup, you can skip the OVA/VMDK steps. ### Step 1 - Uploading and restoring the backup 1. Copy the `.vma.zst` backup file to your Proxmox host’s backup directory: ```bash scp glasswall-halo-backup.vma.zst root@:/var/lib/vz/dump/ ``` 2. In the Proxmox UI, go to **Node -> Local -> Backup**. 3. Locate the `.vma.zst` file and select **Restore**. 4. Assign a new VM ID (e.g., 101) and target storage. ### Step 2 - Verifying disk attachment 1. After restore, open **VM -> Hardware**. 2. Confirm that the main disk is attached to **IDE0**. 3. Ensure BIOS is set to seaBIOS under **VM -> Options**. ### Step 3 - Starting the VM 1. **Power on** the VM. 2. Watch the console for any errors. 3. Configure the network settings, if not already set. 4. Proceed with any final configuration steps within Halo. * * * ## Troubleshooting and FAQs ### zeroinit error This may occur when using `qm importdisk` with certain storage types or older QEMU versions. >**Solution**: use manual conversion with `qemu-img`. ### Network configuration issues - If the VM interface is named differently (e.g., `ens18` vs. `eth0`), adjust your `ifcfg-` files accordingly. - Use `nmcli device show` or `ip a` to confirm the interface name. ### Why IDE + seaBIOS? - STIG-hardened RHEL images require legacy BIOS for certain boot configurations. - IDE ensures broad compatibility and avoids known issues with some virtual drivers. ### Additional best practices - **Snapshots**: leverage Proxmox snapshots or LVM-Thin snapshots for quick rollbacks. - **Backups**: regularly use `vzdump` to create compressed backups (`.vma.zst`). - **Resource monitoring**: keep an eye on CPU and memory usage for production loads. ### Initiate Glasswall Halo services - Execute the following command to launch Glasswall Halo: >```sh >sudo start_halo.sh >``` - Once the script has completed, confirm all the Glasswall Halo services and pods are running using the following command: >```shell >kubectl get pods >``` #### Access Glasswall Portal - Depending on your network configuration and/or domain configuration, you can now clean files by accessing the Glasswall portal through the domain name and/or ip address assigned to your virtual machine. - Run the following command to confirm network status: >```bash >sudo netstat -tlnp >``` Before you begin the process of deploying Glasswall Halo via VMware, check that you have the following tools and resources installed and setup. ## System requirements - Standalone ESXi host or VMware vSphere with ESXi hypervisor / vcenter - Glasswall Halo v2 OVA - OVA login credentials - This is provided by your Glasswall representative with the receipt of your OVA. ### Minimum hardware requirements * 16 Virtual CPUs (VCPUs) * 32 GB of memory * 100 GB of free storage * To run [Foresight](/halo/deploying-foresight-single-node-vm): at least **24 Virtual CPUs (vCPUs)** and **48 GB of memory** * Disk: * *Throughput*: we recommend using a disk with a high io throughput such as SSD (Solid State Disk), however a hard disk with capacity of 100 GB free storage is sufficient. * *Size*: the disk size depends on the average file size that will be processed. If you scale the cores and memory of the machine above the minimum requirements, the disk size should also be scaled by the same factor. * *Logging*: 40% of the disk should be allocated to */var/log/* where logs of all containers and services are stored. Logs should be retained for a configurable number of days (the default is 30 days). * *Minimum size for PVC*: the size of the PVC should be 50% of the total disk size which allows the remaining amount to be used by running services, logs etc. You can follow the steps below to complete the VMware deployment and setup, but please [contact us](https://www.glasswall.com/support) if you have any questions or require more support. 1. Login to your VMware ESXi console. 2. In the virtual machines page in ESXi select *Create / Register VM* to open the new virtual machine window. 3. Click **Select creation type**. 4. Select *Deploy a virtual machine from OVF or OVA file* and click **Next**. 5. Click **Select OVF and VMDK files**. - Enter a name for your Glasswall Halo VM - Browse and select the Glasswall Halo v2 OVA provided by Glasswall. Enter an appropriate name for the virtual machine. 6. Click **Select storage**. - Select your storage type and datastore 7. Click **Deployment options**. - Select the appropriate settings for the *network mappings and disk provisioning*. - Uncheck *Power on automatically* 8. Click **Finish** and wait for the import to complete. 9. For the best support and feature functionality it is suggested that the image be upgraded to the host client OS compatibility. - Select *Upgrade VM compatibility* under the **Actions** tab. - Select the appropriate minimum level of compatibility from the drop-down (in our example it is ESXi 6.7/8.0). **Note:** this process might negatively impact the performance of other client systems on the host, so it might be best to perform it in less critical Windows. 10. Please note the acknowledgement of potential performance impact and click **Yes**. #### Edit settings (virtual hardware) 11. Select **Edit settings** and from the *Virtual hardware* tab, modify: * **VCPU** to (minimum) 16 * **Memory** to 32 GB RAM * **Hard disk 1** to (minimum) 500 GB SSD/HD. #### Edit settings (VM options) 12A. From the *VM options* tab, select **General options** to open the menu and adjust the server settings as needed. **Note:** with the upgrade completed you can now select the correct *guest operating system*. * Select **Linux** as *Guest OS* and **Red Hat Enterprise Linux 9 (64-bit)** as the *Guest OS Version*. 12B. Click **Save**. ## Launch machine 13. Verify that the **Compatibility**, **CPU and **Memory allocation** are all correct. 14. Power on and launch the virtual machine. 15. Login with your provided credentials. ## Password change 16. For security reasons, you must change the default password provided by Glasswall on your first login. After signing in with the shared credentials, set a new password for the virtual machine. **Note:** your new password must be at least 15 characters long and include a combination of uppercase letters, lowercase letters, numbers, and special characters. Repeating characters are not permitted. ## Network manager configuration - For static IP: follow the Network Manager configuration guide to continue Halo setup. Return here when finished. - For DHCP: ensure your DHCP server is operational. ## Initiate Glasswall Halo services **Note:** if your network has a DHCP server, this step can be skipped as the RKE2 server will be started automatically by *cloud-init*. 1. Execute the following command to launch Glasswall Halo: >```sh >sudo bash start_halo.sh >``` 2. Once the script has completed, confirm all the Glasswall Halo services and pods are running using the following command: >```sh >kubectl get pods >``` ## Access Glasswall Halo 1. Depending on your network configuration and/or domain configuration, you can now access Halo through the domain name and/or IP address assigned to your virtual machine. 2. Run the following command to confirm network status: >``` >sudo netstat -tlnp >``` ## Access Glasswall Swagger API endpoints Navigate to the following URL to access the Glasswall Halo API endpoints: https://[**your domain or ip**]/swagger/index.html **Note:** don't forget to enter your own domain or IP address into the URL. ## Validating deployment You are able to validate the deployment by using our [API Health Check](/halo/glasswall-halo-health-checks) feature. By default, the DHCP client is set to active within the Single Node virtual appliance, and it will allocate an IP address when deployed to a network with an active DHCP server. However, we understand there are certain scenarios where disabling DHCP and configuring your system with a static IP address is required. ## Disable DHCP To disable DHCP within the Glasswall Single Node virtual appliance, follow the steps outlined below: 1. Create an additional configuration file in the following area path: /etc/cloud/cloud.cfg.d/ and append the following to it: network: config: disabled - Create new file to disable cloud-init from overwriting configuration: >```sh >sudo touch /etc/cloud/cloud.cfg.d/99_disable_net_config.cfg >``` - Append settings to your new cloud-init config file: >```sh >sudo echo "network: config: disabled" >> /etc/cloud/cloud.cfg.d/99_disable_net_config.cfg >``` - Or to manually edit the config file: >```sh >sudo vi /etc/cloud/cloud.cfg.d/99_disable_net_config.cfg >``` 2. Modify the network configuration and DHCP configurations by following the Network Manager configuration guide. - Clean cloud-init: >```sh >sudo cloud-init clean >``` - Reboot the system with: >```sh >sudo reboot now >``` 3. Log back in to the system and confirm your network settings: >```sh >ip route | grep default >(i.e. default via dev ens32 proto static metric 100 ) >``` 4. Confirm all Glasswall pods are operational >```shell >kubectl get pods -w >``` Glasswall Halo VM images are not configured with the United States Government (USG) banner login message that is part of the [DoD Security Technical Implementation Guides (STIGs)](https://public.cyber.mil/stigs/downloads/). This banner message (shown below), or any custom banner, can be added by inserting the banner text into the file `/etc/motd` on the VM. The SSH daemon will require a restart to propagate changes (e.g. rebooting the VM or restarting SSH daemon `sudo systemctl restart sshd`). ## USG STIG Login Banner Message ``` You are accessing a U.S. Government (USG) Information System (IS) that is provided for USG-authorized use only. By using this IS (which includes any device attached to this IS), you consent to the following conditions: -The USG routinely intercepts and monitors communications on this IS for purposes including, but not limited to, penetration testing, COMSEC monitoring, network operations and defense, personnel misconduct (PM), law enforcement (LE), and counterintelligence (CI) investigations. -At any time, the USG may inspect and seize data stored on this IS. -Communications using, or data stored on, this IS are not private, are subject to routine monitoring, interception, and search, and may be disclosed or used for any USG-authorized purpose. -This IS includes security measures (e.g., authentication and access controls) to protect USG interests--not for your personal benefit or privacy. -Notwithstanding the above, using this IS does not constitute consent to PM, LE or CI investigative searching or monitoring of the content of privileged communications, or work product, related to personal representation or services by attorneys, psychotherapists, or clergy, and their assistants. Such communications and work product are private and confidential. See User Agreement for details. ``` ## Deployment - Deploy Glasswall Halo (with *syslog-ng*) from OVA/VHD (following the standard instructions). - Allow incoming traffic from Glasswall Halo VM to the syslog server port's (usually TCP 514) in the security group. ## *syslog-ng* service configuration on Glasswall Halo VMs Glasswall Halo OVA/VHD has the option to install and configure *syslog-ng* service. Once the Glasswall Halo instance is up and running, follow the steps below: 1. Make sure the Glasswall Halo VM is registered with RHEL, so that packages can be installed from internet. 2. Start the installation and configuration of *syslog-ng* script with: >```sh >sudo bash ~/syslog-ng-install.sh >``` 3. Once completed, run the next steps below: >```sh >sudo bash ~/syslog_setup.sh >Enter IP address of Syslog server: >Enter Port of Syslog server: >Configuring Syslog server details..... >.... >``` 4. From this point on all logs will be forwarded to the Syslog server and stored in the relevant Syslog server log file configured to receive remote logs. This includes: - Glasswall Halo VM logs (stored in /mnt/logging_data) - All container logs: (stored in/var/log/containers) - Pods logs: (stored in /var/log/pods) 5. Verify connection to the syslog server with the following CLI command: >```sh >$ loggen -i -S -P >count=1853, rate = 951.87 msg/sec >count=2329, rate = 951.65 msg/sec... >``` ### *syslog-ng* TLS configuration on Glasswall Halo VMs In case the remote Syslog server uses TLS transport, the following configuration changes need to be made on the Glasswall Halo machine. 1. Place the server certificate's root ca at a location in the Glasswall Halo machine (e.g. at /opt/syslog-ng/etc/syslog-ng/ca.d). 2. In the *syslog-ng* configuration file at (/etc/syslog-ng/syslog-ng.conf), edit line number 42-0 From: destination remote tcp("**.**.**.**" port(**));; 3. To: destination remote network("**.**.**.**" port(**) Transport("tls") Tls( ca-dir("/opt/syslog-ng/etc/syslog-ng/ca.d") Peer-verify(optional-untrusted) ) ); ; 4. Where: >- Port should be the TLS port of the remote syslog server. >- Peer-verify should be optional-untrusted for untrusted/self-signed CAs or required-trusted for trusted CAs. >- Restart *syslog-ng* with the commands below: >```sh >sudo systemctl restart syslog-ng >sudo systemctl restart gwsyslog.service >``` Glasswall *syslog-ng* on the Glasswall Halo machine should now be connected to the remote syslog server and sending secure logs on the TLS port. :::note If your network has a DHCP server, the virtual machine will be assigned an IP address automatically and you can safely skip this guide. ::: This guide walks through using NMTUI (Network Manager Text User Interface) to assign a static IP to your Single Node VM. **Per your network defined configurations and boundaries** 1. Before configuring the virtual machine's network, you need to know the MAC address of the ethernet the machine is running. 2. Run `nmcli` in the console to determine which network the virtual machine is using. Make a note of this. 3. Once you know which network to edit, run `nmtui` to launch the Network Manager and select **Edit a connection**. 4. Based on your configuration requirements: * Either **Add** network adapter or **Edit** connection. * In our example we edit the connection the virtual machine is using. 5. Select the option next to `IPv4 CONFIGURATION`; in our example, it is **Automatic**. Set this to **Manual** using the dropdown menu that appears. 6. Select **Show** for the IPv4 configuration. This will provide options to edit IP addresses, gateway and DNS servers. 7. Enter the required information, then scroll to the bottom and click **OK**. 8. Navigate back to the Network Manager main screen, then select **Activate a connection** and ensure your connection is set to *Activated*. 9. Exit the Network Manager and run the following command: >```sh >sudo systemctl restart NetworkManager >``` 10. Continue following the appropriate Single Node VM deployment page. ## Glasswall [ICAP Server](/halo/about-glasswall-icap) - The ICAP Server in the OVA is configured with 31344 as an open port and MTLS on port 31345. This is because the default 1344 and 11344 ports do not fall under the NodePort range of 30000 to 32767 in Kubernetes. - The ICAP server is disabled by default, but you can follow the steps below to enable it. ## Enable ICAP Server ```sh bash ~/enable_icap_server.sh ``` ## Enable mTLS By default, mTLS is not configured, and only the open port is active. To enable mTLS on the ICAP server, follow the steps below. 1. Create a set of client and server mTLS certificates along with a CA certificate. The server certificates will be used in the ICAP server and the client certificates will be used in a client application such as a proxy or a firewall. 2. SSH to the Halo Virtual Machine created from the OVA. 3. Save the `cdrplatform-secrets` Kubernetes secret to a file. >```bash >kubectl get secret cdrplatform-secrets -o yaml > cdrp-secrets.yaml >``` 4. Edit the `cdrp-secrets.yaml` file and add 3 keys, `tls-cafile`, `tls-server-cert` and `tls-server-key` under the `data` section. >```yaml >apiVersion: v1 >data: > > > tls-cafile: > tls-server-cert: > tls-server-key: >kind: Secret >``` 5. Apply the edited `cdrp-secrets.yaml`. >```sh >kubectl apply -f cdrp-secrets.yaml >``` 6. Create `acr-secret` to authenticate to glasswallhub container registry. The token_ID and token will be provided by the Glasswall. >```bash >kubectl create secret docker-registry acr-secret -n cdrplatform \ > --docker-server="glasswallhub.azurecr.io" --docker-username="${token_ID}" --docker-password="${token}" >``` 7. Pull the ICAP Server Helm chart: >```bash >helm pull oci://glasswallhub.azurecr.io/helm/icap-server --version 1.5.4 --untar --untardir . >``` 8. Redeploy ICAP Server with mTLS enabled: >```bash >helm upgrade --install icap-server icap-server --atomic \ > --set service.type=NodePort \ > --set cloud_provider=local \ > --set image.tag="2.21.0-201948" \ > --set image.repository=glasswallhub.azurecr.io/icap-server \ > --set image.pullPolicy=IfNotPresent \ > --set secretsFromRef[3].DATABASE__ConnectionString.secretName="mongodb-cdrplatform-cdrp-user" \ > --set secretsFromRef[3].DATABASE__ConnectionString.key="connectionString.standard" \ > --set configuration.ENABLE_MTLS=yes \ > --set configuration.HALO__Version="2.21.0" \ > -f icap-server/values.yaml >``` >Once the above command is successful, the ICAP server will listen on mTLS port 31345 (in OVA) along with the open port (31344). >When connecting to the ICAP Server through a secure connection the URI Scheme submitted in the ICAP Client request should be `icaps`. This indicates that a secure connection is expected. Example Secure-ICAP url: >```sh >icaps://gw-icap-server.net:31345/resp-cdr-service >``` ## Disable ICAP Server To disable the ICAP server, please run the command below: ```sh bash ~/disable_icap_server.sh ``` ReversingLabs file reputation is configured at runtime from the Halo **Portal** or the Halo **API**. This is the supported path on the OVA and other self-hosted appliances, where the appliance owns `cdrplatform-secrets` and the Engine writes the settings directly. A change is appliance-wide and takes effect without a restart. ## Configure from the Portal 1. Sign in to the Portal as an administrator. 2. Open **Settings** and select the **File reputation** tab. 3. Enter the ReversingLabs **username** and **password**, and optionally a **timeout** in seconds (1–120, default 5). Halo hard-codes the **endpoint** (the same for every appliance); the tab shows it read-only with a padlock icon. 4. Select **Test connection** to verify the endpoint and credentials. 5. Turn ReversingLabs **on** and **Save**. Enabling requires a passing test. ## Configure from the Halo API The same operations are available on the Halo API (administrator; `basic` or `bearer` auth): - `GET /api/v1/reversing-labs/settings` — read the current settings. The password is never returned. - `PUT /api/v1/reversing-labs/settings` — save the settings. Enabling is gated on a passing connection test; leave the password blank to keep the stored one. - `POST /api/v1/reversing-labs/test-connection` — test a candidate endpoint and credentials without saving. Storage monitoring is a powerful service that enables automatic sanitization of supported file types at the point of upload. It can continuously monitor a wide range of cloud storage resources and sanitize files in real time. Follow these steps to enable OVA integration with [Storage Monitoring](/halo/storage-monitoring-overview). :::note Enabling Storage Monitoring briefly restarts the Storage Monitor and Portal services. Complete these steps before the appliance is in production use, or during a maintenance window. ::: ## Prerequisites Before you begin, ensure the following requirements are met: - **Domain Name**: configure a domain name for the Storage Monitoring service with a public IP address. Ensure the [SSL certificates](/halo/tls-ssl-configuration) are properly configured for the selected domain. - **Azure Tenant ID**: identify the `tenant_id` for the Azure tenant to be used. - **jq**: install [jq](https://jqlang.github.io/jq/download/) on your machine if not already present: ```sh sudo dnf install -y jq ``` - **Azure CLI**: install the [Azure CLI](https://learn.microsoft.com/en-us/cli/azure/install-azure-cli) and authenticate using: ```sh az login ``` - App Registration Setup: run the provided shell script to create an App registration and an Enterprise application. - **Note:** keep a record of the output values as they’ll be required in later steps. >```sh >bash create-azure-app-registrations-storage-monitor.sh >``` ## Installing the storage monitor The OVA includes the storage monitor Helm chart and container image, pre-installed with 0 replicas. SSH into the VM and complete the following steps to enable it. ### 1. Create Kubernetes secrets The storage monitoring service requires the **tenant ID**, **client ID**, and **client secret** for the service principal. **Important:** - Provide values in plain text, the script will automatically handle Base64 encoding. - Re-running the script with an existing key updates the value stored in the Kubernetes secret. Execute the script as follows: ```sh bash add_secrets.sh graphapi-clientid bash add_secrets.sh graphapi-clientsecret bash add_secrets.sh graphapi-tenantid ``` ### 2. Enable storage monitoring Run the `enable_storage_monitor.sh` script with your domain name and Azure tenant name. This will start the storage monitor service and enable the SharePointMonitoring and OneDriveMonitoring pages in the Halo portal. ```sh bash enable_storage_monitor.sh ``` For example: ```sh bash enable_storage_monitor.sh sm.example.com Contoso ``` ### 3. Post-installation Once the installation completes successfully, follow the instructions in [Storage Monitoring — Microsoft SharePoint](/halo/storage-monitoring-sharepoint) to configure monitors for Microsoft SharePoint, OneDrive, and Outlook. ## Glasswall Halo VM troubleshooting ### I have created a new VM using an OVA or VHD, however the number of Engine pods is not running as expected. The cloud-init configuration on the virtual machine runs a shell script that configures the number of Engine pods based on the available CPU cores. If the cloud-init script does not complete within the expected time, run the following commands to trigger a retry: ```sh sudo cloud-init clean sudo reboot now ``` ## Upgrading Glasswall OVA Please follow the steps below to upgrade your Glasswall OVA: 1. Backup MongoDB data and create a *mongodump*. ```sh kubectl port-forward svc/mongo-svc 27017 ``` 2. Open another terminal on the VM: ```sh username=$(kubectl get secret mongodb-cdrplatform-cdrp-user -o json | jq -r '.data.username' | base64 -d) password=$(kubectl get secret mongodb-cdrplatform-cdrp-user -o json | jq -r '.data.password' | base64 -d) mongodump -h localhost -p 27017 --db cdrplatform --out halo_backup -u "${username:?}" -p "${password:?}" zip -r halo_backup.zip halo_backup ``` 3. Copy the backup zip file from the VM to a local machine. ```sh scp glasswall@:~/halo_backup.zip halo_backup.zip ``` 4. Follow the steps in the deployment page based on [VMWare](/halo/vmware-deployment) or [Hyper-V](/halo/HyperV-deployment) to create a new VM using the new OVA/VHD. 5. Copy the `halo_backup.zip` from the local machine to the vVM. ```sh scp halo_backup.zip glasswall@:~/ ``` 6. Restore the data in the new VM. ```sh kubectl port-forward svc/mongo-svc 27017 ``` 7. Open another terminal on the VM: ```sh unzip halo_backup.zip username=$(kubectl get secret mongodb-cdrplatform-cdrp-user -o json | jq -r '.data.username' | base64 -d) password=$(kubectl get secret mongodb-cdrplatform-cdrp-user -o json | jq -r '.data.password' | base64 -d) mongorestore --host localhost --port 27017 -u "${username:?}" -p "${password:?}" --authenticationMechanism=SCRAM-SHA-256 --authenticationDatabase=cdrplatform halo_backup ``` 8. Clean up the backup files after the restore: ```sh rm -rf halo_backup halo_backup.zip ``` ## ReversingLabs after upgrade After upgrading, ReversingLabs is **disabled** until you reconfigure it. To restore it, follow [Enable ReversingLabs](/halo/enable-reversing-labs): open the **File reputation** tab in the Portal (or use the Halo API), re-enter the endpoint and credentials, run **Test connection**, then turn it on and save. ## Deployment options ### Cloud service provider managed Kubernetes - Deploys Halo artefacts including Helm charts and container images into a Customer managed Kubernetes cluster hosted in public, private, or government cloud environments such as Azure, AWS, or Oracle. - Uses native cloud PaaS and SaaS services for storage and key or secret management and is validated internally against each supported platform. - Customers typically pull Halo artefacts directly from the Glasswall container registry at `glasswallhub.azurecr.io`. - Artefacts are preconfigured for each supported cloud platform. - Installation guidance is available in the [Glasswall documentation](/halo/glasswall-halo-deployment-overview). --- ### On prem customer managed Kubernetes - Deploys Halo artefacts into a Customer managed Kubernetes cluster hosted in an on prem environment such as Rancher Enterprise, VMware Tanzu, or Red Hat OpenShift. - Requires integration with Customer managed storage and key or secret management services. - Customers typically pull artefacts from `glasswallhub.azurecr.io`. - Some environments may require additional configuration. - General deployment guidance is available in the [Glasswall documentation](/halo/glasswall-halo-deployment-overview). - Due to platform variability, professional services support is commonly required during deployment. --- ### On prem customer managed virtual machine single node - Deploys a Halo single node VHD or OVA into a Customer virtualisation environment such as VMware. - Artefacts are typically downloaded via Kiteworks. - Delivered preconfigured, with scalability limited by single virtual machine resources. - Scaling or high availability requires Customer managed networking such as load balancing. - Installation guidance is available in the [Glasswall documentation](/halo/glasswall-halo-single-node-vm-overview). - While simpler than Kubernetes based deployments, targeted support is often required during installation. --- ## Shared responsibility matrix | Responsibility | Cloud service provider managed Kubernetes | On prem Customer managed Kubernetes | On prem Customer managed virtual machine | | --- | --- | --- | --- | | Access to deployment assets | Glasswall | Glasswall | Glasswall | | CDR functionality | Glasswall | Glasswall | Glasswall | | Technical and compliance documentation | Glasswall | Glasswall | Glasswall | | Application logging accuracy and guidance | Glasswall | Glasswall | Glasswall | | Helm chart configuration and testing | Glasswall | Glasswall and Customer | Glasswall | | Deployment documentation and scripts | Glasswall | Customer | Glasswall | | Infrastructure integration guidance and scripts | Glasswall | Customer | Customer | | Integration with business applications | Customer | Customer | Customer | | Identity providers, credentials, and cluster integration | Customer | Customer | Customer | | External network integration including TLS and security | Customer | Customer | Customer | | External storage configuration and security | CSP and Customer | Customer | Customer | | Infrastructure and cluster monitoring and logging | CSP and Customer | Customer | Customer | | Infrastructure access, resiliency, and availability | CSP | Customer | Customer | | Host operating system security and hardening | CSP and Customer | Customer | Glasswall | Glasswall Artifact Registry provides secure access to container images and Helm charts that can be pulled directly into your cluster during deployment. You are issued a token and token ID to authenticate access to the registry, which are used as part of the installation process. ## Available repositories | Container repositories | Helm repositories | |------------------------|-------------------| | Cdrplatform Engine eval | Helm cdrplatform RabbitMQ | | Cdrplatform Engine (licensed Customers only) | Helm cdrplatform External secrets | | Cdrplatform Report Aggregator | Helm cdrplatform Storage | | Cdrplatform Sync API | Helm cdrplatform Engine | | | Helm cdrplatform Sync API | | | Helm cdrplatform Report aggregator | Several services include configurable settings that can be customized. Below are the configuration values available for each service. ## How to update configuration The configuration values can be changed by setting helm chart values while deploying the helm chart using `--set configuration.=`. For example, to set `ASPNETCORE_SHUTDOWNTIMEOUTSECONDS` value to 90 seconds and set `RATELIMITING__MaxMessageCount` value to 300 messages, the helm command looks like below - ``` helm upgrade --install cdrplatform-sync-api cdrplatform-sync-api \ --set configuration.ASPNETCORE_SHUTDOWNTIMEOUTSECONDS=90 \ --set configuration.RATELIMITING__MaxMessageCount=300 ``` ## API Access | Configuration Key | Description | Valid Values | |--|--|--| | ASPNETCORE_SHUTDOWNTIMEOUTSECONDS | Specifies the amount of time to wait for Web Host to shut down. | Any valid integer | | AuthenticationScheme | Authentication Scheme for the API Access. | None, Bearer, Basic | | Authentication__Schemes__Bearer__ValidAudiences__0 | Valid audience for API Access when AuthenticationScheme is set to Bearer| Any valid string ( e.g. api://cdrplatform-api-access) | | Authentication__Schemes__Bearer__ValidIssuer | Valid Issuer when AuthenticationScheme is set to Bearer | `https://sts.windows.net/\/` | | Authentication__Schemes__Bearer__Authority | Authority for the API Access when AuthenticationScheme is set to Bearer | `https://login.microsoftonline.com/\/v2.0/` | | CLIENTS__Policy__BaseAddress | The base URL the proxy Policy Management API requests to | ```http://policy-api:8080``` | | CLIENTS__License__BaseAddress | The base URL the proxy License Management API requests to| ```http://license-management.license-management.svc.cluster.local:8080``` | | CLIENTS__SyncApi__BaseAddress | The base URL the proxy Sync API requests to | ```http://api:8080``` | | CLIENTS__AsyncApi__BaseAddress | The base URL the proxy Async API requests to | ```http://async-api:8080``` | | CLIENTS__IcapProfile__BaseAddress | The base URL to proxy ICAP Profile Management requests to | ```http://policy-api:8080``` | ## Sync API | Configuration Key | Description | Valid Values | |--|--|--| | ASPNETCORE_SHUTDOWNTIMEOUTSECONDS | Specifies the amount of time to wait for Web Host to shut down. | Any valid integer | | RATELIMITING__MaxMessageCount | The max number of messages allowed on the request queue before rate limiting kicks in | Any valid integer | | ARCHIVE__MaxLevel | Maximum layers of nested archives that will be processed [more info](/halo/glasswall-halo-archive-support) | Positive integer | | ARCHIVE__MaxFileCount | Maximum file count allowed in an archive before failure [more info](/halo/glasswall-halo-archive-support) | Positive integer | | ARCHIVE__MaxArchiveCount | Maximum nested archive count allowed in an archive before failure [more info](/halo/glasswall-halo-archive-support) | Positive integer | | ARCHIVE__MaxUnpackedSizeBytes | Maximum allowed size of unpacked files from an archive before failure [more info](/halo/glasswall-halo-archive-support) | Positive integer (in bytes) | ## Engine | Configuration Key | Description | Valid Values | |--|--|--| | DOTNET_SHUTDOWNTIMEOUTSECONDS | Specifies the amount of time to wait for Host to shut down. | Any valid integer | | QUEUE__RetryLimit | Specifies the amount of times to retry Async requests | Any valid integer | | foresight.enabled | Enables the Foresight AI threat prediction sidecar that runs alongside the engine. Requires the Foresight license entitlement. Set via `--set foresight.enabled=true`. | `true`, `false` (Defaults to `false`) | ReversingLabs file reputation is not configured through this table. It is set at runtime — see [Enable ReversingLabs](/halo/enable-reversing-labs). ## Portal Access | Configuration Key | Description | Valid Values | |--|--|--| | AuthenticationScheme | Authentication Scheme for the Portal Access API | None, Bearer | | Authentication__Schemes__Bearer__ValidAudiences__0 | Valid audience for Portal Access API. Set to the Application (client) ID of the `cdrplatform-portal-access` app registration. | Any valid string | | Authentication__Schemes__Bearer__ValidIssuer | Valid issuer for the Portal Access API | `https://sts.windows.net/\/` | | Authentication__Schemes__Bearer__Authority | Authority for the Portal Access API | `https://login.microsoftonline.com/\/v2.0/` | | REBUILD__RequireAuthenticatedUser | Determines whether users must be authenticated to perform rebuild requests. Defaults to false. | true/false | ## Portal | Configuration Key | Description | Valid Values | |--|--|--| | BackendUrl | Domain of the CDR Platform API | https://\ (A valid string) | | OIDC.ProviderOptions.Authority | Authority for the Portal service | `https://login.microsoftonline.com/\/v2.0` | | OIDC.ProviderOptions.ClientId | Client ID of the Portal App registration (cdrplatform-portal-client) | A valid string | | OIDC.ProviderOptions.RedirectUri | Redirect URI after SSO Login | `https://\/authentication/login-callback` | | OIDC.ProviderOptions.PostLogoutRedirectUri | Redirect URI after Logout | `https://\/authentication/logout-callback` | | REBUILD.RequireAuthenticatedUser | Determines whether users must be authenticated access the "Clean a file" page. Defaults to false. | true/false | ## License Management | Configuration Key | Description | Valid Values | |--|--|--| | ASPNETCORE_SHUTDOWNTIMEOUTSECONDS | Specifies the amount of time to wait for Web Host to shut down. | Any valid integer | | DATABASE__Provider | The database provider used for caching purposes | Mongo / Cosmos | | DATABASE__DatabaseName | The name of the database which will be created in Mongo / Cosmos | A valid string depending on provider | | | DATABASE__ConnectionString | Connection string to the Mongo or Cosmos database | A valid connection string for the configured provider | ## Cleanup | Configuration Key | Description | Valid Values | |--|--|--| | DOTNET_SHUTDOWNTIMEOUTSECONDS | Specifies the amount of time to wait for Host to shut down. | Any valid integer | | CleanupAmount | Specifies the amount of files to clean up per cron job. This can be set via `--set cron.CleanupAmount=5000`| Any valid integer | | maxAge | Specifies the max age of files to keep in storage. This can be set via `--set cron.maxAge=01.00:00:00` | Any valid Timespan | | schedule | Specifies the cron schedule for the cleanup cron job to run on. This can be set via `--set cron.schedule=0 */1 * * *` | Any valid cron schedule expression | ## ICAP | Configuration Key | Description | Valid Values | |--|--|--| | ASPNETCORE_SHUTDOWNTIMEOUTSECONDS | Specifies the amount of time to wait for Host to shut down. | Any valid integer | | ICAP__ServiceHeader | An Identifier that gets inserted into ICAP headers. | Any valid string, defaults to 'Glasswall ICAP Server 1.0"' | | ICAP__OptionsTTL | The amount of time in seconds which an ICAP options response sent by the server is valid to the icap client. | Any valid integer that the icap client supports | | ICAP__IdleTimeout | Defaults to infinite. Sets a deadline indicating how long the client must take before the server will end the connection | A valid timespan e.g 00:00:30 | | CACHE__MaxSizeInMb | The amount of data in megabytes which the ICAP server will store inside its cache for rebuilt files | Defaults to 1Gb "1000" | | DATABASE__Provider | The database provider used for caching purposes | Mongo / Cosmos | | DATABASE__DatabaseName | The name of the database which will be created in Mongo / Cosmos | A valid string depending on provider | | | DATABASE__ConnectionString | Connection string to the Mongo or Cosmos database | A valid connection string for the configured provider | | CERTIFICATE__VerificationFlags | Flags used to customize certificate chain verification in the ICAP server. Please see [X509VerificationFlags](https://learn.microsoft.com/en-us/dotnet/api/system.security.cryptography.x509certificates.x509verificationflags?view=net-8.0) for an explanation of the verification flags. | A valid integer within the enum range | | PROFILE__UseFallback | Boolean flag that indicates whether or not to use a fallback profile when the Policy API cannot be contacted. | 'true' to enable the fallback profile or 'false' to disable it. By default, this is 'false' | ## Async API | Configuration Key | Description | Valid Values | |--|--|--| | ASPNETCORE_SHUTDOWNTIMEOUTSECONDS | Specifies the amount of time to wait for Web Host to shut down. | Any valid integer | | ARCHIVE__MaxLevel | Maximum layers of nested archives that will be processed [more info](/halo/glasswall-halo-archive-support) | Positive integer | | ARCHIVE__MaxFileCount | Maximum file count allowed in an archive before failure [more info](/halo/glasswall-halo-archive-support) | Positive integer | | ARCHIVE__MaxArchiveCount | Maximum nested archive count allowed in an archive before failure [more info](/halo/glasswall-halo-archive-support) | Positive integer | | ARCHIVE__MaxUnpackedSizeBytes | Maximum allowed size of unpacked files from an archive before failure [more info](/halo/glasswall-halo-archive-support) | Positive integer (in bytes) | | DATABASE__Provider | Specifies which Database provider to use when storing Async Requests | `Mongo`, `Cosmos` (Defaults to `Mongo`) | | DATABASE__DatabaseName | The name of the database which will be created in Mongo / Cosmos | A valid string depending on provider | | | DATABASE__ConnectionString | Connection string to the Mongo or Cosmos database | A valid connection string for the configured provider | | QUEUE__MessageDelayInMs | Time in Milliseconds before the result of an ASYNC request is cleaned up. This should be set lower than the maxAge Timespan of the Cleanup service. | Any valid positive integer. The maximum value is `(2^32)-1 milliseconds` which is just under 50 days. | ## Tally Accumulator | Configuration Key | Description | Valid Values | |--|--|--| | DATABASE__Provider | Specifies which Database provider to use when storing Async Requests | `Mongo`, `Cosmos` (Defaults to `Mongo`) | | DATABASE__DatabaseName | The name of the database which will be created in Mongo / Cosmos | A valid string depending on provider | | | DATABASE__ConnectionString | Connection string to the Mongo or Cosmos database | A valid connection string for the configured provider | ## Policy API | Configuration Key | Description | Valid Values | |--|--|--| | DATABASE__Provider | Specifies which Database provider to use when storing Async Requests | `Mongo`, `Cosmos` (Defaults to `Mongo`) | | DATABASE__DatabaseName | The name of the database which will be created in Mongo / Cosmos | A valid string depending on provider | | | DATABASE__ConnectionString | Connection string to the Mongo or Cosmos database | A valid connection string for the configured provider | ## Metrics Collation (Deprecated) > As of version 2.6.2 of Halo, the Metrics Collation service is no longer required. | Configuration Key | Description | Valid Values | |--|--|--| | DATABASE__Provider | Specifies which Database provider to use when storing Async Requests | `Mongo`, `Cosmos` (Defaults to `Mongo`) | | DATABASE__DatabaseName | The name of the database which will be created in Mongo / Cosmos | A valid string depending on provider | | | DATABASE__ConnectionString | Connection string to the Mongo or Cosmos database | A valid connection string for the configured provider | ## Metrics Projection | Configuration Key | Description | Valid Values | |--|--|--| | DATABASE__Provider | Specifies which Database provider to use when storing Async Requests | `Mongo`, `Cosmos` (Defaults to `Mongo`) | | DATABASE__DatabaseName | The name of the database which will be created in Mongo / Cosmos | A valid string depending on provider | | | DATABASE__ConnectionString | Connection string to the Mongo or Cosmos database | A valid connection string for the configured provider | ## Storage Monitor | Configuration Key | Description | Valid Values | |--|--|--| | DATABASE__Provider | Specifies which Database provider to use when storing monitor data | `Mongo`, `Cosmos` (Defaults to `Mongo`) | | DATABASE__DatabaseName | The name of the database which will be created in Mongo / Cosmos | A valid string depending on provider | | DATABASE__ConnectionString | Connection string to the Mongo or Cosmos database | A valid connection string for the configured provider | | GRAPHAPI__ClientId | Client ID for Microsoft Graph API authentication | A valid GUID | | GRAPHAPI__ClientSecret | Client secret for Microsoft Graph API authentication | A valid secret string | | GRAPHAPI__TenantId | Tenant ID for Microsoft Graph API authentication | A valid GUID | | GRAPHAPI__MonitorValidationDelay | Delay before validating Graph API monitor subscriptions | A valid timespan (Defaults to `00:00:30`) | | GRAPHAPI__SubscriptionDuration | Duration for Graph API subscriptions before auto-renewal | A valid timespan (Defaults to `2.00:00:00`) | | GRAPHAPI__TenantName | The name of the Azure tenant used by Graph API | A valid tenant name - e.g `Glasswall` | | CLIENTS__SyncApi__Timeout | Timeout for calls to the Sync API | A valid timespan (Defaults to `00:03:00`) | | ENABLE_OUTLOOK_BANNER | Enables or disables the Outlook banner feature | `true`, `false` (Defaults to `false`) | | EXCHANGEONLINE__AppId | Application ID for Exchange Online authentication | A valid GUID - Required if `ENABLE_OUTLOOK_BANNER` is set to `true` | | EXCHANGEONLINE__Organization | The Exchange Online organization domain | A valid `.onmicrosoft.com` domain - Required if `ENABLE_OUTLOOK_BANNER` is set to `true` | | EXCHANGEONLINE__DistributionGroupName | Name given to the distribution group created in Exchange Online | A valid string (Defaults to `GWStorageMonitoringUsers`) | | EXCHANGEONLINE__TransportRuleName | Name given to the mail flow rule which will apply the banner to those in the distribution group | A valid string (Defaults to `Glasswall Attachment Processing Notice`)| | EXCHANGEONLINE__DistributionListSyncHaloUri | Base URI for the Storage Monitor's API for retrieving monitors | A valid URI (Defaults to `http://localhost:8080` - The Storage Monitor's default HTTP port) | | EXCHANGEONLINE__DistributionListSyncCronSchedule | Schedule for the Monitor/Banner sync job to run on | A valid cron expression (Defaults to `0 */3 * * *` or `At the beginning of every 3rd hour`) | | MONITORING__BlockUnprocessedAttachments | Whether to block Email attachments in Outlook that have not been processed | `true`, `false` (Defaults to `true`) | | MONITORING__ProcessUnsupportedExtensions | Specifies whether to send files with unsupported file type extensions to the Sync API for processing | `true`, `false` | | MONITORING__HealthcheckCronSchedule | Cron schedule for running health checks | A valid cron expression (Defaults to `0 */3 * * *` or `At the beginning of every 3rd hour`) | | MONITORING__MaxDegreeOfParallelism | Maximum number of parallel monitoring operations | A positive integer (Defaults to `5`) | | MONITORING__MonitorLastSeenThreshold | Threshold duration after which a monitor is considered inactive | A valid timespan (Defaults to `03:00:00`) | ## API authentication overview Glasswall Halo supports two authentication methods: Basic and Bearer. Before making any API requests, authenticate using the method configured for your deployment. ### Basic authentication If your system uses Basic authentication, obtain the organisation ID and token from your system administrator. Combine these values in the format `organisation_id:token`, then base64 encode the result. Include the encoded value in the request header as shown below. ```json Authorization: Basic ZGVtbzpwQDU1dzByZA== ``` ### Bearer authentication For Bearer authentication, obtain a bearer token from your identity provider. Once you have the token, include it in the request header as shown below. ```json Authorization: Bearer ZGVtbzpwQDU1dzByZA== ``` **Note**: please ensure that you use the appropriate authentication method as per your system's configuration. Using the correct authentication ensures successful access to the Glasswall Halo API and prevents any authorization issues. * * * ## API authentication on an SSO-enabled deployment Single Sign-On (SSO) and the API authentication described above are **independent**. SSO (OpenID Connect) controls how **users sign in to the Halo portal**; it does **not** change how the **REST API** is authenticated. This means that on a deployment where the portal uses SSO (Microsoft Entra ID, Keycloak, Okta and so on): - If API authentication is enabled, API clients must send **Basic** credentials (organization ID and token) or a **Bearer** token on every request, exactly as on a non-SSO deployment. API authentication is separate from SSO and can be turned off by deploying with `configuration.AuthenticationScheme=None`—it is disabled by default until you enable it—in which case the API accepts requests without credentials. - When API authentication *is* enabled, a `401 Unauthorized` on `/api/...` requests on an SSO-enabled appliance almost always means the **API credentials were never provisioned** (or are incorrect), rather than an SSO/identity-provider issue. ### Provisioning and rotating Basic-auth API credentials Basic-auth credentials consist of an **organization ID** and one or more **tokens**, supplied to the API at deployment time as secrets and enabled with `configuration.AuthenticationScheme=Basic`. Multiple tokens can be associated with an organization (comma-separated), which lets you **rotate** a token by adding a new one and later removing the old one. For the platform-specific steps to create these secrets and enable Basic authentication, see: - **EKS / AWS:** [Enable API authentication](/halo/eks-step-7#73c---enable-api-authentication) - **Single Node VM (appliance):** [Authentication](/halo/single-node-vm-authentication) If you do not know whether API credentials were provisioned for your appliance, or you need them reset, contact [Glasswall Support](https://www.glasswall.com/support). * * * #### Learn more * [User roles](/halo/glasswall-halo-user-roles) * [API roles to action mapping](/halo/api-roles-to-action-mapping) * [Portal roles to action mapping](/halo/portal-roles-to-action-mapping) ## Single Sign-On (SSO) roles in the Portal Glasswall Halo offers the capability to configure Portal authentication by utilizing Single Sign-On (SSO). To enable authentication on the Portal, please refer to our [Single Sign-On Guide](/halo/glasswall-halo-sso). ### User roles | Role | Description | |-|-| | Admin | This role is designated for users executing administrative tasks. Users with this role can manage licenses and protection settings. | | ReadOnly | This role is for users who can see protection settings and reporting dashboards, however they do not have the ability to update the settings. | **Note:** file rebuilding capabilities are accessible from the Portal to all users, regardless of whether they are logged in or not. ## Roles for Azure AD authentication in API Glasswall Halo API also provides support for authentication configuration using Azure Active Directory (Azure AD). This feature supports two user roles as detailed below: | Role | Description | License Management | Policy Management| |-|-|-|-| | Admin | This role is designated for users executing administrative tasks. They can manage licenses through the License Management API and policies through the Policy Management API. | Read, Write | Read, Write | | User | This role is for users who can rebuild files and also manage policies through the Policy Management API. | Read | Read, Write | The following roles only apply when authenticating with the OIDC Client Credentials flow. When using Basic authentication, the user/app will have the same permissions as the `Admin` role. --- ### Synchronous API #### V3 | Resource | Action | User | Admin | | ------------------- | ------ | ---- | ----- | | /api/v3/cdr-file | POST | x | x | | /api/v3/cdr | POST | x | x | | /api/v3/export-file | POST | x | x | | /api/v3/export | POST | x | x | | /api/v3/import-file | POST | x | x | | /api/v3/import | POST | x | x | #### V2 | Resource | Action | User | Admin | | --------------------------------- | ------ | ---- | ----- | | /api/FileTypeDetection/file | POST | x | x | | /api/FileTypeDetection/base64 | POST | x | x | | /api/Analyse/file | POST | x | x | | /api/Analyse/base64 | POST | x | x | | /api/Rebuild/file | POST | x | x | | /api/Rebuild/base64 | POST | x | x | | /api/Rebuild/archivefile | POST | x | x | | /api/Rebuild/protectedZipfile | POST | x | x | | /api/Composite/file | POST | x | x | | /api/Composite/base64 | POST | x | x | --- ### Asynchronous API #### V1 | Resource | Action | User | Admin | | --------------------------------------------- | ------ | ---- | ----- | | /api/v1/cdr-async | POST | x | x | | /api/v1/cdr-async/\{transactionId\} | GET | x | x | | /api/v1/cdr-file-async | POST | x | x | | /api/v1/cdr-file-async/\{transaction\} | GET | x | x | --- ### Prompt injection detection API #### V1 | Resource | Action | User | Admin | | --------------------------------- | ------ | ---- | ----- | | /api/v1/detect/prompt-injection | POST | x | x | --- ### Policy Management API #### V1 | Resource | Action | User | Admin | | --------------------------------------------- | ------ | ---- | ----- | | /api/v1/policies/\{policyName\} | POST | x | x | | /api/v1/policies/\{policyName\} | PUT | x | x | | /api/v1/policies/\{policyName\} | GET | x | x | | /api/v1/policies/\{policyName\} | DELETE | x | x | | /api/v1/policies/ | GET | x | x | | /api/v1/policies/\{policyName\}/reset | PUT | x | x | --- ### License Management API #### V1 | Resource | Action | User | Admin | | --------------------------------- | ------ | ---- | ----- | | /api/v1/license-usage/\{licenseId\} | GET | | x | | /api/v1/license-usage/active | GET | | x | | /api/v1/license | POST | | x | | /api/v1/license | GET | x | x | | /api/v1/license | DELETE | | x | | /api/v1/license-usage | GET | x | x | #### V2 | Resource | Action | User | Admin | | --------------- | ------ | ---- | ----- | | /api/v2/license | GET | x | x | --- ### ICAP Profile Management API #### V1 | Resource | Action | User | Admin | | --------------------------------------------- | ------ | ---- | ----- | | /api/v1/profiles/\{profileName\} | POST | x | x | | /api/v1/profiles/\{profileName\} | PUT | x | x | | /api/v1/profiles/\{profileName\} | GET | x | x | | /api/v1/profiles/\{profileName\} | DELETE | x | x | | /api/v1/profiles/ | GET | x | x | | /api/v1/profiles/\{profileName\}/reset | PUT | x | x | --- ### File reputation settings API #### V1 | Resource | Action | User | Admin | | --------------------------------------- | ------ | ---- | ----- | | /api/v1/reversing-labs/settings | GET | | x | | /api/v1/reversing-labs/settings | PUT | | x | | /api/v1/reversing-labs/test-connection | POST | | x | --- ### XML Validation API #### V1 | Resource | Action | User | Admin | | -------------------- | ------ | ---- | ----- | | /api/v1/validate-xml | POST | x | x | --- ### XML Policy Management API #### V1 | Resource | Action | User | Admin | | ----------------------------------------------- | ------ | ---- | ----- | | /api/v1/xml-policies/\{XMLPolicyName\} | POST | x | x | | /api/v1/xml-policies/\{XMLPolicyName\} | PUT | x | x | | /api/v1/xml-policies/\{XMLPolicyName\} | GET | x | x | | /api/v1/xml-policies/\{XMLPolicyName\} | DELETE | x | x | | /api/v1/xml-policies/ | GET | x | x | | /api/v1/xml-policies/\{XMLPolicyName\}/reset | PUT | x | x | --- ### Storage Monitor API #### V1 | Resource | Action | User | Admin | | ------------------------------------------------------------------------------- | ------ | ---- | ----- | | /api/storage-monitor/sharepoint/v1/sites | GET | x | x | | /api/storage-monitor/sharepoint/v1/sites/\{siteId\}/drives | GET | x | x | | /api/storage-monitor/sharepoint/v1/sites/\{siteId\}/drives/\{driveId\}/monitors | POST | x | x | | /api/storage-monitor/sharepoint/v1/sites/\{siteId\}/drives/\{driveId\}/monitors | PATCH | x | x | | /api/storage-monitor/sharepoint/v1/sites/\{siteId\}/drives/\{driveId\}/monitors | DELETE | x | x | | /api/storage-monitor/sharepoint/v1/monitors | GET | x | x | | /api/storage-monitor/sharepoint/v1/monitors | POST | x | x | | /api/storage-monitor/sharepoint/v1/monitors | PATCH | x | x | | /api/storage-monitor/sharepoint/v1/monitors | DELETE | x | x | | /api/storage-monitor/sharepoint/v1/status | GET | x | x | | /api/storage-monitor/onedrive/v1/users | GET | x | x | | /api/storage-monitor/onedrive/v1/users/\{userId\}/monitors | POST | x | x | | /api/storage-monitor/onedrive/v1/users/\{userId\}/monitors | PATCH | x | x | | /api/storage-monitor/onedrive/v1/users/\{userId\}/monitors | DELETE | x | x | | /api/storage-monitor/onedrive/v1/monitors | GET | x | x | | /api/storage-monitor/onedrive/v1/monitors | POST | x | x | | /api/storage-monitor/onedrive/v1/monitors | PATCH | x | x | | /api/storage-monitor/onedrive/v1/monitors | DELETE | x | x | | /api/storage-monitor/onedrive/v1/status | GET | x | x | | /api/Outlook/v1/users | GET | x | x | | /api/Outlook/v1/monitors | GET | x | x | | /api/Outlook/v1/monitors | POST | x | x | | /api/Outlook/v1/monitors | PATCH | x | x | | /api/Outlook/v1/monitors | DELETE | x | x | | /api/Outlook/v1/status | GET | x | x | | Resource | Action | Roles | | | ------------------------- | ------------------ | -------- | ----- | | | | ReadOnly | Admin | | Reporting/Risk report | Read | x | x | | Reporting/Processed files | Read | x | x | | Reporting/ICAP report | Read | x | x | | Reporting/ICAP requests | Read | | x | | Clean a file | CDR / rebuild file | x | x | | Policy settings | List | x | x | | Policy settings | Read | x | x | | Policy settings | Create | | x | | Policy settings | Delete | | x | | Policy settings | Update | | x | | Policy settings | Reset | | x | | ICAP settings | List | x | x | | ICAP settings | Read | x | x | | ICAP settings | Create | | x | | ICAP settings | Delete | | x | | ICAP settings | Update | | x | | ICAP settings | Reset | | x | | ICAP settings | Clear Cache | | x | | Validation settings | List | x | x | | Validation settings | Read | x | x | | Validation settings | Create | | x | | Validation settings | Delete | | x | | Validation settings | Update | | x | | Validation settings | Reset | | x | | File reputation settings | Read | | x | | File reputation settings | Update | | x | | File reputation settings | Test connection | | x | | License management | Read | | x | | License management | Upload License | | x | | License management | Download Usage | | x | | Storage monitoring | List | x | x | | Storage monitoring | Read | x | x | | Storage monitoring | Create | | x | | Storage monitoring | Delete | | x | | Storage monitoring | Update | | x | Glasswall Halo deployment and upgrades are carried out via the use of Helm charts. The Helm charts (v3) are registered as OCI (Open Container Initiative) artifacts in an ACR (Azure Container Registry). Glasswall provides customers with a secure token to access the Helm charts and all associated container images. You can follow the steps below to complete the AKS deployment and setup, but please [contact us](https://www.glasswall.com/contact/) if you have any questions or require more support. ## Glasswall Halo Installation via Azure Kubernetes Service (AKS) * **[Prerequisites](/halo/aks-prerequisites)** * **[Step 1 - Set your Kubernetes context to AKS](/halo/aks-step-1)** * **[Step 2 - Create Glasswall Halo namespace](/halo/aks-step-2)** * **[Step 3 - Add secrets in key vault](/halo/aks-step-3)** * **[Step 4 - Enable your AKS cluster to access key vault](/halo/aks-step-4)** * **[Step 5 - Enable access to Glasswall's artifact registry](/halo/aks-step-5)** * **[Step 6 - Pull Helm charts](/halo/aks-step-6)** * **[Step 7 - Install and configure prerequisite components](/halo/aks-step-7)** * **[Step 8 - Install CDR components](/halo/aks-step-8)** * * * Let's get started → Finally, install the CDR application services. For these charts, ensure you set the image tag to the corresponding tag found in the latest release notes. The examples below are pre-populated with the tags for [v2.21.0](/halo/v2210): ## 8.1 Engine ```sh helm upgrade --install cdrplatform-engine cdrplatform-engine \ --set image.tag=2.21.0-201948 \ --set image.registry=glasswallhub.azurecr.io \ --atomic ``` ### 8.1A - Enable ReversingLabs (optional) To enable ReversingLabs file reputation: 1. Store the credentials in Azure Key Vault under `halo-reversinglabs-username` and `halo-reversinglabs-password`, as described in [Step 3](/halo/aks-step-3). external-secrets syncs them into `cdrplatform-secrets`. 2. Set the non-secret fields on the Engine-owned `engine-reversinglabs` ConfigMap. The endpoint is supplied by you; use the value below. ```sh kubectl patch configmap engine-reversinglabs -n cdrplatform --type merge \ -p '{"data":{"enabled":"true","endpoint":"https://data.reversinglabs.com/api/databrowser/malware_presence/query","timeoutSeconds":"5"}}' ``` The Engine picks up the change on its next lookup; no restart is required. `timeoutSeconds` accepts 1–120 and defaults to 5. ## 8.2 - Sync API ```sh helm upgrade --install cdrplatform-sync-api cdrplatform-sync-api \ --set image.tag=2.21.0-201948 \ --set image.registry=glasswallhub.azurecr.io \ --atomic ``` ## 8.3 API Access service The API Access service acts as a gateway to the Glasswall Halo Synchronous API and the Policy Management API. It exposes CDR functionality over HTTP. If your environment requires secure communication, configure the service to use HTTPS with TLS or SSL by following the TLS or SSL installation instructions. Otherwise, follow the instructions for installation without TLS or SSL. ### 8.3A - For deployments without TLS/SSL ```sh helm upgrade --install cdrplatform-api-access cdrplatform-api-access \ --set image.registry=glasswallhub.azurecr.io \ --set image.tag=2.21.0-201948 \ --atomic ``` ### 8.3B - For deployments with TLS/SSL To enable SSL on the CDR API, create a private key and certificate for the domain in use. Create a Kubernetes secret from the private key and certificate files using the command below. This command creates a secret named `tls-secret` using `server.key` as the private key and `server.crt` as the certificate. In this example, the private key must not be protected by a passphrase. ```sh kubectl create secret tls tls-secret --key server.key --cert server.crt ``` This secret can then be used to enable TLS on the ingress ensuring that the domain name is set in the command below: ```sh helm upgrade --install cdrplatform-api-access cdrplatform-api-access \ --set image.tag=2.21.0-201948\ --set image.registry=glasswallhub.azurecr.io \ --set ingress.tls.enabled=true \ --set ingress.tls.domain= \ --set ingress.tls.secretName=tls-secret \ --atomic ``` ### 8.3C - To enable basic API authentication Authentication in Glasswall Halo is disabled by default. When enabled, it applies to both the Glasswall Halo Synchronous API and the Policy API. To enable authentication: - Create two secrets in Azure Key Vault. One secret stores the organisation ID, and the other stores the organisation tokens. When specifying multiple tokens, separate them with commas. Ensure that individual tokens do not contain commas. Azure Key Vault secrets must follow the naming conventions below: >- **Organisation ID secret** >>- Must start with `organisation` >>- Must end with `-id` >>- Include a numeric index between the prefix and suffix >>- Examples: `organisation0-id`, `organisation1-id`, `organisation2-id` >- **Organisation tokens secret** >>- Must start with `organisation` >>- Must end with `-tokens` >>- Include a numeric index between the prefix and suffix >>- Examples: `organisation0-tokens`, `organisation1-tokens`, `organisation2-tokens` - Set `configuration.AuthenticationScheme=Basic` while deploying the Helm chart. For example: >```sh >helm upgrade --install cdrplatform-api-access cdrplatform-api-access \ > --set image.registry=glasswallhub.azurecr.io \ > --set image.tag=2.21.0-201948 \ > --set configuration.AuthenticationScheme=Basic \ > --atomic >``` ### 8.3D - To enable Azure AD API authentication To enable Azure AD based authentication, set `configuration.AuthenticationScheme=Bearer` and configure the `tenant_id` and `domain_name` variables when deploying the Helm chart. Note that the app registration URI may vary depending on when the app was created. Always copy the app registration URI from the Azure portal and use it as the valid audience. ```sh tenant_id="" domain_name="" valid_audiences="api://cdrplatform-api-access" or "api://${tenant_id}/cdrplatform-api-access" # (verify app registration URI from Azure) helm upgrade --install cdrplatform-api-access cdrplatform-api-access --wait --atomic \ --set image.tag="2.21.0-201948" \ --set image.registry=glasswallhub.azurecr.io \ --set ingress.tls.enabled=true \ --set ingress.tls.domain=${domain_name} \ --set ingress.tls.secretName=tls-secret \ --set configuration.AuthenticationScheme="Bearer" \ --set configuration.Authentication__Schemes__Bearer__ValidAudiences__0=${valid_audiences} \ --set configuration.Authentication__Schemes__Bearer__ValidIssuer=https://sts.windows.net/${tenant_id}/ \ --set configuration.Authentication__Schemes__Bearer__Authority=https://login.microsoftonline.com/${tenant_id}/v2.0/ ``` Azure US Government Users Use the following command when deploying to Azure US Government. These settings ensure the correct endpoints and environment values are used. ```bash tenant_id="" domain_name="" valid_audiences="api://cdrplatform-api-access" or "api://${tenant_id}/cdrplatform-api-access" # (verify app registration URI from Azure portal) helm upgrade --install cdrplatform-api-access cdrplatform-api-access --wait --atomic \ --set image.tag="2.21.0-201948" \ --set image.registry=glasswallhub.azurecr.io \ --set ingress.tls.enabled=true \ --set ingress.tls.domain=${domain_name} \ --set ingress.tls.secretName=tls-secret \ --set configuration.AuthenticationScheme="Bearer" \ --set configuration.Authentication__Schemes__Bearer__ValidAudiences__0=${valid_audiences} \ --set configuration.Authentication__Schemes__Bearer__ValidIssuer=https://sts.windows.net/${tenant_id}/ \ --set configuration.Authentication__Schemes__Bearer__Authority=https://login.microsoftonline.us/${tenant_id}/v2.0/ ``` ## 8.4 Portal To deploy the Portal service, run the following commands: ### 8.4A - For deployments without TLS/SSL **Note:** the `` mentioned in this command refers to the Load Balancer's public IP address. This can be retrieved through the "Portal & API Access" steps below. ```sh helm upgrade --install cdrplatform-portal cdrplatform-portal \ --set image.registry=glasswallhub.azurecr.io \ --set configuration.BackendUrl="http://" \ --set image.tag=2.21.0-201948 \ --set configuration.HaloVersion=2.21.0 \ --atomic ``` ### 8.4B - For deployments with TLS/SSL If TLS is required, add the `--set ingress.tls.enable_tls=true` parameter and set `portal_domain=`. In the example below, the same Kubernetes secret created for the API Access service is used to retrieve the TLS certificates. ```sh portal_domain="" helm upgrade --install cdrplatform-portal cdrplatform-portal \ --set image.tag=2.21.0-201948 \ --set image.registry=glasswallhub.azurecr.io \ --set ingress.tls.enabled=true \ --set ingress.tls.domain=${portal_domain} \ --set ingress.tls.secretName=tls-secret \ --set configuration.BackendUrl="https://${portal_domain}" \ --set configuration.HaloVersion=2.21.0 \ --atomic ``` ### 8.4C - Enable admin access without authentication When Single Sign-On (SSO) login is not enabled in the Portal, administrative features, such as license management and policy configuration are not accessible. To enable access to these admin features without setting up SSO, deploy the Helm chart with the following configuration: ```sh helm upgrade --install cdrplatform-portal cdrplatform-portal \ --set image.tag=2.21.0-201948 \ --set image.registry=glasswallhub.azurecr.io \ --set ingress.tls.enabled=true \ --set ingress.tls.domain=${portal_domain} \ --set ingress.tls.secretName=tls-secret \ --set configuration.AutoAdmin="true" \ --atomic ``` ### 8.4D To configure SSO login in Glasswall Halo's Portal Set the `portal_domain`, `portal_client_id`, and `tenant_id` variables in the commands below, then run them. The `portal_domain` is the domain name used to configure TLS for the Portal service and must match the domain used in the `cdrplatform-portal-client` app registration. `portal_client_id` is the application (client) ID of the `cdrplatform-portal-client` app registration created in the prerequisites step. `tenant_id` is of the tenant where APP registrations are created. `enabled_pages` should contain various pages that needs to be enabled. Pass the values separated by comma(,). For example it should be set to `SystemSettings\,PolicySettings\,ValidationSettings\,IcapSettings\,IcapRequests\,IcapReporting` when Policy API, ICAP server and ReversingLabs is deployed, Or set it to `SystemSettings\,PolicySettings\,ValidationSettings` if only Policy API is deployed. If none of the Policy API and ICAP server are deployed, set it to `enabled_pages="SystemSettings"`. ```sh portal_domain="" portal_client_id="" tenant_id="" enabled_pages="SystemSettings\,PolicySettings\,ValidationSettings" helm upgrade --install cdrplatform-portal cdrplatform-portal \ --set image.registry=glasswallhub.azurecr.io \ --set image.tag=2.21.0-201948 \ --set ingress.tls.enabled=true \ --set ingress.tls.domain=${portal_domain} \ --set ingress.tls.secretName=tls-secret \ --set configuration.BackendUrl="https://${portal_domain}" \ --set configuration.EnabledPages=${enabled_pages} \ --set configuration.OIDC.ProviderOptions.Authority="https://login.microsoftonline.com/${tenant_id}/v2.0" \ --set configuration.OIDC.ProviderOptions.RedirectUri="https://${portal_domain}/authentication/login-callback" \ --set configuration.OIDC.ProviderOptions.ClientId="${portal_client_id}" \ --set configuration.OIDC.ProviderOptions.PostLogoutRedirectUri="https://${portal_domain}/authentication/logout-callback" \ --set configuration.HaloVersion=2.21.0 \ --atomic ``` Azure US Government Users Use the following command when deploying to Azure US Government. These settings ensure the correct endpoints and environment values are used. ```bash portal_domain="" portal_client_id="" tenant_id="" enabled_pages="SystemSettings\,PolicySettings\,ValidationSettings" helm upgrade --install cdrplatform-portal cdrplatform-portal \ --set image.registry=glasswallhub.azurecr.io \ --set image.tag=2.21.0-201948 \ --set ingress.tls.enabled=true \ --set ingress.tls.domain=${portal_domain} \ --set ingress.tls.secretName=tls-secret \ --set configuration.BackendUrl="https://${portal_domain}" \ --set configuration.EnabledPages=${enabled_pages} \ --set configuration.OIDC.ProviderOptions.Authority="https://login.microsoftonline.us/${tenant_id}/v2.0" \ --set configuration.OIDC.ProviderOptions.RedirectUri="https://${portal_domain}/authentication/login-callback" \ --set configuration.OIDC.ProviderOptions.ClientId="${portal_client_id}" \ --set configuration.OIDC.ProviderOptions.PostLogoutRedirectUri="https://${portal_domain}/authentication/logout-callback" \ --set configuration.HaloVersion=2.21.0 \ --set appenvironment.HTTP_CSP_CONNECT_SRC="'self'https://login.microsoftonline.us https://graph.microsoft.us" \ --set appenvironment.HTTP_CSP_FRAME_SRC="'self' https://login.microsoftonline.us" \ --set appenvironment.HTTP_CSP_FRAME_ANCESTORS="'self' https://login.microsoftonline.us" \ --atomic ``` ## 8.5 Portal access Portal Access acts as the backend for the Portal. It enables the Halo Portal to access the Policy Management API and the Synchronous API. ### 8.5A For deployments with TLS/SSL If TLS is required, add the `--set ingress.tls.enable_tls=true` and `--set ingress.tls.domain=` parameters. In the example below, the same Kubernetes secret created for the API Access service is used to retrieve the TLS certificates. ```sh helm upgrade --install cdrplatform-portal-access cdrplatform-portal-access \ --set image.registry=glasswallhub.azurecr.io \ --set image.tag=2.21.0-201948 \ --set ingress.tls.enabled=true \ --set ingress.tls.domain= \ --set ingress.tls.secretName=tls-secret \ --set configuration.AuthenticationScheme=None \ --atomic ``` ### 8.5B For deployments without TLS/SSL ```sh helm upgrade --install cdrplatform-portal-access cdrplatform-portal-access \ --set image.registry=glasswallhub.azurecr.io \ --set configuration.AuthenticationScheme=None \ --set image.tag=2.21.0-201948 \ --atomic ``` ### 8.5C Without authentication ```sh helm upgrade --install cdrplatform-portal-access cdrplatform-portal-access \ --set image.registry=glasswallhub.azurecr.io \ --set image.tag=2.21.0-201948 \ --set configuration.AuthenticationScheme=None \ --atomic ``` ### 8.5D To enable Azure AD authentication Portal Access enables SSO authentication using Azure AD. Set the `tenant_id` and `portal_domain` variables in the commands below, then run them. Note that the APP registration URI may vary depending on when it was created. Always copy the app registration URI from the Azure portal and use it as the valid audience. ```sh tenant_id="" portal_domain="" valid_audiences="" # Application (client) ID of the cdrplatform-portal-access app registration (copy from Azure portal) helm upgrade --install cdrplatform-portal-access cdrplatform-portal-access \ --set image.registry=glasswallhub.azurecr.io \ --set image.tag=2.21.0-201948 \ --set ingress.tls.enabled=true \ --set ingress.tls.domain=${portal_domain} \ --set ingress.tls.secretName=tls-secret \ --set configuration.AuthenticationScheme=Bearer \ --set configuration.Authentication__Schemes__Bearer__ValidAudiences__0=${valid_audiences} \ --set configuration.Authentication__Schemes__Bearer__ValidIssuer=https://sts.windows.net/${tenant_id}/ \ --set configuration.Authentication__Schemes__Bearer__Authority=https://login.microsoftonline.com/${tenant_id}/v2.0/ \ --atomic ``` Azure US Government Users Use the following command when deploying to Azure US Government. These settings ensure the correct endpoints and environment values are used. ```bash tenant_id="" portal_domain="" valid_audiences="" # Application (client) ID of the cdrplatform-portal-access app registration (copy from Azure portal) helm upgrade --install cdrplatform-portal-access cdrplatform-portal-access \ --set image.registry=glasswallhub.azurecr.io \ --set image.tag=2.21.0-201948 \ --set ingress.tls.enabled=true \ --set ingress.tls.domain=${portal_domain} \ --set ingress.tls.secretName=tls-secret \ --set configuration.AuthenticationScheme=Bearer \ --set configuration.Authentication__Schemes__Bearer__ValidAudiences__0=${valid_audiences} \ --set configuration.Authentication__Schemes__Bearer__ValidIssuer=https://sts.windows.net/${tenant_id}/ \ --set configuration.Authentication__Schemes__Bearer__Authority=https://login.microsoftonline.us/${tenant_id}/v2.0/ \ --atomic ``` ## 8.6 Policy Management API The Policy Management API is used to manage Glasswall Halo content management flags. This service is optional and should be installed only if you plan to create and use custom policies. ```sh helm upgrade --install cdrplatform-policy-api cdrplatform-policy-api \ --set image.registry=glasswallhub.azurecr.io \ --set image.tag=2.21.0-201948 \ --set configuration.DATABASE__Provider=${database_provider} ``` - The Policy Management API swagger page can be accessed using: >``` >http:///swagger/index.html >``` ## 8.7 License Management The License Management service is used to manage licenses in the Glasswall Halo Portal. ```sh helm upgrade --install cdrplatform-license-management cdrplatform-license-management -n license-management\ --set image.registry=glasswallhub.azurecr.io \ --set image.tag=2.21.0-201948 \ --set configuration.DATABASE__Provider=${database_provider} \ -n license-management \ --atomic ``` ## 8.8 Clean up service The Clean up service deletes original and rebuilt files from persistent storage after processing is complete. ```sh helm upgrade --install cdrplatform-cleanup cdrplatform-cleanup \ --set image.registry=glasswallhub.azurecr.io \ --set image.tag=2.21.0-201948 \ --atomic ``` ## 8.9 Asynchronous API The Asynchronous API can be deployed using the command below. A MongoDB database is required for the Asynchronous API. ```sh helm upgrade --install cdrplatform-async-api cdrplatform-async-api \ --set image.tag=2.21.0-201948\ --set image.registry=glasswallhub.azurecr.io \ --set configuration.DATABASE__Provider=${database_provider} \ --atomic ``` ## 8.10 Metrics projection The Metrics projection service is used to pull reporting data from MongoDB to display in the Halo Portal UI. ```sh helm upgrade --install cdrplatform-metrics-projection cdrplatform-metrics-projection \ --set image.tag=2.21.0-201948 \ --set image.registry=glasswallhub.azurecr.io \ --set configuration.DATABASE__Provider=${database_provider} \ --set cloud_provider=azure \ --atomic ``` ## 8.11 Report extractor The Report extractor service extracts analysis reports and publishes them for reporting. ```sh helm upgrade --install cdrplatform-report-extractor cdrplatform-report-extractor \ --set image.tag=2.21.0-201948 \ --set image.registry=glasswallhub.azurecr.io \ --atomic ``` ## 8.12 Tally accumulator The Tally accumulator service tracks and maintains a tally of usage statistics. ```sh helm upgrade --install cdrplatform-tally-accumulator cdrplatform-tally-accumulator \ --set image.tag=2.21.0-201948 \ --set image.registry=glasswallhub.azurecr.io \ --set configuration.DATABASE__Provider=${database_provider} \ --atomic ``` ## 8.13 MongoDB **Note:** if MongoDB is already configured in Azure and you do not want to install or manage MongoDB within your cluster, you can skip this step and proceed. ### MongoDB operator ```sh helm install community-operator mongodb/community-operator --namespace cdrplatform \ --set operator.version=0.9.0 \ --set agent.version=107.0.0.8465-1 \ --atomic ``` ### Install MongoDB ```sh helm upgrade -i cdrplatform-mongodb cdrplatform-mongodb -n cdrplatform --atomic \ --set cloud_provider=azure ``` - Retrieve the connection string from the Kubernetes secret ```sh kubectl get secret mongodb-cdrplatform-cdrp-user -o jsonpath='{.data.connectionString\.standard}' | base64 -d ``` - Update the Azure Key Vault secret with the MongoDB connection string ```sh az keyvault secret set --name "mongodb-connectionstring" --vault-name "${kvname}" --value "" ``` ## 8.14 Storage Monitoring Service The Storage Monitoring service provides an API to subscribe and track changes made to documents in configured storage mechanisms such as SharePoint and OneDrive. TLS must be enabled for the Webhook. Add the `--set ingress.tls.enable_tls=true` and `--set ingress.tls.domain=` parameters. In the example below, the same Kubernetes secret created for the API Access service is used to retrieve the TLS certificates. ```sh helm upgrade --install cdrplatform-storage-monitor cdrplatform-storage-monitor --wait --atomic \ --set image.tag="2.21.0-201948" \ --set image.registry=glasswallhub.azurecr.io \ --set configuration.DATABASE__Provider="${database_provider}" \ --set configuration.WEBHOOKS__CallbackBaseAddress="https://" \ --set ingress.tls.enabled=true \ --set ingress.tls.domain= \ --set ingress.tls.secretName=tls-secret \ --atomic ``` ## 8.15 Optional Prometheus scaling Install the Prometheus scaling chart only if you want to enable Prometheus-based scaling. ```sh helm upgrade --install prometheus-scaling prometheus-scaling --wait --atomic \ --set image.registry=glasswallhub.azurecr.io \ --set image.tag=3.13.2-full \ --set imagePullSecrets[0].name=acr-secret \ --set keda.icap.enabled= ``` ## 8.16 Portal & API Access Use the following command to determine the external IP address associated with your cluster. The external IP corresponds to the public IP address assigned to the Azure load balancer. ```sh kubectl get services --namespace cdrplatform nginx-ingress-ingress-nginx-controller --output jsonpath='{.status.loadBalancer.ingress[0].ip}' ``` You can now use the IP returned above to navigate to the Halo Portal and API documentation (use https if TLS enabled): ``` Portal: http:// API Documentation: http:///swagger/index.html ``` Need help → Before you begin the process of deploying Glasswall Halo, ensure that you have the following tools and resources installed and setup. ## System requirements The following guide has been written assuming it is being run on a Linux shell. if running on a different shell, some of the commands may need to be modified to run. If running on Windows, please ensure you are using Windows subsystem for Linux. ## Required tools * [Helm](https://Helm.sh/docs/intro/install/) > 3.8 (to enable full support for OCI repositories) * [Kubectl](https://Kubernetes.io/docs/tasks/tools/) * [Azure CLI](https://learn.microsoft.com/en-us/cli/azure/install-azure-cli) ## Required Azure resources ### 1. AKS instance * Recommended total of at least 8 vCPU and 32 GB RAM. * Minimum node size is 4 vCPU and 16 GB RAM. * For production workloads a minimum of 2 nodes is recommended. * In the steps below, the AKS cluster is referred to as: **aksname** **Note:** Glasswall Halo does not support arm64 node VMs. For guidance on creating an AKS cluster please refer to: * [Create AKS cluster - Portal](https://learn.Microsoft.com/en-us/Azure/AKS/learn/quick-Kubernetes-deploy-portal) * [Create AKS cluster - CLI](https://learn.Microsoft.com/en-us/Azure/AKS/learn/quick-Kubernetes-deploy-cli) * [Azure Kubernetes service - best practices](https://learn.Microsoft.com/en-us/Azure/AKS/best-practices) ### 2. Key Vault * Glasswall Halo secrets are managed using Azure Key Vault. These secrets are synced into the AKS cluster via [External secrets](https://external-secrets.io/latest/). * In the steps below, the Key Vault is referred to as: **kvname** For guidance on creating an Azure Key Vault please refer to: * [Create Key Vault - Portal](https://learn.Microsoft.com/en-gb/Azure/key-vault/general/quick-create-portal) * [Create Key Vault - CLI](https://learn.Microsoft.com/en-us/Azure/key-vault/general/quick-create-cli) * [Azure Key Vault - best practices](https://learn.Microsoft.com/en-us/Azure/key-vault/general/best-practices) ### 3. Storage account * A storage account is used to store reports for each file processed by the platform. The SKU of the storage account does not impact performance, but we recommend at least GRS configuration. * An additional storage account is used as Azure file share backing the cluster's persistent volume. This is added to the AKS cluster's resource group automatically. * In the steps below, the storage account that collects the CDR reports is referred to as: **saname** File shares: * 1 X 10 TB provisioned capacity (1100.0 MiB/s throughput) * Used for the Glasswall Halo transient file store and Glasswall Halo RabbitMQ ### **4. Database** #### **Option 1 - MongoDB database** MongoDB can be used to store the Glasswall Halo content management policies and data required for asynchronous file processing. MongoDB is used by `cdrplatform-policy-API`, `cdrplatform-async-API`, `cdrplatform-metrics-projection`, `cdrplatform-tally-accumulator` services. * Deploy a CosmosDB with MongoDB API compatibility. * Ensure that the instance is setup with the appropriate [throughput allocation](https://learn.Microsoft.com/en-us/azure/cosmos-db/mongodb/how-to-provision-throughput), depending on how much data is expected to pass through Glasswall Halo. **Note:** alternatively, MongoDB can deployed directly inside of your cluster. Deployable using the MongoDB Helm charts, as seen in Step 8. #### Option 2 - CosmosDB database Alternatively, CosmosDB can be used to store Glasswall Halo content management policies, data for asynchronous file processing, and metrics. * Deploy a CosmosDB instance in Azure. * Ensure that the instance is setup with the appropriate [throughput allocation](https://learn.microsoft.com/en-us/azure/cosmos-db/mongodb/how-to-provision-throughput), depending on how much data is expected to pass through Glasswall Halo. ### 5. App registrations A domain name must be configured for Glasswall Halo's Portal to use SSO authentication. Azure app registrations are required whenever SSO authentication using Azure AD needs to be enabled. There are 3 app registrations that need to be created: * **cdrplatform-API-access** * **cdrplatform-portal-access** * **cdrplatform-portal-client** The [attached shell script](/.attachments/create-azure-app-registrations.sh) helps to create these 3 app registrations. Enter the domain name you would like to use for the portal as an argument to the shell script. For example, use [halo.glasswall.com](https://halo.glasswall.com) as the domain name and run the script: ``` bash create-azure-app-registrations.sh halo.glasswall.com ``` ## Access to Glasswall Artifact Registry * You are provided with a token & token ID to access [Glasswall's Artifact Registry](/halo/glasswall-artifact-registry). * This allows you to directly pull container images and Helm charts from your AKS cluster. * In the steps below, the token and token ID will be referred to as: **token** and **token_id.** * * * ## Assigning variables ### Variables assigned before you begin The variables mentioned in [Required Tools](#required-tools) above need to be assigned before you can begin Glasswall Halo installation. You'll need to replace **"..." **with your own values. * Azure resource group name: **rgp="..."** * Azure Kubernetes service name: **aksname="..."** * Azure Key Vault name: **kvname="..."** * Azure storage account name: **saname="..."** * Azure container registry token ID: **token_id="..."** * Azure container registry token: **token="..."** **Note: **the steps above assume each resource is in the same resource group referred to as: **rgp**.** ** ### Variables assigned during steps The following variables will need to be assigned during the installation process as resources are created. * Azure storage account connection string: **saconnstring** ="..." * Managed identity access to key vault get and list permissions using objectid: **objectid** ="..." * Managed identity access to key vault using client ID for Helm install: **miclientid** ="..." **Note:** ensure the correct database provider is setup and relevant secrets are in the Key Vault as mentioned in Step 3. Variable| provider ---|--- `Mongo`| MongoDB `Cosmos`| CosmosDB * * * Continue Need help? Set your Kubernetes context to your AKS cluster by logging into the Azure CLI. ```shell az aks get-credentials -g ""$rgp"" -n ""$aksname"" ``` * * * Continue Need help? All services associated with Glasswall Halo are deployed to a namespace within the target cluster. Create the namespace and then set the default context for the subsequent commands to be this newly created namespace. ```sh kubectl create namespace "cdrplatform" kubectl create namespace "license-management" kubectl config set-context --current --namespace="cdrplatform" ``` * * * Continue Need help? ## Storage account connection string To enable Glasswall Halo's access to the reporting storage account ('saname'), the connection string is required in Key Vault. You can retrieve the connection string through the Azure Portal or via the Azure CLI as shown in the example below. Just remember to enter your storage account and resource group. ```sh az storage account show-connection-string --name "${saname}" -g "${rgp}" ``` - Enter your connection string (highlighted in the screenshot) by replacing `${saconnstring}`, and enter your Key Vault by replacing `${kvname}` (as below). ```sh az keyvault secret set --name "azure-storage-connectionstring" --vault-name "${kvname}" --value "${saconnstring}" ``` ## 3.2A - Database option 1 - MongoDB connection string To enable Halo's connection to the Database, the connection string of the MongoDB should be created in the Key Vault - You can retrieve the connection string of the CosmosDB (MongoDB compatible) through the Azure Portal or via the Azure CLI as shown in the example below. **Note:** remember to enter your CosmosDB account name and resource group. The command will provide a list of 4 connection strings. You may use any one of the first 2 connections strings. You cannot use the read-only connection strings. ```sh az cosmosdb list-connection-strings --name "${cosmosdb_name}" -g "${rgp}" ``` - Enter your connection string (highlighted in the screenshot) by replacing `${mongodb_connstring}`, and enter your Key Vault by replacing `${kvname}` (as below). ```sh az keyvault secret set --name "mongodb-connectionstring" --vault-name "${kvname}" --value "${mongodb_connstring}" ``` ### Add the MongoDB passwords to Azure Key Vault as a secret **Note:** if you have previously configured and set up MongoDB within Azure and created your MongoDB connection string as listed above, you can skip this step. If not, to enable Glasswall Halo's Policy Management API to create and manage the policies in MongoDB, and Asynchronous API to create and manage the requests, MongoDB needs to be deployed using Helm charts listed in Step 8. Two users will be created by the MongoDB Helm chart and the corresponding user's password needs to be set in the Vault secret. ```shell az keyvault secret set --name "mongodb-cdrp-password" --vault-name "${kvname}" --value "" az keyvault secret set --name "mongodb-admin-password" --vault-name "${kvname}" --value "" ``` ## 3.2B - Database option 2 - CosmosDB connection string If CosmosDB is set up instead of MongoDB, the Cosmos connection string should be added to Key Vault. You can retrieve the connection string of CosmosDB through the Azure Portal or via the Azure CLI as shown in the example below. Just remember to enter your Cosmos DB account name and resource group. The command will provide a list of 4 connection strings. You may use any one of the first 2 connection strings. You cannot use the read-only connection strings. ```shell az cosmosdb list-connection-strings --name "${cosmosdb_name}" -g "${rgp}" ``` Enter your connection string (highlighted in the screenshot) by replacing `${mongodb_connstring}`, and enter your Key Vault by replacing `${kvname}` (as below). ```shell az keyvault secret set --name "mongodb-connectionstring" --vault-name "${kvname}" --value "${mongodb_connstring}" ``` ## 3.3 - Optional: Add ReversingLabs credentials To integrate Halo with ReversingLabs, add the credentials of ReversingLabs to the Key Vault. Replace `${reversinglabs_username}` and `${reversinglabs_password}` with actual username and password. ```shell az keyvault secret set --name "halo-reversinglabs-username" --vault-name "${kvname}" --value "${reversinglabs_username}" az keyvault secret set --name "halo-reversinglabs-password" --vault-name "${kvname}" --value "${reversinglabs_password}" ``` ## 3.4 - Optional: Add ICAP MTLS certificates ICAP servers can be configured for mutual client authentication using MTLS certificates. The certificates will be mounted to the ICAP server pods using Kubernetes secrets. Add the server certificates and certificate authority to the Key Vault so that they can be synced to Kubernetes secrets. ```shell az keyvault secret set --vault-name "${kvname}" --name tls-server-cert -f az keyvault secret set --vault-name "${kvname}" --name tls-server-key -f az keyvault secret set --vault-name "${kvname}" --name tls-cafile -f ``` ## 3.5 - Optional: Configure storage monitoring for M365 To enable M365 storage monitoring integration, you’ll need to add your Microsoft Entra app registration credentials to the Key Vault. This integration requires the following values: - Client ID - Tenant ID - Client secret Use the commands below to add these secrets to your Key Vault: ```shell az keyvault secret set --name "graphapi-clientid" --vault-name "${kvname}" --value "" az keyvault secret set --name "graphapi-tenantid" --vault-name "${kvname}" --value "" az keyvault secret set --name "graphapi-clientsecret" --vault-name "${kvname}" --value "" ``` If you don’t have these values yet, refer to our [Storage Monitoring M365 Setup Guide](/halo/registering-with-microsoft-entra-id) for instructions on how to obtain them. * * * Continue Need help? There are **two** methods for authentication; select the one which applies to you: - [**4A - Managed identity**](/halo/aks-step-4#4a---managed-identity) **(recommended)** - [**4B - Service principal**](/halo/aks-step-4#4b---service-principal) - Use this method only if managed identities are not available or not desired in your cluster. --- ## 4A - Managed identity If your AKS cluster was not created with managed identities, this can be added via: ```shell az aks update -g "${rgp}" -n "${aksname}" --enable-managed-identity ``` To sync Key Vault secrets to Kubernetes secrets, the AKS kubelet identity needs **get** and **list** access to Key Vault. - First, retrieve the object ID of the kubelet identity: ```shell az aks show -g "${rgp}" -n "${aksname}" ``` This will return a large JSON response. Scroll until you find: ``` identityProfile → kubeletidentity → objectId ``` - Now set access permissions on the Key Vault: ```shell az keyvault set-policy --name "${kvname}" --object-id "${objectid}" --secret-permissions get list ``` --- ## 4B - Service principal If you are using a service principal, you will need: - `objectId` - `appId` - `tenantId` - `clientSecret` These can all be found in **Azure Active Directory**. ```shell az keyvault set-policy --name "${kvname}" --object-id "${objectid}" --secret-permissions get list ``` - Create the Kubernetes secret: ```shell kubectl create secret generic keyvault-service-principal \ --from-literal=ClientID="${appid}" \ --from-literal=ClientSecret="${clientsecret}" ``` * * * Continue Need help? The Glasswall Artifact Registry (`glasswallhub.azurecr.io`) hosts the container images and Helm charts required to complete your Glasswall Halo installation. To pull container images and Helm charts from the Glasswall Artifact Registry, use your **token** and **token_id** with the commands shown below. ```shell kubectl create secret docker-registry acr-secret -n cdrplatform \ --docker-server="glasswallhub.azurecr.io" \ --docker-username="$token_id" \ --docker-password="$token" kubectl create secret docker-registry acr-secret -n license-management \ --docker-server="glasswallhub.azurecr.io" \ --docker-username="$token_id" \ --docker-password="$token" ``` * Login to the Helm registry using your **token_id** and **token** as shown below. ```shell helm registry login "glasswallhub.azurecr.io" -u ""$token_id"" -p ""$token"" ``` *** Please see [Glasswall Artifact Registry ](/halo/glasswall-artifact-registry)for a complete list of hosted container images and Helm charts. * * * Continue Need help? The Glasswall Halo consists of Helm charts hosted in our authenticated Artifact Registry, along side a few industry recognized third-party charts. - Before the charts can be installed, the right versions need to be pulled by Helm. The required versions are listed in the corresponding Release Notes. - An example pull command is shown below; this needs to be executed for each Helm chart listed. ```sh helm pull oci://glasswallhub.azurecr.io/ --version --untar --untardir . ``` - The following example is pre-populated with the versions for [v2.21.0](/halo/v2210): ```sh helm pull oci://glasswallhub.azurecr.io/helm/cdrplatform-rabbitmq --version 0.9.4 --untar --untardir . helm pull oci://glasswallhub.azurecr.io/helm/cdrplatform-external-secrets --version 0.6.0 --untar --untardir . helm pull oci://glasswallhub.azurecr.io/helm/cdrplatform-storage --version 0.8.1 --untar --untardir . helm pull oci://glasswallhub.azurecr.io/helm/cdrplatform-engine --version 0.6.3 --untar --untardir . helm pull oci://glasswallhub.azurecr.io/helm/cdrplatform-sync-api --version 0.4.2 --untar --untardir . helm pull oci://glasswallhub.azurecr.io/helm/cdrplatform-portal --version 0.4.2 --untar --untardir . helm pull oci://glasswallhub.azurecr.io/helm/cdrplatform-policy-api --version 0.3.2 --untar --untardir . helm pull oci://glasswallhub.azurecr.io/helm/cdrplatform-api-access --version 1.2.3 --untar --untardir . helm pull oci://glasswallhub.azurecr.io/helm/cdrplatform-portal-access --version 0.2.2 --untar --untardir . helm pull oci://glasswallhub.azurecr.io/helm/cdrplatform-license-management --version 0.4.0 --untar --untardir . helm pull oci://glasswallhub.azurecr.io/helm/cdrplatform-cleanup --version 0.3.4 --untar --untardir . helm pull oci://glasswallhub.azurecr.io/helm/cdrplatform-async-api --version 0.2.3 --untar --untardir . helm pull oci://glasswallhub.azurecr.io/helm/prometheus-scaling --version 0.3.1 --untar --untardir . helm pull oci://glasswallhub.azurecr.io/helm/cdrplatform-metrics-projection --version 0.2.2 --untar --untardir . helm pull oci://glasswallhub.azurecr.io/helm/cdrplatform-report-extractor --version 0.2.2 --untar --untardir . helm pull oci://glasswallhub.azurecr.io/helm/cdrplatform-tally-accumulator --version 0.2.4 --untar --untardir . helm pull oci://glasswallhub.azurecr.io/helm/cdrplatform-storage-monitor --version 0.2.5 --untar --untardir . helm pull oci://glasswallhub.azurecr.io/helm/cdrplatform-mongodb --version 0.3.0 --untar --untardir . ``` - In order to gain access to the repositories from which our prerequisite components are installed - they must be added: ```sh helm repo add bitnami https://charts.bitnami.com/bitnami helm repo add kedacore https://kedacore.github.io/charts helm repo add ingress-nginx https://kubernetes.github.io/ingress-nginx helm repo add external-secrets https://charts.external-secrets.io helm repo add mongodb https://mongodb.github.io/helm-charts helm repo update ``` Continue Need help? Now that you have access to the Helm charts, they can be deployed. - First install the prerequisite components. ```sh helm upgrade --install rabbitmq-cluster-operator oci://glasswallhub.azurecr.io/docker/bitnamicharts/rabbitmq-cluster-operator \ --atomic \ --version 4.4.34 \ --set global.imageRegistry=glasswallhub.azurecr.io \ --set global.imagePullSecrets[0]=acr-secret \ --set global.security.allowInsecureImages=true \ --set msgTopologyOperator.fullnameOverride=rabbitmq-messaging-topology-operator \ --set clusterOperator.image.repository="cgr.dev/rabbitmq-cluster-operator" \ --set clusterOperator.image.tag=2.17.0 \ --set msgTopologyOperator.image.repository="cgr.dev/rabbitmq-messaging-topology-operator" \ --set msgTopologyOperator.image.tag=1.18.3 \ --set credentialUpdaterImage.repository="cgr.dev/rabbitmq-default-user-credential-updater" \ --set credentialUpdaterImage.tag=1.0.16 \ --set rabbitmqImage.repository="cgr.dev/rabbitmq" \ --set rabbitmqImage.tag=4.2.9 \ --set clusterOperator.watchAllNamespaces=false \ --set clusterOperator.watchNamespaces={cdrplatform} \ --set msgTopologyOperator.watchAllNamespaces=false \ --set msgTopologyOperator.watchNamespaces={cdrplatform} \ --set clusterOperator.resources.requests.cpu=100m \ --set clusterOperator.resources.requests.memory=256Mi \ --set clusterOperator.resources.limits.cpu=100m \ --set clusterOperator.resources.limits.memory=256Mi \ --set msgTopologyOperator.resources.requests.cpu=100m \ --set msgTopologyOperator.resources.requests.memory=256Mi \ --set msgTopologyOperator.resources.limits.cpu=100m \ --set msgTopologyOperator.resources.limits.memory=256Mi helm upgrade --install keda "oci://glasswallhub.azurecr.io/ghcr/home-operations/charts-mirror/keda" --atomic \ --set global.image.registry="glasswallhub.azurecr.io" \ --set imagePullSecrets[0].name=acr-secret \ --set image.keda.repository="cgr.dev/keda" \ --set image.keda.tag=2.20.2 \ --set image.metricsApiServer.repository="cgr.dev/keda-metrics-apiserver" \ --set image.metricsApiServer.tag=2.20.2 \ --set image.webhooks.repository="cgr.dev/keda-admission-webhooks" \ --set image.webhooks.tag=2.20.2 \ --version 2.20.2 helm upgrade --install nginx-ingress oci://glasswallhub.azurecr.io/k8s/ingress-nginx/charts/ingress-nginx --atomic \ --set imagePullSecrets[0].name=acr-secret \ --set global.image.registry="glasswallhub.azurecr.io" \ --set controller.image.image="cgr.dev/ingress-nginx-controller" \ --set controller.image.tag=1.14.5-nginx.1.27 \ --set controller.admissionWebhooks.patch.image.image="cgr.dev/kube-webhook-certgen" \ --set controller.admissionWebhooks.patch.image.tag=1.14.5 \ --set controller.image.digest="" \ --set controller.admissionWebhooks.patch.image.digest="" \ --set controller.service.annotations."service\.beta\.kubernetes\.io/azure-load-balancer-health-probe-request-path"=/healthz \ --version v4.15.1 helm upgrade --install external-secrets oci://glasswallhub.azurecr.io/ghcr/external-secrets/charts/external-secrets \ --atomic \ --set imagePullSecrets[0].name=acr-secret \ --set webhook.imagePullSecrets[0].name=acr-secret \ --set certController.imagePullSecrets[0].name=acr-secret \ --set image.repository="glasswallhub.azurecr.io/cgr.dev/external-secrets" \ --set image.tag=2.8.0 \ --set webhook.image.repository="glasswallhub.azurecr.io/cgr.dev/external-secrets" \ --set webhook.image.tag=2.8.0 \ --set certController.image.repository="glasswallhub.azurecr.io/cgr.dev/external-secrets" \ --set certController.image.tag=2.8.0 \ --version 2.8.0 \ --set installCRDs=true ``` >**Note:** when upgrading the `rabbitmq-cluster-operator` Helm chart CRDs will not be installed. If new CRDs are introduced in the new releases, the CRDs needs to be installed manually to avoid issues with running the RabbitMQ cluster operator pods. ```sh helm pull oci://glasswallhub.azurecr.io/docker/bitnamicharts/rabbitmq-cluster-operator --untar kubectl apply -f rabbitmq-cluster-operator/crds/ ``` * Then, install the supporting components. ```sh helm upgrade --install cdrplatform-storage cdrplatform-storage helm upgrade --install cdrplatform-rabbitmq cdrplatform-rabbitmq \ --set image.registry=glasswallhub.azurecr.io \ --set image.tag="2.21.0-201948" \ --set cloud_provider=azure ``` For the next step, select the same method which you used to configure your access to the Key Vault in Step 4 (Managed identity or Service principal), and follow the corresponding steps below to configure external secrets and install the secret synchronization. ## 7A - Managed identity > * Obtain the Kubelet Managed Identity Client ID via the following command. > > ```sh > az aks show -g "${rgp}" -n "${aksname}" > ``` > > * Find the same element as last time "identityProfile/kubeletidentity" but this time you need the Client ID value. > > * Next, install the supporting components (ensuring `${kvname}` is substituted for the Key Vault name and `${MIclientID}` is replaced with the value sourced above): > > ```sh > helm upgrade --install cdrplatform-external-secrets cdrplatform-external-secrets \ > --set cloud_providers.azurekv.enabled=true \ > --set cloud_providers.azurekv.vaultUrl="https://${kvname}.vault.azure.net" \ > --set cloud_providers.azurekv.clientId=${MIclientID} > ``` US Government Users Use the following command when deploying to Azure US Government. These settings ensure the correct endpoints and environment values are used: ```bash helm upgrade --install cdrplatform-external-secrets cdrplatform-external-secrets \ --set cloud_providers.azurekv.enabled=true \ --set cloud_providers.azurekv.vaultUrl="https://${kvname}.vault.usgovcloudapi.net" \ --set cloud_providers.azurekv.clientId=${MIclientID} \ --set cloud_providers.azurekv.environmentType=USGovernmentCloud ``` ***OR*** ## 7B - Service principal > In addition to the '`ClientID`' and '`ClientSecret`' you set in the previous Service principal step, you will also need to enter the '`tenantId`' of the Service principal. The '`tenantID`' can be found in Azure Active Directory. > > * Once obtained, enter the following command to install cdrplatform external secrets to use the desired Service principal. > > ```sh > helm upgrade --install cdrplatform-external-secrets cdrplatform-external-secrets --create-namespace \ > --set cloud_providers.azurekv.enabled=true \ > --set cloud_providers.azurekv.authType=ServicePrincipal \ > --set cloud_providers.azurekv.tenantId=${tenantid} \ > --set cloud_providers.azurekv.authSecretRef.clientId.name=keyvault-service-principal \ > --set cloud_providers.azurekv.authSecretRef.clientId.key=ClientID \ > --set cloud_providers.azurekv.authSecretRef.clientSecret.name=keyvault-service-principal \ > --set cloud_providers.azurekv.authSecretRef.clientSecret.key=ClientSecret \ > --set cloud_providers.azurekv.vaultUrl="https://"${kvname}".vault.azure.net" > ``` Azure US Government Users Use the following command when deploying to Azure US Gov. These settings ensure the correct endpoints and environment values are used ```bash helm upgrade --install cdrplatform-external-secrets cdrplatform-external-secrets --create-namespace \ --set cloud_providers.azurekv.enabled=true \ --set cloud_providers.azurekv.authType=ServicePrincipal \ --set cloud_providers.azurekv.tenantId=${tenantid} \ --set cloud_providers.azurekv.authSecretRef.clientId.name=keyvault-service-principal \ --set cloud_providers.azurekv.authSecretRef.clientId.key=ClientID \ --set cloud_providers.azurekv.authSecretRef.clientSecret.name=keyvault-service-principal \ --set cloud_providers.azurekv.authSecretRef.clientSecret.key=ClientSecret \ --set cloud_providers.azurekv.vaultUrl="https://"${kvname}".vault.usgovcloudapi.net" \ --set cloud_providers.azurekv.environmentType=USGovernmentCloud ``` * * * Continue Need help? Follow the steps below to complete the Amazon EC2 deployment and setup. Please [contact us](https://www.glasswall.com/contact/) if you have any questions or require further support. ## Glasswall Halo Installation via Amazon EC2 - [**Step 1 – Create Instance**](/halo/aws-ec2-step-1) - [**Step 2 – Configure Instance**](/halo/aws-ec2-step-2) - [**Step 3 – TLS/SSL Configuration**](/halo/aws-ec2-step-3) - [**Step 4 – Authentication**](/halo/aws-ec2-step-4) Let's get started → To deploy an EC2 instance with Glasswall Halo using the Glasswall Halo AMI, follow the steps below. ## Prerequisites - AWS account - Permissions to deploy an EC2 instance - Basic knowledge of Linux ## Create instance ### Steps to deploy an EC2 instance with Glasswall Halo 1. Log in to AWS. - Navigate to the EC2 service and click **Launch instance**. 2. Name the instance. - Enter a name for your instance, e.g., my Halo instance. 3. Select the Glasswall Halo AMI. - Click **Browse more AMIs**, search for Glasswall Halo, and select the **Glasswall Halo 2.5.4** AMI. 4. Choose instance type. - Select an instance type with at least **16 cores and 32 GB memory**, e.g., **c5.4xlarge**. 5. Set up a key pair. - Select or create a key pair to securely log in to the instance using a private key. 6. Configure network settings. - Choose your desired **VPC** and **Subnet**. 7. Set up security group. - Create a security group to allow inbound access on: - **Port 22** for SSH - **Port 443** for the Halo API 8. Update storage. - Increase storage to **500 GB**. 9. .Launch the instance. - Click `launch instance` to complete the setup. * * * Proceed to Step 2 → ## Configure Halo Ec2 instance ### Step 1 - Health check 1. After launching the EC2 instance, wait 15 minutes for the Halo services to start. 2. Visit `https://` in a browser to check if the Glasswall Halo Portal is running. 3. Access Halo APIs via the Swagger portal at `https:///swagger/index.html`. You are now ready to use the Glasswall Halo Portal and API. ### Step 2 - License Management Set up a license by following the steps at [Glasswall License Management Guide](/halo/license-management-api-setup-guide). ### Step 3 - Troubleshooting #### What should I do if the Halo Portal and API are not accessible? 1. If you receive timeout errors, verify that the instance security groups allow inbound access on the required ports. 2. For other errors, SSH into the instance using the `glasswall` username and the private key used during instance creation. 3. Run the following command to confirm that all pods are running. ```shell kubectl get pods ``` #### What if the Embedded Engine pods aren't running as expected? The `cloud-init` config sets the number of Engine pods based on the VM's CPU cores. If it doesn’t finish on time, run the commands below to retry. ```sh sudo cloud-init clean sudo reboot now ``` * * * Proceed to Step 3 → ## Using SSL certificates ### Steps to configure HTTPs access to Glasswall Halo APIs using SSL certificates 1. Copy your `.key` and `.crt` files to the home directory of the virtual machine. 2. Run the `configure_tls.sh` script from the same directory, providing the path to the private key, certificate, and domain name: ```bash bash configure_tls.sh private_key_path certificate_path domain_name ``` Example: ```bash bash configure_tls.sh my-key.key my-cert.crt mydomain.com ``` 3. If prompted to overwrite existing TLS secrets, type `y`. 4. The script will finish with the message: *TLS certificates configured for Synchronous API and Portal*. * * * Proceed to Step 4 → **Note:** this page is for integrating the Halo Portal SSO login with Azure Entra ID. ## Prerequisites - To use SSO, select a domain for the Portal. - Identify the `tenant_id` for the desired AWS tenant. - Ensure the [Azure CLI](https://learn.microsoft.com/en-us/cli/azure/install-azure-cli) is installed in a machine and login using `az login`. - Run the [attached shell script](/.attachments/create-azure-app-registrations.sh) to create 3 app registrations and enterprise applications. Make a note of the outputs from the script which will be used in the next steps: - cdrplatform-api-access - cdrplatform-portal-access - cdrplatform-portal-client ```sh bash create-azure-app-registrations.sh cleanroom.glasswall.com ``` ## Portal authentication installation To set up SSO with AWS in Glasswall Halo's portal: 1. SSH to the VM to run the below commands. - **Note:** The cdrplatform-portal and cdrplatform-portal-access helm charts are present in the `/home/glasswall` directory. 2. Find the image tag of the portal in the cluster and set it as the`image_tag`Variable. ```sh k get deploy portal -o json | jq -r '.spec.template.spec.containers[0].image' | cut -d":" -f2 ``` 3. Deploy the portal with AWS IAM settings, making sure to assign the correct values to the variables below. ```sh tenant_id="" portal_domain="" portal_client_id="" portal_access_uri="" image_tag="" helm upgrade --install cdrplatform-portal cdrplatform-portal \ --set image.repository=glasswallacr.azurecr.io/cdrplatform-portal \ --set image.tag="${image_tag:?}" \ --set image.pullPolicy=IfNotPresent \ --set ingress.tls.enabled=true \ --set ingress.tls.domain=${portal_domain:?} \ --set ingress.tls.secretName=tls-secret \ --set cloud_provider=local \ --set resources.requests.cpu=500m \ --set resources.requests.memory=500Mi \ --set resources.limits.cpu=500m \ --set resources.limits.memory=500Mi \ --set securityContext.seccompProfile.type=RuntimeDefault \ --set configuration.BackendScope="${portal_access_uri}/PortalUserScope" \ --set configuration.BackendUrl="https://${portal_domain}" \ --set configuration.EnabledPages="SystemSettings\,PolicySettings" \ --set configuration.OIDC.ProviderOptions.Authority="https://login.microsoftonline.com/${tenant_id:?}/v2.0" \ --set configuration.OIDC.ProviderOptions.RedirectUri="https://${portal_domain}/authentication/login-callback" \ --set configuration.OIDC.ProviderOptions.ClientId="${portal_client_id}" \ --set configuration.OIDC.ProviderOptions.PostLogoutRedirectUri="https://${portal_domain}/authentication/logout-callback" \ --atomic ``` 4. Find the image tag of portal-access in the cluster and set it as the`image_tag`Variable. ```sh k get deploy portal-access -o json | jq -r '.spec.template.spec.containers[0].image' | cut -d":" -f2 ``` 5. Deploy portal access with Azure AD configuration: ```sh tenant_id="" portal_domain="" portal_access_uri="" image_tag="" helm upgrade --install cdrplatform-portal-access cdrplatform-portal-access \ --set image.repository=glasswallacr.azurecr.io/cdrplatform-portal-access \ --set image.tag="${image_tag:?}" \ --set image.pullPolicy=IfNotPresent \ --set ingress.tls.enabled=true \ --set ingress.tls.domain=${portal_domain:?} \ --set ingress.tls.secretName=tls-secret \ --set cloud_provider=local \ --set resources.requests.cpu=1 \ --set resources.requests.memory=2Gi \ --set resources.limits.cpu=1 \ --set resources.limits.memory=2Gi \ --set securityContext.seccompProfile.type=RuntimeDefault \ --set configuration.AuthenticationScheme=Bearer \ --set configuration.Authentication__Schemes__Bearer__ValidAudiences__0="${portal_access_uri}" \ --set configuration.Authentication__Schemes__Bearer__ValidIssuer=https://sts.windows.net/${tenant_id:?}/ \ --set configuration.Authentication__Schemes__Bearer__Authority=https://login.microsoftonline.com/${tenant_id}/v2.0/ \ --atomic ``` 6. Open the portal domain in a browser and click **Login with SSO** in the bottom left. 7. Log in through Azure AD and grant the app permissions for the organization. ## API authentication API authentication can be configured in 2 ways: - [Basic authentication](#basic-authentication-installation) - [Bearer authentication](#bearer-authentication-installation) ### Basic authentication installation 1. SSH to the VM to run the commands below. **Note:** the cdrplatform-api-access Helm chart is present in the `/home/glasswall` directory. 2. Set the **Username** and **Password** in the command below. Use commas to separate multiple passwords. ```sh bash add_secrets.sh organisation0-id bash add_secrets.sh organisation0-tokens ``` 3. Deploy API-access with basic authentication. ```sh image_tag=$(k get deploy api-access -o json | jq -r '.spec.template.spec.containers[0].image' | cut -d":" -f2) enable_tls="true" OR "false" api_domain="" # ignore if enable_tls is false helm upgrade --install cdrplatform-api-access cdrplatform-api-access \ --set image.tag="${image_tag:?}" \ --set image.pullPolicy=IfNotPresent \ --set image.repository="glasswallacr.azurecr.io/cdrplatform-api-access" \ --set ingress.tls.enabled="${enable_tls:?}" \ --set ingress.tls.domain="${api_domain}" \ --set configuration.CLIENTS__Policy__BaseAddress="http://policy-api:8080" \ --set configuration.CLIENTS__License__BaseAddress="http://license-management.license-management.svc.cluster.local:8080" \ --set configuration.CLIENTS__AsyncApi__BaseAddress="http://async-api:8080" \ --set configuration.AuthenticationScheme="Basic" \ --set cloud_provider=local \ --set resources.requests.cpu=1 \ --set resources.requests.memory=3Gi \ --set resources.limits.cpu=1 \ --set resources.limits.memory=3Gi \ --set securityContext.seccompProfile.type=RuntimeDefault \ --atomic ``` ### Bearer authentication installation 1. Identify the `tenant_id` for the desired Azure tenant. 2. SSH to the VM to run the commands below. **Note:** the cdrplatform-API-access Helm chart should be present in the `/home/glasswall` directory. 4. Deploy API-access with Azure AD configuration. ```sh tenant_id="" api_valid_audience="api://cdrplatform-api-access" image_tag=$(k get deploy api-access -o json | jq -r '.spec.template.spec.containers[0].image' | cut -d":" -f2) enable_tls="true" OR "false" api_domain="" # ignore if enable_tls is false helm upgrade --install cdrplatform-api-access cdrplatform-api-access \ --set image.tag="${image_tag}" \ --set image.pullPolicy=IfNotPresent \ --set image.repository="glasswallacr.azurecr.io/cdrplatform-api-access" \ --set ingress.tls.enabled="${enable_tls}" \ --set ingress.tls.domain="${api_domain}" \ --set configuration.CLIENTS__Policy__BaseAddress="http://policy-api:8080" \ --set configuration.CLIENTS__License__BaseAddress="http://license-management.license-management.svc.cluster.local:8080" \ --set configuration.CLIENTS__AsyncApi__BaseAddress="http://async-api:8080" \ --set configuration.AuthenticationScheme="Bearer" \ --set configuration.Authentication__Schemes__Bearer__ValidAudiences__0="${api_valid_audience}" \ --set configuration.Authentication__Schemes__Bearer__ValidIssuer=https://sts.windows.net/${tenant_id}/ \ --set configuration.Authentication__Schemes__Bearer__Authority=https://login.microsoftonline.com/${tenant_id}/v2.0/ \ --set cloud_provider=local \ --set resources.requests.cpu=1 \ --set resources.requests.memory=3Gi \ --set resources.limits.cpu=1 \ --set resources.limits.memory=3Gi \ --set securityContext.seccompProfile.type=RuntimeDefault \ --atomic ``` * * * Congratulations, you have successfully deployed Glasswall Halo! Glasswall Halo deployment and upgrades are carried out via the use of Helm charts. Glasswall provides customers with a secure token to access the Helm charts and all associated container images. You can follow the steps below to complete the EKS deployment and setup, but please [contact us](https://www.glasswall.com/contact/) if you have any questions or require more support. ## Glasswall Halo installation via Elastic Kubernetes Service (EKS) * [**Prerequisites**](/halo/eks-prerequisites) * [**Step 1 - Set your Kubernetes context to EKS**](/halo/eks-step-1) * [**Step 2 - Create Glasswall Halo namespace**](/halo/eks-step-2) * [**Step 3 - Create secrets in Secrets Manager**](/halo/eks-step-3) * [**Step 4 - Enable access to Glasswall's Artifact Registry**](/halo/eks-step-4) * [**Step 5 - Pull Helm charts**](/halo/eks-step-5) * [**Step 6 - Install and configure prerequisites**](/halo/eks-step-6) * [**Step 7 - Install CDR components**](/halo/eks-step-7) Let's get started → Before you begin the process of deploying Glasswall Halo, ensure that you have the following tools and resources installed and setup. ## System requirements The following guide has been written assuming it is being run on a Linux shell. If running on a different shell, some of the commands may need to be modified to run. If running on Windows, please ensure you are using Windows subsystem for Linux. ## Required tools * [Helm](https://helm.sh/docs/intro/install/) * [Kubectl](https://kubernetes.io/docs/tasks/tools/) * [AWS CLI](https://aws.amazon.com/cli/) ## Required AWS resources ### 1. EKS cluster * Recommended total of at least 8 vCPU and 32 GB RAM. * Minimum node size is 4 vCPU and 16 GB RAM. * For production workloads a minimum of 2 nodes is recommended. * Make sure the EKS cluster has an IAM OIDC provider. * Please refer to [AWS documentation](https://docs.aws.amazon.com/eks/latest/userguide/enable-iam-roles-for-service-accounts.html) to determine if the cluster has an IAM OIDC provider or if you need to create one. * In the steps below, the EKS cluster is referred to as: **eksname** **Note** : Glasswall Halo does not support ARM64 node VMs. For guidance on creating an EKS cluster please refer to: * [Create EKS Cluster - Console](https://docs.aws.amazon.com/eks/latest/userguide/create-cluster.html) * [Create EKS Cluster - CLI ](https://docs.aws.amazon.com/cli/latest/reference/eks/create-cluster.html) * [AWS Elastic Kubernetes Service - Best Practices](https://aws.github.io/aws-eks-best-practices/) ### 2. EFS An AWS Elastic File System (EFS) is used to create persistent volumes in EKS. * Create the EFS in the same VPC where the EKS is created. For guidance on creating EFS please refer to: * [Create EFS - Console ](https://docs.aws.amazon.com/efs/latest/ug/gs-step-two-create-efs-resources.html) * [Create EFS - CLI](https://docs.aws.amazon.com/cli/latest/reference/efs/index.html) ### 3. IAM roles for service accounts Two IAM roles need to be created for 2 Kubernetes service accounts deployed through the Helm charts. The Amazon Resource Names (ARN) of these roles are required during deployment of the Helm charts. >- **Role 1:** role-cdrp-efs-csi-`` >- **Role 2:** role-cdrp-ext-secrets-`` - This [shell script](/.attachments/create-aws-roles.sh) can be used to create the IAM roles. - Pass AWS region, EKS cluster name, filesystem ID and suffix as 4 arguments to the script. for e.g. ```bash bash create-aws-roles.sh eu-west-1 eks-cdrp-dev fs-582a03f3 dev ``` For guidance on creating IAM roles for service accounts, please refer to [AWS documentation](https://docs.aws.amazon.com/eks/latest/userguide/iam-roles-for-service-accounts.html). ### 4. MongoDB database MongoDB stores Glasswall Halo content management policies and is used by the `cdrplatform-policy-api` service. - Deploy a DocumentDB instance that is compatible with the MongoDB API. >- You can skip this step if you do not require the Policy Management API or the Asynchronous API. - If DocumentDB is not available in AWS GovCloud, use the MongoDB Kubernetes Operator to deploy MongoDB in the EKS cluster. **Note:** alternatively, MongoDB can be deployed directly within your cluster using the MongoDB Helm charts, as described in [Step 7](/halo/eks-step-7). ## Access to Glasswall Artifact Registry * You are provided with a token & token ID to access Glasswall's Artifact Registry (`glasswallhub.azurecr.io`). * This allows you to directly pull container images and Helm charts from your AKS cluster. * In the steps below, the token and token ID will be referred to as: **token** and **token_id**. * * * Continue Need help? The following steps require the use of [Amazon Web Services (AWS) CLI](https://AWS.amazon.com/cli/). - There are various methods of logging into the AWS CLI, such as using a SSO login to your AWS [_named profile_](https://docs.AWS.amazon.com/cli/latest/userguide/cli-configure-profiles.html): >```bash >AWS sso login --profile >``` - Once you've logged in, set your Kubernetes context by running the following command: >```bash >aws eks update-kubeconfig --region \ --name \ --profile \ >``` **Note** : ensure that you replace **\** and **\** with the corresponding AWS region (e.g. _eu-west-1_) and the name of your EKS cluster. You have now merged your EKS settings into your Kubernetes configuration files (typically stored in the `~/.kube/config` file). Continue Need help? All services associated with Glasswall Halo are deployed to a namespace within the target cluster. - Create the namespace and set the default context for the subsequent commands to be this newly created namespace by running the following commands. ```sh kubectl create namespace "cdrplatform" kubectl create namespace "license-management" kubectl config set-context --current --namespace="cdrplatform" ``` Continue Need help? To enable Glasswall Halo's access to the S3 bucket (`s3name`) That contains the CDR reports, an iam user and role is required. * Create an IAM user (`external_secrets_iam_user`) And assign a role (`external_secrets_iam_role`) With policy that allows read and write on the s3 bucket which will store the CDR reports. this bucket should have been created in the [prerequisites](/halo/eks-prerequisites) step. * Create`AWS_ACCESS_KEY_ID`Secret that stores the iam user access key id. * Create`AWS_SECRET_ACCESS_KEY`Secret that stores the iam user secret access key. ## 3.1 - MongoDB connection string To enable the Glasswall Halo Policy Management API to create and manage policies in MongoDB, store the MongoDB connection string in AWS Secrets Manager. - You can retrieve the DocumentDB MongoDB compatible connection string from the AWS Console, as shown in the example below. >```sh >mongodb://${username}:${password}@${endpoint}:${port}/?ssl=true&ssl_ca_certs=rds->combined-ca-bundle.pem&replicaSet=rs0&readPreference=secondaryPreferred&retryWrites=false >``` - Enter your connection string (highlighted in the screenshot) by replacing `${mongodb_connstring}`, and enter your AWS region by replacing `${region}` (as below). >```sh >aws secretsmanager create-secret --name "mongodb-connectionstring" --secret-string >"${mongodb_connstring}" --region "${region}" >``` ### Add the MongoDB passwords to AWS Secret Manager as a secret **Note:** if you have previously configured and setup MongoDB within AWS and have created your MongoDB connection string as listed above, you can skip this step. If not, to enable Glasswall Halo's Policy Management API to create and manage the policies in MongoDB, and Asynchronous API to create and manage the requests, MongoDB needs to be deployed using Helm charts listed in Step 8. Two users will be created by the MongoDB Helm chart and the corresponding user's password needs to be set in the Key Vault secret. Sensitive data, such as passwords, should be in the AWS Secrets Manager. ```sh aws secretsmanager create-secret --name "mongodb-cdrp-password" --secret-string "" --region "${region}" aws secretsmanager create-secret --name "mongodb-admin-password" --secret-string "" --region "${region}" ``` ## 3.2 - Amazon DocumentDB Certificate Authority (CA) For the **cdrplatform-policy-API** service to successfully authenticate to MongoDB, it should trust the Amazon DocumentDB Certificate Authority. * Create a secret called `cdrp-rds-ca-bundle` in Secrets Manager and add the content of the [certificate authority](https://s3.amazonaws.com/rds-downloads/rds-combined-ca-bundle.pem) provided by Amazon. ## 3.3 - [Optional] Add ReversingLabs credentials To integrate Glasswall Halo with ReversingLabs, store the ReversingLabs credentials in AWS Secrets Manager. Replace `$reversinglabs_username` and `$reversinglabs_password` with the actual username and password. ```sh aws secretsmanager create-secret --name "halo-reversinglabs-username" --secret-string "${reversinglabs_username}" --region "${region}" aws secretsmanager create-secret --name "halo-reversinglabs-password" --secret-string "${reversinglabs_password}" --region "${region}" ``` ## 3.4 - [Optional] Add ICAP MTLS certificates ICAP servers can be configured for mutual client authentication using the MTLS certificates. The certificates will be mounted to the ICAP server pods using the Kubernetes secrets. Add the server certificates and certificate authority to the AWS Secrets Manager so that they can be synced to Kubernetes secrets. ```sh aws secretsmanager create-secret --name "tls-server-cert" --region "$region" --secret-string aws secretsmanager create-secret --name "tls-server-key" --region "$region" --secret-string aws secretsmanager create-secret --name "tls-cafile" --region "$region" --secret-string ``` * * * Continue Need help? [Glasswall's Artifact Registry](/halo/glasswall-artifact-registry)stores container images and Helm charts which you need to complete your Glasswall Halo installation. To pull container images and Helm charts from Glasswall Artifact Registry, you need to use your **token** and **token_id** in two commands: ```bash kubectl create secret docker-registry acr-secret -n cdrplatform \ --docker-server="glasswallhub.azurecr.io" --docker-username="${TOKEN_ID}" --docker-password="${TOKEN}" kubectl create secret docker-registry acr-secret -n license-management \ --docker-server="glasswallhub.azurecr.io" --docker-username="${token_ID}" --docker-password="${token}" ``` Login to the Helm registry using your **token_id** and **token** as shown below. ```bash helm registry login "glasswallhub.azurecr.io" -u "${TOKEN_ID}" -p "${TOKEN}" ``` See [Glasswall's Artifact Registry](/halo/glasswall-artifact-registry)for a complete list of hosted container images and Helm charts. * * * Continue Need help? Glasswall Halo consists of Glasswall Helm charts hosted in our authenticated Artifact Registry, along side a few industry recognized third-party charts. - Before the charts can be installed, the right versions need to be pulled by Helm. The required versions are listed in the corresponding release notes. - An example pull command is shown below; this needs to be executed for each Helm chart listed. ```bash helm pull oci://glasswallhub.azurecr.io/ --version --untar --untardir . ``` - The following example is pre-populated with the versions for [v2.21.0](/halo/v2210): ```sh helm pull oci://glasswallhub.azurecr.io/helm/cdrplatform-rabbitmq --version 0.9.4 --untar --untardir . helm pull oci://glasswallhub.azurecr.io/helm/cdrplatform-external-secrets --version 0.6.0 --untar --untardir . helm pull oci://glasswallhub.azurecr.io/helm/cdrplatform-storage --version 0.8.1 --untar --untardir . helm pull oci://glasswallhub.azurecr.io/helm/cdrplatform-engine --version 0.6.3 --untar --untardir . helm pull oci://glasswallhub.azurecr.io/helm/cdrplatform-sync-api --version 0.4.2 --untar --untardir . helm pull oci://glasswallhub.azurecr.io/helm/cdrplatform-portal --version 0.4.2 --untar --untardir . helm pull oci://glasswallhub.azurecr.io/helm/cdrplatform-policy-api --version 0.3.2 --untar --untardir . helm pull oci://glasswallhub.azurecr.io/helm/cdrplatform-api-access --version 1.2.3 --untar --untardir . helm pull oci://glasswallhub.azurecr.io/helm/cdrplatform-portal-access --version 0.2.2 --untar --untardir . helm pull oci://glasswallhub.azurecr.io/helm/cdrplatform-license-management --version 0.4.0 --untar --untardir . helm pull oci://glasswallhub.azurecr.io/helm/cdrplatform-cleanup --version 0.3.4 --untar --untardir . helm pull oci://glasswallhub.azurecr.io/helm/cdrplatform-async-api --version 0.2.3 --untar --untardir . helm pull oci://glasswallhub.azurecr.io/helm/prometheus-scaling --version 0.3.1 --untar --untardir . helm pull oci://glasswallhub.azurecr.io/helm/cdrplatform-metrics-projection --version 0.2.2 --untar --untardir . helm pull oci://glasswallhub.azurecr.io/helm/cdrplatform-report-extractor --version 0.2.2 --untar --untardir . helm pull oci://glasswallhub.azurecr.io/helm/cdrplatform-tally-accumulator --version 0.2.4 --untar --untardir . helm pull oci://glasswallhub.azurecr.io/helm/cdrplatform-storage-monitor --version 0.2.5 --untar --untardir . helm pull oci://glasswallhub.azurecr.io/helm/cdrplatform-mongodb --version 0.3.0 --untar --untardir . ``` In order to gain access to the repositories from which our prerequisite components are installed - they must be added: ```sh helm repo add aws-efs-csi-driver https://kubernetes-sigs.github.io/aws-efs-csi-driver helm repo add mongodb https://mongodb.github.io/helm-charts helm repo update ``` * * * Continue Need help? ## Gather role ARNs If you have the role ARNs from the roles created in Prerequisites step, you can skip this part. Instead, if you have role names, please use below commands to get the role ARNs. Make sure to set the `prefix` and `profile` in the below commands before running them. ```bash prefix="" profile="" external_secrets_iam_role_arn=$(aws iam get-role --role-name role-cdrp-ext-secrets-${prefix} --profile ${profile} --query 'Role.Arn') echo "external_secrets_iam_role_arn=${external_secrets_iam_role_arn}" efs_iam_role_arn=$(aws iam get-role --role-name role-cdrp-efs-csi-${prefix} --profile ${profile} --query 'Role.Arn') echo "external_secrets_iam_role_arn=${efs_iam_role_arn}" ``` Now that you have access to the Helm charts, they can be deployed. - First install the prerequisite components: ```sh # Install RabbitMQ cluster operator helm upgrade --install rabbitmq-cluster-operator oci://glasswallhub.azurecr.io/docker/bitnamicharts/rabbitmq-cluster-operator \ --atomic \ --version 4.4.34 \ --set global.imageRegistry=glasswallhub.azurecr.io \ --set global.imagePullSecrets[0]=acr-secret \ --set global.security.allowInsecureImages=true \ --set msgTopologyOperator.fullnameOverride=rabbitmq-messaging-topology-operator \ --set clusterOperator.image.repository="cgr.dev/rabbitmq-cluster-operator" \ --set clusterOperator.image.tag=2.17.0 \ --set msgTopologyOperator.image.repository="cgr.dev/rabbitmq-messaging-topology-operator" \ --set msgTopologyOperator.image.tag=1.18.3 \ --set credentialUpdaterImage.repository="cgr.dev/rabbitmq-default-user-credential-updater" \ --set credentialUpdaterImage.tag=1.0.16 \ --set rabbitmqImage.repository="cgr.dev/rabbitmq" \ --set rabbitmqImage.tag=4.2.9 \ --set clusterOperator.watchAllNamespaces=false \ --set clusterOperator.watchNamespaces={cdrplatform} \ --set msgTopologyOperator.watchAllNamespaces=false \ --set msgTopologyOperator.watchNamespaces={cdrplatform} \ --set clusterOperator.resources.requests.cpu=100m \ --set clusterOperator.resources.requests.memory=256Mi \ --set clusterOperator.resources.limits.cpu=100m \ --set clusterOperator.resources.limits.memory=256Mi \ --set msgTopologyOperator.resources.requests.cpu=100m \ --set msgTopologyOperator.resources.requests.memory=256Mi \ --set msgTopologyOperator.resources.limits.cpu=100m \ --set msgTopologyOperator.resources.limits.memory=256Mi # Install KEDA helm upgrade --install keda "oci://glasswallhub.azurecr.io/ghcr/home-operations/charts-mirror/keda" --atomic \ --namespace cdrplatform \ --set imagePullSecrets[0].name=acr-secret \ --set global.image.registry="glasswallhub.azurecr.io" \ --set image.keda.repository="cgr.dev/keda" \ --set image.keda.tag=2.20.2 \ --set image.metricsApiServer.repository="cgr.dev/keda-metrics-apiserver" \ --set image.metricsApiServer.tag=2.20.2 \ --set image.webhooks.repository="cgr.dev/keda-admission-webhooks" \ --set image.webhooks.tag=2.20.2 \ --version 2.20.2 # Install nginx ingress controller helm upgrade --install nginx-ingress oci://glasswallhub.azurecr.io/k8s/ingress-nginx/charts/ingress-nginx --atomic \ --set imagePullSecrets[0].name=acr-secret \ --set global.image.registry="glasswallhub.azurecr.io" \ --set controller.image.image="cgr.dev/ingress-nginx-controller" \ --set controller.image.tag=1.14.5-nginx.1.27 \ --set controller.admissionWebhooks.patch.image.image="cgr.dev/kube-webhook-certgen" \ --set controller.admissionWebhooks.patch.image.tag=1.14.5 \ --set controller.image.digest="" \ --set controller.admissionWebhooks.patch.image.digest="" \ --version v4.15.1 # Install External Secrets helm upgrade --install external-secrets oci://glasswallhub.azurecr.io/ghcr/external-secrets/charts/external-secrets \ --atomic \ --set imagePullSecrets[0].name=acr-secret \ --set webhook.imagePullSecrets[0].name=acr-secret \ --set certController.imagePullSecrets[0].name=acr-secret \ --set image.repository="glasswallhub.azurecr.io/cgr.dev/external-secrets" \ --set image.tag=2.8.0 \ --set webhook.image.repository="glasswallhub.azurecr.io/cgr.dev/external-secrets" \ --set webhook.image.tag=2.8.0 \ --set certController.image.repository="glasswallhub.azurecr.io/cgr.dev/external-secrets" \ --set certController.image.tag=2.8.0 \ --version 2.8.0 \ --set installCRDs=true # Replace ${external_secrets_iam_role_arn} with the ARN value of the role that has access to Secrets Manager, # and ${region} with your AWS region. helm upgrade --install cdrplatform-external-secrets -n cdrplatform cdrplatform-external-secrets --atomic --create-namespace \ --set cloud_providers.aws.enabled=true \ --set cloud_providers.aws.secretsManager.iam_role="${external_secrets_iam_role_arn}" \ --set cloud_providers.aws.secretsManager.region="${region}" # Replace ${efs_iam_role_arn} with the ARN value of the role for EFS CSI driver. helm upgrade -i aws-efs-csi-driver aws-efs-csi-driver/aws-efs-csi-driver \ --namespace cdrplatform \ --set image.repository=602401143452.dkr.ecr.${region}.amazonaws.com/eks/aws-efs-csi-driver \ --set controller.serviceAccount.create=true \ --set controller.serviceAccount.name=sa-efs-csi-controller \ --set controller.serviceAccount.annotations."eks\.amazonaws\.com/role-arn"="${efs_iam_role_arn}" ``` - Note, when upgrading the `rabbitmq-cluster-operator` Helm chart CRDs will not be installed. If new CRDs are introduced in the new releases, the CRDs need to be installed manually to avoid issues with running the RabbitMQ cluster operator pods. ```sh helm pull oci://glasswallhub.azurecr.io/docker/bitnamicharts/rabbitmq-cluster-operator --untar kubectl apply -f rabbitmq-cluster-operator/crds/ ``` - Then, install the supporting components: ```sh # Replace ${file_system_id} and ${storageAmount} with values for your environment. helm upgrade --install cdrplatform-storage -n cdrplatform cdrplatform-storage \ --set cloud_provider=aws \ --set aws.efs.file_system_id=${file_system_id} \ --set storageAmount=${storageAmount} helm upgrade --install cdrplatform-rabbitmq -n cdrplatform cdrplatform-rabbitmq \ --set image.registry=glasswallhub.azurecr.io \ --set image.tag=2.21.0-201948 \ --set cloud_provider=aws \ --atomic ``` * * * Continue Need help? Finally, install the Glasswall Halo application services. For each Helm chart, ensure that the image tag is set to the corresponding version listed in the release notes. The examples below are pre populated with the tags for [v2.21.0](/halo/v2210). ## 7.1 - Engine ```sh helm upgrade --install cdrplatform-engine cdrplatform-engine \ --set image.tag=2.21.0-201948 \ --set image.registry=glasswallhub.azurecr.io \ --set cloud_provider=aws \ --timeout 10m \ --atomic ``` ### 7.1A - Enable ReversingLabs (optional) To enable ReversingLabs file reputation: 1. Store the credentials in AWS Secrets Manager under `halo-reversinglabs-username` and `halo-reversinglabs-password`, as described in [Step 3](/halo/eks-step-3). external-secrets syncs them into `cdrplatform-secrets`. 2. Set the non-secret fields on the Engine-owned `engine-reversinglabs` ConfigMap. The endpoint is supplied by you; use the value below. ```sh kubectl patch configmap engine-reversinglabs -n cdrplatform --type merge \ -p '{"data":{"enabled":"true","endpoint":"https://data.reversinglabs.com/api/databrowser/malware_presence/query","timeoutSeconds":"5"}}' ``` The Engine picks up the change on its next lookup; no restart is required. `timeoutSeconds` accepts 1–120 and defaults to 5. ## 7.2 - Synchronous API ```sh helm upgrade --install cdrplatform-sync-api cdrplatform-sync-api \ --set image.tag=2.21.0-201948 \ --set image.registry=glasswallhub.azurecr.io \ --set cloud_provider=aws ``` ## 7.3 - API Access service The API Access service acts as a gateway to the Glasswall Halo Synchronous API and the Policy Management API. It exposes CDR functionality over HTTP. If your environment requires HTTPS with TLS or SSL, follow the instructions to install the service with TLS or SSL enabled. Otherwise, follow the instructions for installation without TLS or SSL. ### 7.3A - for deployments without tls/ssl ```sh helm upgrade --install cdrplatform-api-access -n cdrplatform cdrplatform-api-access \ --set image.registry=glasswallhub.azurecr.io \ --set image.tag=2.21.0-201948 \ --atomic ``` ### 7.3B - Deployments with TLS or SSL To use SSL certificates with the CDR API, create a private key and certificate for the required domain. Create a Kubernetes secret from the private key and certificate files using the command below. This command creates a secret named `tls-secret` using `server.key` as the private key and `server.crt` as the certificate. In this example, the private key must not be passphrase protected. ```sh kubectl create secret tls tls-secret --key server.key --cert server.crt ``` This secret can then be used to enable TLS on the ingress ensuring that the domain name is set on the command below: ```sh helm upgrade --install cdrplatform-api-access -n cdrplatform cdrplatform-api-access \ --set image.registry=glasswallhub.azurecr.io \ --set image.tag=2.21.0-201948 \ --set cloud_provider=aws \ --set ingress.tls.enabled=true \ --set ingress.tls.domain=${domain_name} \ --set ingress.tls.secretName=tls-secret \ --atomic ``` ### 7.3C - Enable API authentication Authentication for the Glasswall Halo API is disabled by default. To enable authentication, complete the steps below. - Create two secrets in AWS Secrets Manager. - One secret for the organisation ID - One secret for the organisation tokens When specifying multiple tokens, separate them with commas. Ensure that individual tokens do not contain commas. - Secrets in AWS Secrets Manager must follow the naming conventions below. - **Organisation ID secret** - Must start with `organisation` - Must end with `-id` - Include a numeric index between the prefix and suffix - Examples: `organisation0-id`, `organisation1-id`, `organisation2-id` - **Organisation tokens secret** - Must start with `organisation` - Must end with `-tokens` - Include a numeric index between the prefix and suffix - Examples: `organisation0-tokens`, `organisation1-tokens`, `organisation2-tokens` - Set `configuration.AuthenticationScheme=Basic` while deploying the Helm chart. For example: ```sh helm upgrade --install cdrplatform-api-access -n cdrplatform cdrplatform-api-access \ --set image.registry=glasswallhub.azurecr.io \ --set image.tag=2.21.0-201948 \ --set configuration.AuthenticationScheme=Basic \ --atomic ``` ### 7.4 - MongoDB **Note:** if you have previously configured and setup MongoDB within AWS and if you do not wish to manage and install MongoDB within your cluster, you can skip this step, and proceed to installing the Portal Services below. ### MongoDB operator ```sh helm install community-operator mongodb/community-operator --namespace cdrplatform \ --set operator.version=0.9.0 \ --set agent.version=107.0.0.8465-1 \ --atomic ``` ### Install MongoDB ```sh helm upgrade -i cdrplatform-mongodb cdrplatform-mongodb -n cdrplatform --atomic \ --set cloud_provider=aws ``` After MongoDB is deployed, retrieve the MongoDB connection string and store it in AWS Secrets Manager. - Retrieve the connection string from the k8s secret ```sh mongodb_conn_str=$(kubectl get secret mongodb-cdrplatform-cdrp-user -o jsonpath='{.data.connectionString\.standard}' | base64 -d) ``` - Create a new secret in the AWS Secrets Manager with the MongoDB connection string. ```sh aws secretsmanager create-secret --name "mongodb-connectionstring" --secret-string "${mongodb_conn_str}" --region "${region}" ``` ## 7.5 - Portal To deploy the Portal service, run the commands below. ### 7.5A - Deployments without TLS or SSL **Note:** The `` referenced in these commands is the public IP address of the load balancer. This can be retrieved by following the **Portal and API Access** steps below. ```sh helm upgrade --install cdrplatform-portal cdrplatform-portal \ --set image.registry=glasswallhub.azurecr.io \ --set configuration.OIDC=null \ --set image.tag=2.21.0-201948 \ --set configuration.BackendUrl=http:// \ --set configuration.HaloVersion=2.21.0 \ --atomic ``` ### 7.5B - Deployments with TLS or SSL If TLS is required, add the `--set ingress.tls.enable_tls=true` and `--set ingress.tls.domain=` parameters. In the example below, the same Kubernetes secret created for the API Access service is used to retrieve the TLS certificates. ```sh helm upgrade --install cdrplatform-portal cdrplatform-portal \ --set image.tag=2.21.0-201948 \ --set image.registry=glasswallhub.azurecr.io \ --set ingress.tls.enabled=true \ --set ingress.tls.domain=${domain_name} \ --set ingress.tls.secretName=tls-secret \ --set configuration.BackendUrl=https://${domain_name} \ --set configuration.OIDC=null \ --set configuration.HaloVersion=2.21.0 \ --atomic ``` ### 7.5C - Enable admin access without authentication When Single Sign On login is not enabled for the Portal, administrative features such as license management and policy configuration are not accessible. To enable access to these administrative features without configuring SSO, deploy the Helm chart with the following configuration. ```sh helm upgrade --install cdrplatform-portal cdrplatform-portal \ --set image.tag=2.21.0-201948 \ --set image.registry=glasswallhub.azurecr.io \ --set ingress.tls.enabled=true \ --set ingress.tls.domain=${portal_domain} \ --set ingress.tls.secretName=tls-secret \ --set configuration.AutoAdmin="true" \ --atomic ``` ### 7.5D - Configure Halo Portal to authenticate APIs If API authentication is enabled, replace `${organisation1_id}` and `${organisation1_tokens}` with the appropriate secret names created in AWS Secrets Manager. ```sh helm upgrade --install cdrplatform-portal cdrplatform-portal \ --set image.tag=2.21.0-201948 --set image.registry=glasswallhub.azurecr.io \ --set configuration.EnableAuth=true \ --set AuthSecrets.orgIdKey=${organisation1-id} \ --set AuthSecrets.orgTokenKey=${organisation1-tokens} \ --atomic ``` ## 7.6 - Policy Management API The Policy Management API is used to manage policies for Glasswall Halo content management flags. This is an optional service, install this if you would like to create and use custom policies. ```sh helm upgrade --install cdrplatform-policy-api cdrplatform-policy-api \ --set image.registry=glasswallhub.azurecr.io \ --set image.tag=2.21.0-201948 --set cloud_provider=aws \ --atomic ``` Policy API swagger page can be accessed using ``` http:///swagger/index.html ``` ## 7.7 - Portal access The Portal Access service acts as a backend for Portal. It enables the Portal to access the Policy Management API and Synchronous API. ### 7.7A - Deployments with TLS or SSL If TLS is required, add the `--set ingress.tls.enable_tls=true` and `--set ingress.tls.domain=` parameters. In the example below, the same Kubernetes secret created for the API Access service is used to retrieve the TLS certificates. ```sh helm upgrade --install cdrplatform-portal-access cdrplatform-portal-access \ --set image.registry=glasswallhub.azurecr.io \ --set image.tag=2.21.0-201948 \ --set ingress.tls.enabled=true \ --set ingress.tls.domain=${domain_name} \ --set ingress.tls.secretName=tls-secret \ --set configuration.AuthenticationScheme=None \ --atomic ``` ### 7.7B - for deployments without TLS/SSL ```sh helm upgrade --install cdrplatform-portal-access cdrplatform-portal-access \ --set image.registry=glasswallhub.azurecr.io \ --set configuration.AuthenticationScheme=None \ --set image.tag=2.21.0-201948 \ --atomic ``` ## 7.8 - License management The License management service is used to manage licenses in Glasswall Halo. ```sh helm upgrade --install cdrplatform-license-management cdrplatform-license-management -n license-management \ --set image.registry=glasswallhub.azurecr.io \ --set image.tag=2.21.0-201948 \ --set cloud_provider=aws \ --atomic ``` ## 7.9 - Clean up service The Clean up service deletes the original and rebuilt files from the persistent storage after the files are processed. ```sh helm upgrade --install cdrplatform-cleanup cdrplatform-cleanup \ --set image.registry=glasswallhub.azurecr.io \ --set image.tag=2.21.0-201948 \ --set cloud_provider=aws \ --atomic ``` ## 7.10 - Asynchronous API The Asynchronous API can be deployed using the command below. A MongoDB database is a prerequisite for the Async API. ```sh helm upgrade --install cdrplatform-async-api cdrplatform-async-api \ --set image.tag=2.21.0-201948 \ --set image.registry=glasswallhub.azurecr.io \ --set cloud_provider=aws \ --atomic ``` ## 7.11 - Metrics projection The Metrics projection service is used to pull reporting data from MongoDB to display it in the Portal UI. ```sh helm upgrade --install cdrplatform-metrics-projection cdrplatform-metrics-projection \ --set image.tag=2.21.0-201948 \ --set image.registry=glasswallhub.azurecr.io \ --set cloud_provider=aws \ --atomic ``` ## 7.12 - Report extractor The Report extractor service extracts analysis reports and publishes them for reporting. ```sh helm upgrade --install cdrplatform-report-extractor cdrplatform-report-extractor \ --set image.tag=2.21.0-201948 \ --set image.registry=glasswallhub.azurecr.io \ --set cloud_provider=aws \ --atomic ``` ## 7.13 - Tally accumulator The Tally accumulator service tracks and maintains a tally of usage statistics. ```sh helm upgrade --install cdrplatform-tally-accumulator cdrplatform-tally-accumulator \ --set image.tag=2.21.0-201948 \ --set image.registry=glasswallhub.azurecr.io \ --set cloud_provider=aws \ --atomic ``` ## 7.14 - Storage monitoring service The Storage Monitoring service provides an API to subscribe to and track changes made to documents in configured storage services such as SharePoint. TLS must be enabled for the webhook. Add the `--set ingress.tls.enable_tls=true` and `--set ingress.tls.domain=` parameters. In the example below, the same Kubernetes secret created for the API Access service is used to retrieve the TLS certificates. ```sh helm upgrade --install cdrplatform-storage-monitor cdrplatform-storage-monitor --wait --atomic \ --set image.tag="2.21.0-201948" \ --set image.registry="glasswallhub.azurecr.io" \ --set configuration.DATABASE__Provider="${database_provider}" \ --set configuration.WEBHOOKS__CallbackBaseAddress="https://${domain_name}" \ --set ingress.tls.enabled=true \ --set ingress.tls.domain=${domain_name} \ --set ingress.tls.secretName=tls-secret \ --atomic ``` ## 7.15 - Optional Prometheus scaling Install the Prometheus scaling chart only if you want to enable Prometheus-based scaling. ```sh helm upgrade --install prometheus-scaling prometheus-scaling --wait --atomic \ --set image.registry=glasswallhub.azurecr.io \ --set image.tag=3.13.2-full \ --set imagePullSecrets[0].name=acr-secret \ --set keda.icap.enabled= ``` ## 7.16 - Portal & API access Use the command below to determine the _external-ip_ associated with your cluster: **Note:** _external-ip_ will be the same as the public ip address attached to the AWS load balancer. ``` kubectl get services --namespace cdrplatform nginx-ingress-ingress-nginx-controller --output jsonpath='{.status.loadBalancer.ingress[0].hostname}' ``` You can now use the IP returned above to navigate to the Portal and API documentation (use https if TLS enabled): ``` Portal: http:// API documentation: http:///swagger ``` * * * Need help → Glasswall Halo deployment and upgrades are carried out via the use of Helm charts. Glasswall provides customers with a secure token to access the Helm charts and all associated container images. You can follow the steps below to complete the GKE deployment and setup, but please [contact us](https://www.glasswall.com/contact/) if you have any questions or require more support. ## Glasswall Halo installation via google Kubernetes Engine (GKE) * [**Prerequisites**](/halo/gke-prerequisites) * [**Step 1 - Set your Kubernetes context to GKE**](/halo/gke-step-1) * [**Step 2 - Create Glasswall Halo namespace**](/halo/gke-step-2) * [**Step 3 - Enable your GKE cluster to access secrets**](/halo/gke-step-3) * [**Step 4 - Manage secrets**](/halo/gke-step-4) * [**Step 5 - Enable access to Glasswall's artifact registry**](/halo/gke-step-5) * [**Step 6 - Pull Helm charts**](/halo/gke-step-6) * [**Step 7 - Install and configure prerequisite components**](/halo/gke-step-7) * [**Step 8 - Install CDR components**](/halo/gke-step-8) Let's get started → Finally, install the CDR application services. For these charts, ensure you set the image tag to the corresponding tag found in the release notes. The examples below are pre-populated with the tags for v2.21.0: ## 8.1 - Engine ```sh helm upgrade --install cdrplatform-engine cdrplatform-engine \ --set image.tag=2.21.0-201948 \ --set image.registry=glasswallhub.azurecr.io \ --set cloud_provider=gcp \ --atomic ``` ### 8.1A - Enable ReversingLabs (optional) To enable ReversingLabs file reputation: 1. Add the credentials to `cdrplatform-secrets` under the keys `halo-reversinglabs-username` and `halo-reversinglabs-password`, using the same mechanism you use to populate that secret in [Step 4](/halo/gke-step-4). 2. Set the non-secret fields on the Engine-owned `engine-reversinglabs` ConfigMap. The endpoint is supplied by you; use the value below. ```sh kubectl patch configmap engine-reversinglabs -n cdrplatform --type merge \ -p '{"data":{"enabled":"true","endpoint":"https://data.reversinglabs.com/api/databrowser/malware_presence/query","timeoutSeconds":"5"}}' ``` The Engine picks up the change on its next lookup; no restart is required. `timeoutSeconds` accepts 1–120 and defaults to 5. ## 8.2 - Synchronous API ```sh helm upgrade --install cdrplatform-sync-api cdrplatform-sync-api \ --set image.tag=2.21.0-201948 \ --set image.registry=glasswallhub.azurecr.io \ --set cloud_provider=gcp ``` ## 8.3 - API access service The API Access service acts as a gateway to the Glasswall Halo Synchronous API and the Policy Management API. It exposes CDR functionality over HTTP. If your environment requires HTTPS with TLS or SSL, follow the installation instructions for enabling TLS or SSL. Otherwise, follow the instructions for installation without TLS or SSL. ### 8.3A - For deployments without TLS/SSL ```sh helm upgrade --install cdrplatform-api-access cdrplatform-api-access \ --set image.registry=glasswallhub.azurecr.io \ --set image.tag=2.21.0-201948 \ --atomic ``` ### 8.3B - For deployments with TLS/SSL To use SSL certificates on CDR API, create a private key and certificate for the domain to be used. Create a Kubernetes secret using the key and CRT files using the command below. With this command we create a secret with the name "tls-secret" from the files server.key (private key) and server.crt (certificate). The key should not be passphrase protected in this example. ```sh kubectl create secret tls tls-secret --key server.key --cert server.crt ``` This secret can then be used to enable TLS on the ingress ensuring that the domain name is set on the command below: ```sh domain_name="" helm upgrade --install cdrplatform-api-access cdrplatform-api-access \ --set image.tag=2.21.0-201948 \ --set image.registry=glasswallhub.azurecr.io \ --set ingress.tls.enabled=true \ --set ingress.tls.domain=${domain_name} \ --set ingress.tls.secretName=tls-secret \ --atomic ``` ### 8.3C - Enable admin access without authentication When Single Sign-On (SSO) login is not enabled in the portal, administrative features, such as license management and policy configuration are not accessible. To enable access to these admin features without setting up SSO, deploy the Helm chart with the following configuration: ```sh helm upgrade --install cdrplatform-portal cdrplatform-portal \ --set image.tag=2.21.0-201948 \ --set image.registry=glasswallhub.azurecr.io \ --set ingress.tls.enabled=true \ --set ingress.tls.domain=${portal_domain} \ --set ingress.tls.secretName=tls-secret \ --set configuration.AutoAdmin="true" \ --atomic ``` ### 8.3D - To enable basic API authentication The Authentication in the Glasswall Halo is disabled by default, when enabled the authentication will be enabled for both Glasswall Halo Synchronous API and Policy Management API. If authentication needs to be enabled: - Create two secrets in Google Secrets Manager: one for the organisation ID and one for the organisation tokens. - When specifying multiple tokens, separate them with commas. - Ensure that individual tokens do not contain commas. - Secrets in Google Secrets Manager must follow the naming conventions below. - **Organisation ID secret** - Must start with `organisation` - Must end with `-id` - Include a numeric index between the prefix and suffix - Examples: `organisation0-id`, `organisation1-id`, `organisation2-id` - **Organisation tokens secret** - Must start with `organisation` - Must end with `-tokens` - Include a numeric index between the prefix and suffix - Examples: `organisation0-tokens`, `organisation1-tokens`, `organisation2-tokens` - Set `configuration.AuthenticationScheme=Basic` while deploying the Helm chart. For example: >```sh >helm upgrade --install cdrplatform-api-access cdrplatform-api-access \ > --set image.registry=glasswallhub.azurecr.io \ > --set image.tag=2.21.0-201948 \ > --set configuration.AuthenticationScheme=Basic \ > --atomic >``` ## 8.4 - Portal To deploy the Halo Portal service, run the commands below. ### 8.4A - For deployments without TLS/SSL **Note:** the `ip_address` mentioned in this command refers to the load balancer's public IP address. This can be retrieved through the "Portal & API Access" steps below. ```sh ip_address="" helm upgrade --install cdrplatform-portal cdrplatform-portal \ --set image.registry=glasswallhub.azurecr.io \ --set configuration.BackendUrl="${ip_address}" \ --set image.tag=2.21.0-201948 \ --set configuration.HaloVersion=2.21.0 \ --set configuration.OIDC=null ``` ### 8.4B - For deployments with TLS/SSL If TLS is required, add the `--set ingress.tls.enable_tls=true` parameter and set `portal_domain=`. In the example below, the same Kubernetes secret created for the API Access service is used to retrieve the TLS certificates. ```sh portal_domain="" helm upgrade --install cdrplatform-portal cdrplatform-portal \ --set image.tag=2.21.0-201948 \ --set image.registry=glasswallhub.azurecr.io \ --set ingress.tls.enabled=true \ --set ingress.tls.domain=${portal_domain} \ --set ingress.tls.secretName=tls-secret \ --set configuration.BackendUrl="https://${portal_domain}" \ --atomic ``` ## 8.5 - Portal access The Portal access service acts a backend for Halo Portal. It enables the Halo Portal to access the Policy Management API and Synchronous API. ### 8.5A - For deployments with TLS/SSL If TLS is required, add the `--set ingress.tls.enable_tls=true` and `--set ingress.tls.domain=` parameters. In the example below, the same Kubernetes secret created for the API access service is used to retrieve the TLS certificates. ```sh domain_name="" helm upgrade --install cdrplatform-portal-access cdrplatform-portal-access \ --set image.registry=glasswallhub.azurecr.io \ --set image.tag=2.21.0-201948 \ --set ingress.tls.enabled=true \ --set ingress.tls.domain=${domain_name} \ --set ingress.tls.secretName=tls-secret \ --set configuration.AuthenticationScheme=None \ --atomic ``` ### 8.5B - For deployments without TLS/SSL and authentication ```sh helm upgrade --install cdrplatform-portal-access cdrplatform-portal-access \ --set image.registry=glasswallhub.azurecr.io \ --set configuration.AuthenticationScheme=None \ --set image.tag=2.21.0-201948 \ --atomic ``` ## 8.6 - MongoDB ### MongoDB operator ```sh helm install community-operator mongodb/community-operator --namespace cdrplatform \ --set operator.version=0.9.0 \ --set agent.version=107.0.0.8465-1 ``` ### MongoDB installation ```sh helm upgrade -i cdrplatform-mongodb cdrplatform-mongodb -n cdrplatform --atomic \ --set cloud_provider=gcp ``` - Retrieve the connection string from the Kubernetes secret ```sh kubectl get secret mongodb-cdrplatform-cdrp-user -o jsonpath='{.data.connectionString\.standard}' | base64 -d ``` - Update the Kubernetes secret with the MongoDB connection string (remove and recreate) ```sh kubectl create secret generic cdrplatform-secrets -n cdrplatform --from-literal=mongodb-cdrp-password= --from-literal=mongodb-admin-password= --from-literal=mongodb-connectionstring= ``` ## 8.7 - Policy Management API The Policy Management API is used to manage policies for Glasswall Halo content management flags. This is an optional service, install this if you would like to create and use custom policies. ```sh helm upgrade --install cdrplatform-policy-api cdrplatform-policy-api \ --set image.registry=glasswallhub.azurecr.io \ --set image.tag=2.21.0-201948 \ --set cloud_provider=gcp ``` Policy API swagger page can be accessed using ``` http:///swagger/index.html ``` ## 8.8 - License management API The License management service is used to manage licenses in Glasswall Halo. ```sh helm upgrade --install cdrplatform-license-management cdrplatform-license-management -n license-management\ --set image.registry=glasswallhub.azurecr.io \ --set image.tag=2.21.0-201948 \ -n license-management ``` ## 8.9 - Clean up service The Clean up service deletes the original and rebuilt files from the persistent storage after the files are processed. ```sh helm upgrade --install cdrplatform-cleanup cdrplatform-cleanup \ --set image.registry=glasswallhub.azurecr.io \ --set cloud_provider=gcp \ --set image.tag=2.21.0-201948 ``` ## 8.10 - Asynchronous API The Asynchronous API can be deployed using the command below. A MongoDB database is a pre-requisite for the Asynchronous API. ```sh helm upgrade --install cdrplatform-async-api cdrplatform-async-api \ --set image.tag=2.21.0-201948 \ --set image.registry=glasswallhub.azurecr.io \ --set cloud_provider=gcp ``` ## 8.11 - Metrics projection The Metrics projection service is used to pull reporting data from MongoDB to display in the Portal UI. ```sh helm upgrade --install cdrplatform-metrics-projection cdrplatform-metrics-projection \ --set image.tag=2.21.0-201948 \ --set image.registry=glasswallhub.azurecr.io \ --set cloud_provider=gcp ``` ## 8.12 - Report extractor The Report extractor service extracts analysis reports and publishes them for reporting. ```sh helm upgrade --install cdrplatform-report-extractor cdrplatform-report-extractor \ --set image.tag=2.21.0-201948 \ --set image.registry=glasswallhub.azurecr.io \ --set cloud_provider=gcp ``` ## 8.13 - Tally accumulator The Tally accumulator service tracks and maintains a tally of usage statistics. ```sh helm upgrade --install cdrplatform-tally-accumulator cdrplatform-tally-accumulator \ --set image.tag=2.21.0-201948 \ --set image.registry=glasswallhub.azurecr.io \ --set cloud_provider=gcp ``` ## 8.14 - Storage monitoring service The Storage monitoring service provides an API to subscribe to and track changes made to documents in configured storage services such as SharePoint and OneDrive. TLS must be enabled for the Webhook. - Add the `--set ingress.tls.enable_tls=true` and `--set ingress.tls.domain=` parameters. In the example below, the same Kubernetes secret created for the API Access service is used to retrieve the TLS certificates. ```sh domain_name="" helm upgrade --install cdrplatform-storage-monitor cdrplatform-storage-monitor --wait --atomic \ --set image.tag="2.21.0-201948" \ --set image.repository="glasswallhub.azurecr.io/cdrplatform-storage-monitor" \ --set configuration.DATABASE__Provider="${database_provider}" \ --set configuration.WEBHOOKS__CallbackBaseAddress="https://${domain_name}" \ --set ingress.tls.enabled=true \ --set ingress.tls.domain=${domain_name} \ --set ingress.tls.secretName=tls-secret \ --atomic ``` ## 8.15 - Optional Prometheus scaling Install the Prometheus scaling chart only if you want to enable Prometheus-based scaling. ```sh helm upgrade --install prometheus-scaling prometheus-scaling --wait --atomic \ --set image.registry=glasswallhub.azurecr.io \ --set image.tag=3.13.2-full \ --set imagePullSecrets[0].name=acr-secret \ --set keda.icap.enabled= ``` ## 8.16 - Portal & API access Use the command below to determine the **external-IP** associated with your cluster: **Note:** **external-IP** will be the same as the public IP address attached to the AWS load balancer. ```sh kubectl get services --namespace cdrplatform nginx-ingress-ingress-nginx-controller --output jsonpath='{.status.loadBalancer.ingress[0].ip}' ``` You can now use the returned IP above to navigate to the Portal and API documentation. **Note:** use HTTPs if TLS is enabled. ``` Portal: http:// API documentation: http:///swagger/index.html ``` * * * Need help → * * * Congratulations, you have successfully deployed Glasswall Halo! Before you begin the process of deploying Glasswall Halo, ensure that you have the following tools and resources installed and setup. ## Required tools - [Helm](https://Helm.sh/docs/intro/install/) - [Kubectl](https://Kubernetes.io/docs/tasks/tools/) - [Google Cloud CLI](https://cloud.google.com/kubernetes-engine/docs/how-to/cluster-access-for-kubectl#install_plugin) >- Windows OS: >>- Map .bashrc for the Google Cloud SDK with: >>```sh >>export PATH="/usr/lib/google-cloud-sdk/bin:$PATH" >>``` >- Mac OS: >>- Map .zprofile for the Google Cloud SDK with: >>```sh >>export PATH=$PATH:/Users/user-name/google-cloud-sdk/bin >>``` **Note:** for help using the Google Cloud CLI or troubleshooting, please refer to the [GKE documentation](https://cloud.google.com/kubernetes-engine/docs/troubleshooting). ## Required Google resources ### 1. GKE instance * Recommended total of at least 8 vCPU and 32 GB RAM. * Minimum node size is 4 vCPU and 16 GB RAM. * For production workloads a minimum of 2 nodes is recommended. * Recommended GKE cluster release channel and version: stable release channel v1.27.13-GKE.1070002 * Enable the Filestore CSI driver cluster feature on your GKE cluster. * This allows cluster storage to utilize the GKE Filestore CSI driver. * In the steps below, the GKE cluster is referred to as: **gkename** **Note**: Glasswall Halo does not support ARM64 node VMs. For guidance on creating an GKE cluster please refer to: * [Create GKE cluster - Console](https://cloud.google.com/kubernetes-engine/docs/how-to/creating-a-zonal-cluster#console) * [Create GKE Cluster - CLI](https://cloud.google.com/sdk/gcloud/reference/container/clusters/create) * [Google Kubernetes Engine Service - Best Practices](https://cloud.google.com/dotnet/docs/creating-cicd-pipeline-vsts-compute-engine#before-you-begin) ### 2. Enable API services Required Google Cloud APIs that are required (enabled) for setting up and managing a GKE cluster: * Google Kubernetes Engine API * Compute Engine API * Cloud Resource Manager API * IAM Service Account Credentials API * Cloud DNS API * Cloud Storage API * Cloud Filestore API * Identity and access management (IAM) API * Secrets Manager API * Service Networking API ### 3. MongoDB database MongoDB is used to store the Glasswall Halo's content management policies, tally accumulator data, and data for asynchronous file processing and metrics. MongoDB is deployed directly inside of your cluster, via the use of MongoDB Helm charts, as seen in Step 8. **Note:** the steps below assume each resource is in the same GCP project referred to as: **project_id**. ### 4. Access to Glasswall Artifact Registry * You are provided with a token & token ID to access Glasswall's Artifact Registry (`glasswallhub.azurecr.io`). * This allows you to directly pull container images and Helm charts from your GKE cluster. * In the steps below, the token and token ID will be referred to as: **token** and **token_id.** ### **5. Assigning variables** #### Variables assigned before you begin: The variables mentioned in [required tools](#required-tools) above can be assigned before you can begin the Glasswall Halo installation. **Note:** you'll need to replace **"..."** with your own values. * Google GCP project ID: **project_id="..."** * Google Kubernetes cluster name: **gkename="..."** * Google Container Registry token ID: **token_id="..."** * Google Container Registry token: **token="..."** * * * Continue Need help? ## Step 1 - Authenticate to your Google Cloud Instance/Project Authenticate to your GKE cluster by using the following Google CLI command: ```sh gcloud auth login or gcloud auth application-default login ``` ## Step 2 - Set your Kubernetes context to GKE Set your Kubernetes context to your GKE cluster by logging into the Google CLI. ```sh gcloud projects list gcloud config set project [project_id] gcloud container clusters get-credentials [clusterName] --region [clusterregionName] --project [project_id] ``` * * * Continue Need help? All services associated with Glasswall Halo are deployed to a namespace within the target cluster. Create the namespace and then set the default context for the subsequent commands to be this newly created namespace. ```sh kubectl create namespace "cdrplatform" kubectl create namespace "license-management" kubectl config set-context --current --namespace="cdrplatform" ``` * * * Continue Need help? To allow your GKE cluster to access secrets, you must create a service account. ### Managed identity - Create service account Create the following service account to enable the GKE cluster to access secrets. 1. Create the service account. >```sh > gcloud iam service-accounts create external-secrets-sa --display-name="External Secrets >Service Account" >``` 2. Grant permissions to the newly created service account: >```sh >gcloud projects add-iam-policy-binding [project_id] \ > --member="serviceAccount:external-secrets-sa@[project_id].iam.gserviceaccount.com" \ > --role="roles/secretmanager.secretAccessor" >``` 3. Generate a key file from the service account created in the previous step above: >```sh >gcloud iam service-accounts keys create keyfile.json \ > --iam-account="external-secrets-sa@[project_id].iam.gserviceaccount.com" >``` >**Note:** the secret below [gcp-secret] will be used during Step 7 to install `cdrplatform-external-secrets`. 4. Create a Kubernetes secret from the contents of the key file created above: >```sh >kubectl create secret generic gcp-secret --from-file=keyfile.json >``` * * * Continue Need help? ## GKE iam user * A GKE iam user is included in the prerequisites section with all the permissions required: * To enable Halo access to the cloud storage bucket which contains the CDR reports, an iam user (compute@developer.gserviceaccount.com) is assigned the compute storage user/admin role. ## Cloud storage bucket To enable Halo's access to the reporting cloud storage bucket ('`saname`'): * Enable your current project as the default project for interoperable access within the interoperability settings within your cloud storage settings. * Navigate to _cloud storage settings_ and select the **interoperability** tab, then click **default project for interoperable access** and select your project from the drop down menu. * Create access key and secret for storage, like above. * Navigate to _cloud storage settings_ and select the **interoperability** tab, then in the **default project for interoperable access** section, click create key. **Note:** principals can access cloud storage data according to their project roles. to modify other permissions, use these group ids to identify these roles. * Add your access key and secret for storage to Kubernetes secrets: ```shell kubectl create secret generic storage-access -n cdrplatform --from-literal=storage-access-key= --from-literal=storage-secret-key= ``` ## MongoDB connection string To enable Glasswall Halo's Policy Management API to create and manage the policies in MongoDB, and Asynchronous API to create and manage the requests, MongoDB needs to be deployed using Helm charts listed in Step 8. ### Add the MongoDB passwords to Kubernetes secrets Two users will be created by the MongoDB Helm chart and the corresponding user's password needs to be set in Kubernetes secrets. ```shell kubectl create secret generic cdrplatform-secrets -n cdrplatform --from-literal=mongodb-cdrp-password= --from-literal=mongodb-admin-password= ``` * * * Continue Need help? [Glasswall's Artifact Registry](/halo/glasswall-artifact-registry) stores container images and Helm charts which you need to complete your Halo installation. - To pull container images and Helm charts from Glasswall Artifact Registry, you need to use your `token` and `token_ID` in two commands: >```sh >kubectl create secret docker-registry acr-secret -n cdrplatform \ > --docker-server="glasswallhub.azurecr.io" --docker-username="${token_ID}" --docker-password="${token}" >kubectl create secret docker-registry acr-secret -n license-management \ > --docker-server="glasswallhub.azurecr.io" --docker-username="${token_ID}" --docker-password="${token}" >``` - Login to the Helm registry using your `token_ID` and `token` as shown below: >```sh >helm registry login "glasswallhub.azurecr.io" -u "${token_ID}" -p "${token}" >``` See [Glasswall Artifact Registry](/halo/glasswall-artifact-registry) for a complete list of hosted container images and Helm charts. Continue Need help? Glasswall Halo consists of Glasswall Helm charts hosted in our authenticated [Glasswall Artifact Registry](/halo/glasswall-artifact-registry), along side a few industry recognized third-party charts. * Before the charts can be installed, the right versions need to be pulled by Helm. The required versions are listed in the corresponding release notes. * An example pull command is shown below; this needs to be executed for each Helm chart listed. >```sh >helm pull oci://glasswallhub.azurecr.io/ --version --untar --untardir . >``` The following example is pre-populated with the versions for [v2.21.0](/halo/v2210): ```sh helm pull oci://glasswallhub.azurecr.io/helm/cdrplatform-rabbitmq --version 0.9.4 --untar --untardir . helm pull oci://glasswallhub.azurecr.io/helm/cdrplatform-external-secrets --version 0.6.0 --untar --untardir . helm pull oci://glasswallhub.azurecr.io/helm/cdrplatform-storage --version 0.8.1 --untar --untardir . helm pull oci://glasswallhub.azurecr.io/helm/cdrplatform-engine --version 0.6.3 --untar --untardir . helm pull oci://glasswallhub.azurecr.io/helm/cdrplatform-sync-api --version 0.4.2 --untar --untardir . helm pull oci://glasswallhub.azurecr.io/helm/cdrplatform-portal --version 0.4.2 --untar --untardir . helm pull oci://glasswallhub.azurecr.io/helm/cdrplatform-policy-api --version 0.3.2 --untar --untardir . helm pull oci://glasswallhub.azurecr.io/helm/cdrplatform-api-access --version 1.2.3 --untar --untardir . helm pull oci://glasswallhub.azurecr.io/helm/cdrplatform-portal-access --version 0.2.2 --untar --untardir . helm pull oci://glasswallhub.azurecr.io/helm/cdrplatform-license-management --version 0.4.0 --untar --untardir . helm pull oci://glasswallhub.azurecr.io/helm/cdrplatform-cleanup --version 0.3.4 --untar --untardir . helm pull oci://glasswallhub.azurecr.io/helm/cdrplatform-async-api --version 0.2.3 --untar --untardir . helm pull oci://glasswallhub.azurecr.io/helm/prometheus-scaling --version 0.3.1 --untar --untardir . helm pull oci://glasswallhub.azurecr.io/helm/cdrplatform-metrics-projection --version 0.2.2 --untar --untardir . helm pull oci://glasswallhub.azurecr.io/helm/cdrplatform-report-extractor --version 0.2.2 --untar --untardir . helm pull oci://glasswallhub.azurecr.io/helm/cdrplatform-tally-accumulator --version 0.2.4 --untar --untardir . helm pull oci://glasswallhub.azurecr.io/helm/cdrplatform-mongodb --version 0.3.0 --untar --untardir . helm pull oci://glasswallhub.azurecr.io/helm/cdrplatform-storage-monitor --version 0.2.5 --untar --untardir . ``` - In order to gain access to the repositories from which our prerequisite components are installed - they must be added: ```sh helm repo add mongodb https://mongodb.github.io/helm-charts helm repo update ``` * * * Continue Need help? Now that you have access to the Helm charts, they can be deployed. - First install the prerequisite components. ```sh helm upgrade --install rabbitmq-cluster-operator oci://glasswallhub.azurecr.io/docker/bitnamicharts/rabbitmq-cluster-operator \ --atomic \ --version 4.4.34 \ --set global.imageRegistry=glasswallhub.azurecr.io \ --set global.imagePullSecrets[0]=acr-secret \ --set global.security.allowInsecureImages=true \ --set msgTopologyOperator.fullnameOverride=rabbitmq-messaging-topology-operator \ --set clusterOperator.image.repository="cgr.dev/rabbitmq-cluster-operator" \ --set clusterOperator.image.tag=2.17.0 \ --set msgTopologyOperator.image.repository="cgr.dev/rabbitmq-messaging-topology-operator" \ --set msgTopologyOperator.image.tag=1.18.3 \ --set credentialUpdaterImage.repository="cgr.dev/rabbitmq-default-user-credential-updater" \ --set credentialUpdaterImage.tag=1.0.16 \ --set rabbitmqImage.repository="cgr.dev/rabbitmq" \ --set rabbitmqImage.tag=4.2.9 \ --set clusterOperator.watchAllNamespaces=false \ --set clusterOperator.watchNamespaces={cdrplatform} \ --set msgTopologyOperator.watchAllNamespaces=false \ --set msgTopologyOperator.watchNamespaces={cdrplatform} \ --set clusterOperator.resources.requests.cpu=100m \ --set clusterOperator.resources.requests.memory=256Mi \ --set clusterOperator.resources.limits.cpu=100m \ --set clusterOperator.resources.limits.memory=256Mi \ --set msgTopologyOperator.resources.requests.cpu=100m \ --set msgTopologyOperator.resources.requests.memory=256Mi \ --set msgTopologyOperator.resources.limits.cpu=100m \ --set msgTopologyOperator.resources.limits.memory=256Mi helm upgrade --install keda "oci://glasswallhub.azurecr.io/ghcr/home-operations/charts-mirror/keda" --atomic \ --set imagePullSecrets[0].name=acr-secret \ --set global.image.registry="glasswallhub.azurecr.io" \ --set image.keda.repository="cgr.dev/keda" \ --set image.keda.tag=2.20.2 \ --set image.metricsApiServer.repository="cgr.dev/keda-metrics-apiserver" \ --set image.metricsApiServer.tag=2.20.2 \ --set image.webhooks.repository="cgr.dev/keda-admission-webhooks" \ --set image.webhooks.tag=2.20.2 \ --version 2.20.2 helm upgrade --install nginx-ingress oci://glasswallhub.azurecr.io/k8s/ingress-nginx/charts/ingress-nginx --atomic \ --set imagePullSecrets[0].name=acr-secret \ --set global.image.registry="glasswallhub.azurecr.io" \ --set controller.image.image="cgr.dev/ingress-nginx-controller" \ --set controller.image.tag=1.14.5-nginx.1.27 \ --set controller.image.digest="" \ --set controller.admissionWebhooks.patch.image.image="cgr.dev/kube-webhook-certgen" \ --set controller.admissionWebhooks.patch.image.tag=1.14.5 \ --set controller.admissionWebhooks.patch.image.digest="" \ --version v4.15.1 helm upgrade --install external-secrets oci://glasswallhub.azurecr.io/ghcr/external-secrets/charts/external-secrets \ --atomic \ --set imagePullSecrets[0].name=acr-secret \ --set webhook.imagePullSecrets[0].name=acr-secret \ --set certController.imagePullSecrets[0].name=acr-secret \ --set image.repository="glasswallhub.azurecr.io/cgr.dev/external-secrets" \ --set image.tag=2.8.0 \ --set webhook.image.repository="glasswallhub.azurecr.io/cgr.dev/external-secrets" \ --set webhook.image.tag=2.8.0 \ --set certController.image.repository="glasswallhub.azurecr.io/cgr.dev/external-secrets" \ --set certController.image.tag=2.8.0 \ --version 2.8.0 \ --set installCRDs=true ``` Next, install the supporting components below: ```sh helm upgrade --install cdrplatform-storage cdrplatform-storage \ --set cloud_provider=gcp \ --set gcp.network=[projects//global/networks/] \ --set gcp.tier=standard ``` **Note:** confirm storage has successfully deployed and is mounted before continuing to the steps below. ```sh helm upgrade --install cdrplatform-rabbitmq cdrplatform-rabbitmq \ --set image.registry=glasswallhub.azurecr.io \ --set image.tag="2.21.0-201948" \ --set cloud_provider=gcp ``` Note: if your VPC network is shared from another project, use the fully qualified network path: `projects//global/networks/`. ## Managed identity For the next step, select the same method which you used to configure your access to the Secrets Manager in [Step 3](/halo/gke-step-3), and follow the corresponding steps below to configure external secrets and install the secret synchronization. ```sh helm upgrade --install cdrplatform-external-secrets cdrplatform-external-secrets \ --set cloud_providers.gcpsm.enabled=true \ --set cloud_providers.gcpsm.projectID=[project_id] \ --set cloud_providers.gcpsm.auth.workloadIdentity.clusterLocation=[region] \ --set cloud_providers.gcpsm.auth.workloadIdentity.clusterName=[clusterName] \ --set cloud_providers.gcpsm.auth.workloadIdentity.serviceAccountRef.name=external-secrets-sa ``` * * * Continue Need help? ## Overview Glasswall licenses are managed through the License Management API. This API is used to manage active licenses and can be accessed through the main entry point of the cluster. The License Management service is responsible only for license storage and does not affect how licenses are applied within the cluster. Only one license can be stored in Glasswall Halo at a time. However, the stored license can be updated or overwritten as needed. To obtain a new license, please [contact us](https://www.glasswall.com/contact). ## Getting started To enable your new license, follow the step-by-step instructions provided in the [License Management Portal Setup Guide](/halo/license-management-portal-setup-guide). For more information please refer to our [API Documentation](https://cdrdevdocs.z6.web.core.Windows.net/license-v2#license-management-API). The Glasswall Portal allows users to upload their Glasswall license via the *System settings* page, as well as retrieve information about an existing license. If you don't have a Glasswall license but would like to obtain one, please [contact us](https://www.glasswall.com/contact/). **Note:** if you would like to set up license management without a portal installation, please refer to the [API setup guide](/halo/license-management-api-setup-guide). ## License management * [Prerequisites](license-management-portal-setup-guide#prerequisites) * [Upload a new license](license-management-portal-setup-guide#uploading-a-new-license) * [Update an existing license](license-management-portal-setup-guide#updating-an-existing-license) * [Remove an active license](/halo/license-management-portal-setup-guide#removing-an-active-license) * * * ## Prerequisites - Required services: - `Cdrplatform/portal` - `Cdrplatform/portal-access` - Configuration of SSO login in the Glasswall Portal - An Admin SSO login to access the *System settings* page - Valid Glasswall license in **.lic** format * * * ## Uploading a new license 1. Launch the **Glasswall Portal**. 2. Navigate to the *System settings* page. - **Note:** this page is only visible if you are logged in via an Admin account. 3. Click **Browse** and upload your valid Glasswall license. Your license details are displayed, indicating that your license is ready to use. ## Updating an existing license To update an existing license, follow the steps above and click **Change license**. ## Removing an active license Please follow the steps from the [API setup guide](/halo/license-management-api-setup-guide) to remove an active license. The license management subdomain makes it easy to manage licenses, allowing you to upload new license files, retrieve information about the current active license, and delete the existing license. To obtain a valid license, please [contact us](https://www.glasswall.com/contact/). * * * ## License management API setup ### 1. Uploading an active license The API can be reached through Glasswall Halo's entry point. This entry point shares the same ip address/url as the portal. To submit the license file, send a post request to the `/API/v1/license` endpoint. Include a json request body that contains the license file in the `.lic` format. ``` -----BEGIN LICENSE FILE-----\n eyJlbmMiOiJleUprWVhSaElqcDdJbWxrSWpvaU1EYzFZV05sT1RFdFlUaGtP\n UzAwTVdObUxXRmtNMk10TTJJd01EWXpOakF5WTJRMUlpd2lkSGx3WlNJNklt\n...\nYithdmN6WmVTMitrZGxrKzNwZ0RGTXhueHVpWkNYQUI3RlRrYnB5MkRRPT0i\nLCJhbGciOiJiYXNlNjQrZWQyNTUxOSJ9\n-----END LICENSE FILE-----\n ``` Assuming the license is valid and within its expiration date, the API will respond with a JSON body that contains the license details, providing a summary of the license file structure. you can find more information about the license details [here](https://API.docs.glasswall.com/license#tocs_licensedetails). **Note:** please be aware that only a single active license is allowed at any given time. An unsuccessful response from the license API typically indicates an invalid or expired license. The response body will include an error message specifying the reason for the license's failure to pass validation checks. If you encounter this situation, please reach out to us, and our team will assist you in obtaining a valid license file to retry the process successfully. ### 2. Verifying the correct license is installed Upon uploading your active license, the response will contain the license details. to retrieve these license details again, send a get request to `/API/v1/license` using the same entry point mentioned in Step 1. You can find more information about the license details [here](https://API.docs.glasswall.com/license#tocs_licensedetails). To ensure the correctness of the license, please verify the license details. specifically, confirm that the `daysleft`, `startdate`, and `entitlements` align with your expectations for the purchased license. if you suspect that the license provided is invalid or contains inaccuracies, please contact us. Please also refer to our [API documentation](https://API.docs.glasswall.com/license#license-management-API) for more information. ## Updating the active license To update a license, simply follow the same steps as mentioned earlier for installing a new license. Uploading a new license will automatically replace the previous one, ensuring that the new license takes effect. Please allow 60 seconds for the new license to take affect. ## Removing the active license To remove the currently active license, send a delete request to `/API/v1/license` using the same entry point from the previous steps. This action will remove the license and cannot be undone. ## Creating usage reports Users have the ability to create a **license usage report** to show how much the system has been utilised. The report's primary purpose is to allow systems running in passive licensing mode to report to Glasswall how much of the license has been used. License usage data is aggregated daily into the report format, so recent activity may not appear immediately if the report is created soon after files are submitted to Halo. ### Report contents The report contains the following: - The license ID the report is for - The license name - The limits for the license - System usage split into daily buckets - Each bucket includes the number of files and total size of files processed for that day - Life cycle events related to the license (installation, deletion, etc.) - Anomalies describing any issues the system encountered while tracking license usage - The date the report was created - The date the license was activated The `reportsignature` is used to validate that the contents of the license have not been changed after being generated by the Halo system. ### Information not captured The license usage report does **not** include any sensitive or identifying data. specifically, it does not capture: - IP addresses - Host or system details - Metadata about file contents - Any encrypted or user-specific information This ensures that all license reporting remains secure, anonymised, and compliant with privacy and data protection standards. #### Example usage report ```JSON "licenseReport": "licenseId": "1e6e8264-2aef-42ad-8fa3-f9f43bae5908", "licenseName": "test_licensename:001", "entitlementLimits": "haloProcessingDataLimit": "500G", "haloProcessingFileLimit": "10000" , "usage": "2024-02-07": "dayFileSize": 301045177, "dayFileCount": 210, "anomalyCount": 0, "anomalies": [ ], "lifeCycle": [ "detail": "License Installed", "time": "2024-03-05T10:01:26.801Z" ] , "2024-02-08": "dayFileSize": 287259157, "dayFileCount": 379, "anomalyCount": 1, "anomalies": [ "source": "license-monitor", "time": "2024-02-21T15:06:00.457Z", "message": "Tally Accumulator response could not be retrieved" ], "lifeCycle": [ ] , ... , "created": "2024-03-06 11:22:13.970Z", "activationDate": "2024-02-21T11:28:53Z" , "reportSignature": "eyJhbGciOiJIUzI1NiIsImI2NCI6ZmFsc2UsImtpZCI6ImNvbS5nbGFzc3dhbGwudXNhZ2UtcmVwb3J0LmhzMjU2LnYxIiwiY3JpdCI6WyJiNjQiXX0..PX-yzAbgSLCJDmW4Axf6sYvyIF-g_F8aiBQY96g8Fv8" ``` ### Creating a usage report with the API Two new http endpoints are available for programmatically downloading usage reports: - Get `/API/v1/license-usage/active` will return a raw JSON response containing the usage report for the currently installed license. - Get `/API/v1/license-usage/licenseid` will return the same type of response for a license that has been installed in the past. this works in conjunction with get `/API/v1/license-usage` that will return a list of past installed licenses ### Creating a usage report via the portal You can also download a report for the currently installed license directly from the Halo portal. Only **Admin** users can perform this action from the *System settings* page. when downloaded, the report is provided as a `.json` file named using the license id and the report’s creation date. #### To download the usage report 1. Navigate to **License management**. 2. Click **Download usage report**. A `.json` file containing the report is downloaded to your machine. The License management subdomain facilitates the retrieval of usage data for the current day, encompassing details on license entitlement limits and a tally of requests made within a 24-hour period against those limits. ## License management API ### Retrieving your usage data The API can be reached through Glasswall Halo's entry point. this entry point shares the same ip address/url as the Halo portal. To obtain usage data for the past 24 hours against the presently installed license and to view details of the current active license, initiate a get request to the `/API/v2/license` endpoint. This will include the details of your current license. To ensure the correctness of the license, please verify the license details. Specifically, confirm that the `daysleft`, `startdate`, and `entitlements` align with your expectations for the purchased license. If you suspect that the license provided is invalid or contains inaccuracies, please contact us. You should now be able to review your usage data over the current 24 hour period. This resets at midnight (system time) every day. #### Example response body: ```json { "licenseId": "3fa85f64-5717-4562-b3fc-2c963f66afa6", "licenseName": "halo:glasswall:1", "productId": "3fa85f64-5717-4562-b3fc-2c963f66afa6", "policyName": "halo:bronze_level:contract", "licenseHolder": "glasswall", "daysLeft": 100, "startDate": "2023-12-19", "expirationDate": "2023-12-19", "status": "active", "entitlements": [ "Glasswall Embedded Engine: Main", "Glasswall Embedded Engine: Protect", "Glasswall Embedded Engine: Analysis", "Glasswall Embedded Engine: Export / Import", "Glasswall Halo: File Analysis Protect" ], "entitlementLimits": { "haloProcessingFileLimit": { "limit": "12k", "tally": 123 }, "haloProcessingDataLimit": { "limit": "500G", "tally": 4567 } }, "anomalies": [] } ``` ## Understanding usage and enforcement ### What the limits measure A Halo entitlement can constrain usage along two dimensions, each reported under `entitlementLimits`: | Field | What it counts | Example `limit` | | --- | --- | --- | | `haloProcessingFileLimit` | Number of files processed in the current 24-hour period | `"12k"` | | `haloProcessingDataLimit` | Volume of data processed in the current 24-hour period | `"500G"` | For each limit, `tally` is the amount consumed so far in the current period and `limit` is the entitled maximum (for example, `"12k"` files or `"500G"` of data). Compare `tally` against `limit` to calculate how much headroom remains. ### When the tally resets The tally covers a rolling 24-hour period and **resets at midnight (system time)** every day. Usage is therefore measured **per day**, against the per-day entitlement—it is not a single cumulative total for the life of the license. ### What happens when you reach a limit When a tally reaches its entitled limit, further CDR requests **may receive a `401 Unauthorized` response for the remainder of the current 24-hour period**. Processing resumes automatically after the daily reset (or once the license is updated to a higher entitlement). The `anomalies` array is used to surface usage-related conditions (for example, approaching or exceeding a limit). Inspect it to detect issues before requests start being rejected. :::note Enforcement specifics depend on your license Exact enforcement behavior—including any warning thresholds, grace/overage allowance, how a "user" is counted for user-based entitlements and differences between Enterprise and Public-Sector/Government licenses—is determined by your specific **license policy** (`policyName`) and contract, and can differ between Halo versions. Confirm the precise figures for your deployment with your Glasswall account contact or [Glasswall Support](https://www.glasswall.com/support) rather than assuming the example values above. ::: For more information please refer to our [API Documentation](https://API.docs.glasswall.com/license#license-management-API). ### What happens when my active license expires? In the event of an expired license, requests to the sync API fail with a status code of `403 - forbidden`. The response body of such requests will include a description of the failure indicating that your license has expired. Please [contact us](https://www.glasswall.com/contact) to obtain a new license. ### What should I do if the license provided by Glasswall is invalid? If Glasswall has provided an incorrect or invalid license, please contact **support@glasswall.com** so we can help resolve the issue. ### Why do some synchronous API requests return a *403- forbidden* but others succeed? License keys are now constructed with entitlements, which are mapped to various features. From version 2.2.0, the Embedded Engine and Glasswall Halo require the presence of a valid license containing the relevant entitlements. Please contact **support@glasswall.com** where a license upgrade is required. ### The active license was accidentally deleted, what should I do? If the active license was deleted, please follow the [License management portal setup guide](/halo/license-management-portal-setup-guide) to reinstall the license at any time. The license key should persist on the platform it was distributed on. ### I have uploaded a new license but why are requests still failing? Please allow up to 60 seconds for the new license to take affect. If requests continue to fail after this period please contact **support@glasswall.com** for further assistance. Glasswall Foresight brings AI-powered threat prediction to Glasswall Halo. It combines machine learning models with Glasswall's Content Disarm and Reconstruction (CDR) analysis to assess files and predict the likelihood that they are malicious—including unknown and zero-day threats that traditional, signature-based detection can miss. Where Halo's CDR technology rebuilds files to a known-good standard, Foresight adds a predictive layer of intelligence: it inspects a file's structure and returns a probabilistic threat label that you can both *see* on your files and *act* on through policy. This functionality is included from v2.19.0. See [v2.19.0 release notes](/halo/v2190). ## Why use Foresight in Halo? ### Predictive threat detection Foresight uses machine learning to identify malicious files that traditional detection methods may not yet recognize, helping you stay ahead of zero-day and previously unseen threats. ### See the risk on every file The Foresight label is included on all API responses and rendered in the Halo file analysis report, so the prediction travels with the file wherever it is processed. ### Act on the risk through policy Files predicted to be malicious can be blocked automatically using Halo policy, combining Foresight's predictions with Glasswall's existing CDR protection. ## How it works When a file is processed by Halo, Foresight analyzes its structure using machine learning models alongside Glasswall's CDR engine. It produces a probabilistic threat label that reflects the likelihood the file is weaponized. This label is applied together with your existing CDR policy to determine how each file is handled—letting you make informed decisions, or automate the response, when potentially risky files are encountered. ## Risk labels Foresight returns one of three threat labels for each supported file: * **Malicious**—High-probability prediction that the file is weaponized. These files can be blocked automatically through policy. * **Suspicious**—Some indicators of harm were found, but there is insufficient evidence to classify the file as malicious. Caution is advised when handling these files. * **NoThreatsDetected**—No indicators of harm were found. ## Supported file types Foresight currently supports threat prediction for the following file types: * PDF * DOCX * XLSX ## Where Foresight runs Foresight runs as an additional container within the Glasswall CDR engine pod—the `cdrplatform-engine` component of your Halo deployment. When a file is processed, the engine calls Foresight to assess the original file. Foresight is **disabled by default** and is safe to leave deployed in a disabled state. ## Learn more * [Deploying Foresight in Halo](/halo/deploying-foresight-managed-kubernetes) * [Configuring Foresight in Halo](/halo/configuring-foresight-in-halo) * [About Foresight](/foresight/about-glasswall-foresight) Foresight is delivered as part of the Glasswall Halo CDR engine and is **disabled by default**. This page covers deploying and enabling Foresight on a managed Kubernetes Halo deployment (AKS, EKS, GKE or OKE). ## Prerequisites - **Halo version 2.19.0** - a Halo release that includes Foresight. See [v2.19.0 release notes](/halo/v2190). - **Foresight entitlement**—a Halo license that includes the Foresight entitlement (`Glasswall Halo: Foresight`). If the entitlement is missing or invalid, Halo continues to process files normally and Foresight results are reported as unavailable. ## Deploying Foresight Foresight is enabled through the `foresight.enabled` Helm value on the engine, which defaults to `false`. Enable it by upgrading the `cdrplatform-engine` release with the value set to `true` (see [Halo configuration changes](/halo/glasswall-halo-configuration-changes)): ```bash helm upgrade --install cdrplatform-engine cdrplatform-engine --reuse-values --set foresight.enabled=true ``` If you are deploying Halo for the first time, set the same value as part of the standard deployment for your cloud provider in the [deployment](/halo/glasswall-halo-deployment-overview) section. No other Foresight-specific deployment steps are required. ### Disabling Foresight To turn Foresight off again, upgrade the `cdrplatform-engine` release with `foresight.enabled=false`: ```bash helm upgrade --install cdrplatform-engine cdrplatform-engine --reuse-values --set foresight.enabled=false ``` The engine pods roll to remove the Foresight sidecar; standard Glasswall CDR protection is unaffected. ## Verifying Foresight is deployed After enabling Foresight and deploying or upgrading Halo: - Confirm the engine pod is running and now includes the additional `foresight` container—for example, `kubectl get pods` should show an extra container in the engine pod, and the pod should become ready. - The Foresight container runs a liveness health check; if it repeatedly fails, Kubernetes restarts the container. - Foresight is **fail-open**: if the container is unavailable, Halo continues to process files and reports the Foresight result as unavailable, rather than blocking processing. To confirm Foresight is producing results, process a supported file—PDF, DOCX or XLSX—and check that a Foresight result is returned. See [Configuring Foresight in Halo](/halo/configuring-foresight-in-halo) for how results appear and how to act on them. ## Next steps - [Configuring Foresight in Halo](/halo/configuring-foresight-in-halo) Foresight is delivered as part of the Glasswall Halo CDR engine and is **disabled by default**. This page covers enabling Foresight on a Glasswall Halo Single Node VM (OVA) deployment. ## Prerequisites - **Halo version 2.19.0** - a Halo release that includes Foresight. See [v2.19.0 release notes](/halo/v2190). - **Foresight entitlement** - a Halo license that includes the Foresight entitlement (`Glasswall Halo: Foresight`). If the entitlement is missing or invalid, Halo continues to process files normally and Foresight results are reported as unavailable. - **System requirements** - To support Foresight, the VM must have **24 CPU cores and 48GB of memory**. ## Deploying Foresight Foresight ships **disabled by default** on the Single Node VM; the installer sets `foresight.enabled=false`. To enable it, run the `enable_foresight.sh` script included on the appliance: ```bash /home/glasswall/enable_foresight.sh ``` The script enables the Foresight sidecar on the `cdrplatform-engine` deployment and sizes it for the appliance. No further configuration is required: - The Foresight image is pre-pulled into the appliance's local image store at build time, so Foresight works on an **air-gapped** appliance with no outbound registry access. - The sidecar reuses the engine's existing license. The engine pods then roll to add the Foresight sidecar. ### Enabling Foresight manually Alternatively, run the Helm upgrade that the script performs: ```bash helm upgrade cdrplatform-engine ~/cdrplatform-engine -n cdrplatform \ --reuse-values \ --set foresight.enabled=true \ --set foresight.resources.requests.cpu=1.3 \ --set foresight.resources.requests.memory=3Gi \ --set foresight.resources.limits.cpu=1.3 \ --set foresight.resources.limits.memory=3Gi \ --atomic ``` ### Disabling Foresight To turn Foresight off again, run the same upgrade with `foresight.enabled=false`: ```bash helm upgrade cdrplatform-engine ~/cdrplatform-engine -n cdrplatform \ --reuse-values \ --set foresight.enabled=false \ --atomic ``` The engine pods roll to remove the Foresight sidecar; standard Glasswall CDR protection is unaffected. ## Resource considerations The Foresight sidecar performs machine learning inference on the CPU; no GPU is required. On the Single Node VM, `enable_foresight.sh` sizes each sidecar to match the engine container, with requests and limits of **1.3 CPU and 3Gi memory**: | | CPU | Memory | |---|---|---| | Requests | `1.3` | `3Gi` | | Limits | `1.3` | `3Gi` | The appliance runs three engine replicas, so allow headroom for three Foresight sidecars. Because the models are bundled into the image, nothing is fetched at runtime. ## Next steps - [Configuring Foresight in Halo](/halo/configuring-foresight-in-halo) Once Foresight is deployed, you control how it behaves through your content management policy. Each policy decides whether Foresight runs and what happens to files it predicts to be malicious. This page covers enabling Foresight on a policy, blocking malicious files and reading the results. ## Prerequisites - Foresight is enabled on the engine (`foresight.enabled`). - Your Halo license includes the Foresight entitlement. Without it, the threat prediction settings show a **Requires upgrade** state and Foresight does not run. ## Enabling Foresight on a policy Foresight is configured per content management policy and is **disabled by default**, even when it is enabled on the engine. In the Portal, open the **Protection settings** of a policy and find the **AI Powered Threat prediction** section: - **Enable malware threat prediction**—Foresight assesses the original file and returns a threat prediction. - **Disable malware threat prediction**—files are not analyzed by Foresight. Standard Glasswall CDR protection remains active and processing times are unaffected. Enabling Foresight may increase processing times, and only PDF, DOCX and XLSX files are assessed. The equivalent setting in the policy JSON is `foresightSettings.enabled`: ```json "foresightSettings": { "enabled": true, "blockMaliciousFiles": true } ``` ## Blocking malicious files When Foresight is enabled, you choose what happens to files predicted to be **Malicious** using the **Block malicious files** setting (`foresightSettings.blockMaliciousFiles`, enabled by default): - **Block malicious files**—files flagged as malicious are still processed, but the processed file is blocked and not delivered. The threat analysis remains available. - **Allow malicious files to be processed**—files flagged as malicious are processed and a clean file is still delivered. Any malware is removed during CDR processing. In both cases the file is processed and its Foresight result is recorded; the setting only controls whether the processed file is delivered. ## How Foresight works with your CDR policy Foresight assesses the **original** file and runs alongside your CDR policy rather than replacing it—files are still sanitized and rebuilt as normal. The Foresight prediction is additional information about the original file, and blocking on malicious simply withholds delivery of the processed file: - Even when a file is blocked, it is still processed and its threat analysis can be accessed. - Because the rebuilt file is sanitized, any malware has been removed—but the content of a file predicted to be malicious may have been fabricated, so verify it before acting on it. ## Reading Foresight results ### Single files (Sync and Async API) For a single file, the APIs return the prediction in a response header: | Header | Values | |--------|--------| | `X-Foresight-Threat-Prediction` | `No threats detected`, `Suspicious`, `Malicious`, `Unavailable`, `Disabled`, `Licence invalid` | ### Archives and analysis reports For archives and analysis reports, the prediction is included per file in the report's `manifest.cdr-json` under a `foresight` object: ```json "foresight": { "status": "Malicious", "blocked": true } ``` `blocked` is `true` when the processed file was withheld because it was predicted to be malicious and the policy blocks malicious files. :::note A file predicted to be **Malicious** is also blocked when the policy allows risky content to remain, even if **Allow malicious files to be processed** is selected. Because content was allowed rather than removed during CDR, the rebuilt file cannot be guaranteed safe, so Halo withholds it. ::: ### Portal In the Portal, the file analysis page shows an **AI-powered threat prediction** section against the original file, with the prediction and guidance on handling the file. ### Status values | Status | Meaning | |--------|---------| | No threats detected | No malware was predicted in the original file. | | Suspicious | Some indicators of malware were found; treat the file with caution. | | Malicious | The original file is predicted to be malicious. It can be blocked by policy. | | Unavailable | Foresight ran but could not return a prediction—for example, the file is irregular, too small or too complex—or Foresight was temporarily unavailable. | | Disabled | Foresight is turned off for the policy used. | | Licence invalid | The deployment is not licensed for Foresight. | ICAP (Internet Content Adaptation Protocol) is designed to delegate tasks like virus scanning and content filtering to dedicated servers. Our ICAP server mediates between ICAP clients (such as proxy servers, firewalls, or gateways) and the Glasswall Halo service. When a file passes through the ICAP client, it is sent to Glasswall Halo for sanitization via ICAP before returning to the client. This ensures complete protection of content traversing your Zero Trust boundaries, whether incoming or outgoing from your environments. ## Why use Glasswall Halo's ICAP server? * Setup automated file sanitization without user intervention * Seamless integration with existing security infrastructure * Full control over web content, ensuring your business and users are protected * Industry standard way to support transparent inspection of traffic ## Use cases * Safety net for users opening files from seemingly trustworthy URLs * Users uploading files of unknown origin to your servers * Inline security with either a forward or reverse proxy workflow * Where file sandboxes or denotation chambers degrade user experience to become unworkable ## How it works ICAP requests are split into two types: * Response modifications where a user requests to download files or content from a server. * Request modifications where a user sends or uploads a file or content to a server. In both scenarios, files are processed by Glasswall Halo with millisecond speed and returned free from threats. #### Learn more * [ICAP FAQs](/halo/glasswall-icap-faqs) * [Deployment](/halo/icap-deployment) * [Supported media types](/halo/glasswall-supported-media-types) * [ICAP profiles](/halo/icap-profiles-overview) What is ICAP? ICAP (Internet Content Adaptation Protocol) is a protocol used to offload tasks such as virus scanning and content filtering to dedicated servers. What is an ICAP-based CDR solution? A Content Disarm and Reconstruction (CDR) solution sanitizes files by removing potentially harmful elements and reconstructing them to ensure they are safe. Glasswall Halo’s ICAP server integrates with your existing security infrastructure to give you control over the protection and availability of files and web-based content entering or leaving your organisation. What are the key benefits of using an ICAP-based CDR solution? - Real-time threat removal - Seamless integration with existing security infrastructure - Automated file sanitisation without user intervention - Compliance with regulatory requirements - Enhanced security and reduced risk of zero-day attacks How does the ICAP integration work with existing security infrastructure? The Halo ICAP server acts as an intermediary between ICAP clients (such as proxy servers, firewalls, or gateways) and the Glasswall Halo service. When a file passes through the ICAP client, it is sent to Glasswall Halo via ICAP for sanitisation before being returned to the client. Detailed configuration guides for popular ICAP clients are available in the documentation. Which security appliances are compatible with Glasswall Halo's ICAP solution? The Halo ICAP server is compatible with any ICAP-enabled security appliance. Detailed configuration guides are available for the following platforms: - [Fortinet FortiGate](/halo/fortigate) - [F5 BIG-IP LTM](/halo/f5) It is also compatible with Squid Proxy Server v5.x: What type of content is supported by Glasswall Halo's ICAP solution? Any content sent via ICAP includes a media type that can be passed to the Halo ICAP server for processing. Content with a file type supported by Glasswall Halo can be sanitised. For a full list, see the supported file types documentation. Media type to file type mappings are defined in ICAP profiles, which list all configurable media types. Can I control what types of content are processed? Customers have full control over how the Halo ICAP server handles each media type: - **Process** — Sanitise supported file types using CDR and return them - **Block** — Replace content with an error report explaining why it is blocked - **Bypass** — Return content unmodified and unprotected For each request, the ICAP client receives either the original content, sanitised content, or an error report. How do I manage how my content is processed? Content processing rules are defined in an **ICAP profile**. Since the profile is sent with each request, customers can create multiple profiles to support different security use cases. ICAP profiles can be configured via the Profile Management API or through the Halo portal. What happens to content processed by Glasswall Halo's ICAP server? Processed files are returned to the ICAP client, which then forwards them to the intended recipient or destination. The files are safe and free from potential threats. What happens to content that gets blocked or cannot be processed? If Halo is instructed to block content, or is unable to process a file, the content is replaced with an error report. The report explains why the content is inaccessible and includes detailed request information that administrators can use for investigation. Can I monitor requests going through the Halo ICAP server? Halo provides comprehensive auditing and reporting capabilities for ICAP traffic. You can access a full audit trail of ICAP requests, including attributes, content, and processing status. This data is also presented in visual reports that highlight trends and key metrics over time. Is there a cost to enable the Halo ICAP server? No. The Halo ICAP server is included with all Glasswall Halo licence options. You only need to deploy the required infrastructure and enable ICAP. How do I enable and configure the Halo ICAP server? The ICAP server can be enabled by following the ICAP setup guide. For appliance-specific configuration, refer to the detailed guides available for supported ICAP-enabled security platforms. ## Objective Set up the Glasswall Halo ICAP server to sanitize or block files as they pass through your security infrastructure. ## Glasswall Halo’s ICAP Server Integration Guide * [Introduction](/halo/icap-server-integration#introduction) * [Step 1 - Deploy Glasswall Halo](/halo/icap-server-integration#step-1---deploy-glasswall-halo) * [Step 2 - Deploy the ICAP Server](/halo/icap-server-integration#step-2---deploy-the-icap-server) * [Step 3 - Configure your ICAP Server client](/halo/icap-server-integration#step-3---configure-your-icap-server-client) * [Step 4 - Configure the default ICAP profile](/halo/icap-server-integration#step-4---configure-the-default-icap-profile) * [Step 5 - Test by downloading files](/halo/icap-server-integration#step-5---test-by-downloading-files) * * * ### Introduction Our ICAP Server mediates between ICAP clients (such as proxy servers, firewalls, or gateways) and the Glasswall Halo service. When a file passes through the ICAP client, it is sent to Glasswall Halo for sanitization (CDR) via ICAP before returning to the client. This ensures complete control of content encountered by members of your organization as they browse the web. You will be able to define the type of files allowed into your organization and what risky content should be sanitized from them to meet your risk profile needs. #### What is CDR? CDR (Content Disarm and Reconstruction) uses a unique method to keep organizations and individuals safe from file-based threats. Instead of trying to detect threats like most cybersecurity solutions, Glasswall CDR follows a Zero Trust approach. This means that only files that have been checked and cleaned by Glasswall are considered safe because all potential threats have been removed. we don't try to find malicious code; we simply take away its ability to harm the document. Learn more about Glasswall CDR in the main documentation #### What is ICAP? The Internet Content Adaptation Protocol (ICAP) is a lightweight protocol designed to enhance transparent proxy servers by offloading tasks like virus scanning and content filtering to dedicated ICAP servers. This allows web servers to focus on http traffic, improving efficiency. ICAP leverages edge devices to deliver specialized services such as ad insertion, content translation, and multi-av scanning. Our ICAP Server mediates between ICAP clients (such as proxy servers, firewalls, or gateways) and the Glasswall Halo service. When a file passes through the ICAP client, it is sent to Glasswall Halo for sanitization via ICAP before returning to the client. This ensures complete protection of content traversing your Zero Trust boundaries, whether incoming or outgoing from your environments. [Learn more about our ICAP Server](/halo/about-glasswall-icap) ### Step 1 - Deploy Glasswall Halo Before you can begin the ICAP Server integration, you’ll need to ensure that Glasswall Halo is deployed and configured. This is because our ICAP Server is integrated with Glasswall Halo. Please refer to our easy-to-follow [deployment guides](/halo/glasswall-halo-deployment-overview), or skip this step if you’ve already got Halo setup. ### Step 2 - Deploy the ICAP Server If you didn't deploy the ICAP Server during your Halo deployment, please refer to our step-by-step [deployment guide](/halo/icap-deployment). ### Step 3 - Configure your ICAP Server client After the ICAP Server is operational, you'll need to configure your ICAP client to route its requests through the server. Our ICAP server is compatible with any ICAP-enabled web security appliance. For additional support, please see our [Integration guides](/halo/icap-configuration-overview) for popular clients and our [proxy configuration guides](/halo/proxy-configuration) for help setting up proxy environments. ### Step 4 - Configure the default ICAP profile Next, we'll set up the default ICAP profile to manage how different file types are handled. ICAP Server requests use a content type known as media types, which can be tricky to interpret. To make it easier, we've matched these media types to the file types supported by Glasswall Halo and other common formats - please refer to the [supported media types](/halo/glasswall-supported-media-types) for more information. **Note:** you can create multiple ICAP profiles and assign one to each ICAP request. If none is specified, this default profile will be used. * Configure the ICAP profile through the [ICAP profile management API](https://api.docs.glasswall.com/profile-v1#ICAP-profile-management-API) endpoint or directly via [Halo’s ICAP settings](https://halo.glasswall.com/). * For each mapped file type, you can define what action the ICAP Server should take. * Next, configure your policy settings. * You may leave Glasswall's default content management policy which sanitizes all risky content from processed files to ensure maximum protection or amend them as required. ### Step 5 - Test by downloading files You're now ready to browse securely. * To test the file sanitization, download a Word, Excel, or PowerPoint file. * As the file downloads, your web security appliance will intercept it and send it to Glasswall Halo via ICAP. The file should be sanitized in under a second with no noticeable change in user experience. #### Error handling * Since PDFs are set to be blocked by this default profile, test it by trying to view or download a PDF. * The file should be replaced with a Halo protection report explaining why the content is unavailable, preventing the potentially dangerous file from reaching the user. ## Conclusion With all steps completed, your ICAP Server is now ready to handle and sanitize files using Glasswall Halo’s CDR process. The ICAP Server mediates between clients and Glasswall Halo, enforcing security policies, automating file handling, and enhancing protection while ensuring minimal disruption to user experience. This document provides guidance on how **Glasswall Halo ICAP server** performs across different deployment sizes, as detailed in the [Performance summary](/halo/glasswall-halo-performance). These predefined deployment sizes help organizations understand how Halo may perform relative to the scale of their operations. While the three example configurations serve as helpful benchmarks, Halo is highly scalable. it can be deployed to meet a wide range of performance and capacity requirements, provided sufficient compute resources and budget are available. These guidelines offer a framework for setting performance expectations and informing initial deployment decisions. --- ## Deployment use cases ### Use case 1: file downloads only This model applies to organizations using the Halo ICAP server **exclusively for intentional file downloads** — such as documents, installers, or media files. In this scenario, only deliberate, user-initiated downloads are forwarded to the ICAP server. All other web assets (like images, scripts, and stylesheets) are excluded. #### Estimated requests per second (RPS) Each user is assumed to initiate approximately **40 file downloads per day**. This estimate is based on internal Glasswall observations. To convert daily downloads to requests per second: ``` RPS = 40 downloads per day ÷ 86400 seconds per day ≈ 0.00046 ``` For simplicity, this is rounded to **0.00046 RPS per user**. | Number of employees | Estimated requests per second (RPS) | |--------------------|-------------------------------------| | 100 | 0.046 RPS | | 500 | 0.23 RPS | | 1,000 | 0.46 RPS | | 5,000 | 2.3 RPS | **Note:** a "request" refers to a user-initiated file download that is scanned by the ICAP server. examples include downloading a PDF report, a zip archive, or an executable file. > These figures are conservative. Actual usage may vary depending on automation scripts, scheduled downloads, or user behavior. If your environment involves automated systems or frequent batch downloads, adjust your calculations accordingly. #### Example performance result **Small deployment** - Single Node Kubernetes cluster - **5 Engines** - **8 Virtual cores** - **28 GB memory** **Throughput:** 30 Requests per second (RPS) **Equivalent organization size:** Approximately **65,000 users** (file download use case) --- ### Use case 2: all web traffic scanning In this model, the Halo ICAP server scans **all web traffic**, not just deliberate downloads. This assumes a proxy on the user’s browser or machine forwards all web requests to the ICAP server. #### Estimated requests per second (rps) Web usage estimates are based on: - Average web page requires **71 assets** to load ([http archive 2024](https://almanac.httparchive.org/en/2024/page-weight)) - Users visit around **130 web pages per day** ([digital silk web statistics](https://www.digitalsilk.com/digital-trends/top-website-statistics/)) This results in: ``` 9230 asset requests per user per day ``` Converting to RPS: ``` RPS = 9230 ÷ 86400 ≈ 0.1068 ``` For simplicity, this is rounded to **0.1 RPS per user**. | Number of employees | Estimated requests per second (RPS) | |--------------------|-------------------------------------| | 10 | 1 RPS | | 50 | 5 RPS | | 100 | 10 RPS | | 250 | 25 RPS | **Note:** this calculation does not account for browser caching, which could significantly reduce load on the ICAP server. Since reliable cache hit rate data is not available, these estimates assume no caching. Actual system load may be lower in practice. --- ## Example performance results ### Small deployment - Single Node Kubernetes cluster - **5 Engines** - **8 Virtual cores** - **28 GB memory** **Throughput:** 30 Requests per second (RPS) **Equivalent organization size:** approximately **300 users** (all web traffic use case) --- ### Medium deployment - 3 Node Kubernetes cluster - **15 Engines total** - **8 Virtual cores per node** (24 total) - **28 GB memory per node** (84 GB total) **Throughput:** 70 Requests per second (RPS) **Equivalent organization size:** approximately **700 users** --- ### Large deployment - 5 Node Kubernetes cluster - **25 Engines total** - **8 Virtual cores per node** (40 total) - **28 GB memory per node** (140 GB total) **Throughput:** 100 Requests per second (RPS) **Equivalent organization size:** approximately **1000 users** --- ## Summary These deployment examples offer a starting point for sizing your Glasswall Halo ICAP server deployment, whether you are scanning specific file downloads or all web traffic. Performance can be tuned to meet specific needs by scaling horizontally or vertically as required. For detailed guidance or tailored sizing advice, contact the Glasswall team. To deploy the ICAP server and the [ICAP Profile Management API](https://api.docs.glasswall.com/profile-v1#ICAP-profile-management-API): 1. Follow all the steps in the [deployment overview](glasswall-halo-deployment-overview) to deploy Glasswall Halo for your chosen cloud provider. Once Halo is up and running, follow the steps below to deploy the ICAP server. 2. Pull the ICAP server Helm chart: ```shell helm pull oci://glasswallhub.azurecr.io/helm/icap-server \ --version 1.5.4 \ --untar \ --untardir . ``` 3. Install the ICAP-server Helm chart: **Option A: Without MTS enabled** ```shell helm upgrade --install icap-server icap-server --atomic \ --set image.tag="2.21.0-201948" \ --set image.registry="glasswallhub.azurecr.io" \ --set configuration.HALO__Version=2.21.0 \ -n cdrplatform ``` * * * **Option B: With MTLS enabled** - Create MTLS certificates by following Step 3 in the corresponding cloud provider. - Run the command below to sync the Kubernetes secrets with the secrets provider immediately. ```shell kubectl annotate externalsecret external-secret force-sync=$(date +%s) --overwrite ``` 4. Deploy the Helm chart. ```shell helm upgrade --install icap-server icap-server --atomic \ --set image.tag="2.21.0-201948" \ --set image.registry="glasswallhub.azurecr.io" \ --set configuration.HALO__Version=2.21.0 \ --set configuration.ENABLE_MTLS=yes \ -n cdrplatform ``` **Note**: the ICAP-server deployment runs a cache cleanup job post-install and post-upgrade. To find out more, please refer to [caching](caching). 5. Install the ICAP Profile Management API in API-access by setting the `clients__icapprofile__baseaddress` configuration. ```shell helm upgrade --install cdrplatform-api-access cdrplatform-api-access \ --set image.repository=glasswallhub.azurecr.io/cdrplatform-api-access \ --set image.tag=2.21.0-201948 \ --set configuration.CLIENTS__IcapProfile__BaseAddress="http://policy-api:8080" \ --atomic ``` **Note:** the full list of available options for the API-access Helm chart can be found in the [deployment](glasswall-halo-deployment-overview) steps based on the cloud provider used. ## Specifying database provider During Helm chart deployment, the database provider can be configured with the configuration 'database__provider'. at the time of writing, Mongo and Cosmos are supported options. ``` --set configuration.DATABASE__Provider="Mongo" ``` While using Glasswall ICAP, you can configure preferences for the following supported media types. | File type | Media type | Subtype(s) | | --- | --- | --- | | **Word binary file formats** | application/msword | .doc, .dot | | **Office Open XML documents** | application/vnd.openxmlformats-officedocument.wordprocessingml.document and related subtypes | .docx, .docm, .dotx, .dotm | | **Excel binary file formats** | application/vnd.ms-excel | .xls, .xlt | | **Office Open XML workbooks** | application/vnd.openxmlformats-officedocument.spreadsheetml.sheet and related subtypes | .xlsx, .xlsm, .xltx, .xltm | | **PowerPoint binary file formats** | application/vnd.ms-powerpoint | .ppt, .pot, .pps | | **Office Open XML presentations** | application/vnd.openxmlformats-officedocument.presentationml.presentation and related subtypes | .pptx, .pptm, .ppsx, .ppam, .potm, .ppsm, .potx | | **PDF documents** | application/pdf | .pdf | | **JPEG images** | image/jpeg | .jpeg, .jpg, .jpe, .jif, .jfif, .jfi | | **PNG images** | image/png | .png | | **GIF images** | image/gif | .gif | | **TIFF images** | image/tiff | .tif, .tiff, .geotiff | | **BMP images** | image/bmp | .bmp | | **SVG images** | image/svg+xml | .svg | | **WebP images** | image/webp | .webp | | **Enhanced metafiles** | application/x-emf | .emf | | **Windows metafiles** | application/x-wmf | .wmf | | **Waveform audio files** | audio/wav | .wav, .wave | | **MP3 media files** | audio/mpeg | .mp3 | | **MP4 media files** | audio/mp4 and related subtypes | .mp4, .m4a, .m4p, .m4b, .m4r, .m4v | | **Bzip2 compressed files** | application/x-bzip2 | .bz2 | | **Gzip compressed files** | application/gzip | .gz | | **RAR archives** | application/x-rar-compressed | .rar | | **TAR archives** | application/x-tar | .tar | | **XZ compressed files** | application/x-xz | .xz | | **ZIP archives** | application/zip | .zip | | **7z archives** | application/x-7z-compressed | .7z | For more information, please [contact Glasswall support](https://www.glasswall.com/support). can this From Halo version [v2.4.12](/halo/v2412), the ICAP server includes support for caching adapted files. This improves performance when serving commonly requested files that have already been processed by the Halo CDR engine. The cache is implemented and maintained by the ICAP server (`icap-server`) and is backed by Halo’s file storage mechanism. **Once the cache is full, the oldest items are deleted.** **The cache only persists files that have been adapted with a valid response from Halo.** This includes: - Rebuilt files - Files that failed to rebuild but were successfully analysed The following files are **not cached**: - Bypassed files - Blocked files - Errored files --- ## Cache size The ICAP cache has a fixed size of **1 GB**. This size was selected to balance ease of maintenance with I/O performance. ## Cache deployment The cache is deployed automatically as part of the `icap-server` installation. Please follow the relevant deployment instructions for the ICAP server. Note that the cache is **automatically cleared** whenever the `icap-server` is installed or upgraded. --- ## Clearing the cache Although the cache is cleared automatically during installation and upgrades, you may need to manually clear it in some scenarios. Follow the steps below to empty the cache. ### Step 1 – Connect to the ICAP server inside the Halo cluster Use `kubectl` to forward a local port to the `icap-server-api` service running in the `cdrplatform` namespace. ```shell kubectl port-forward service/icap-server-api :8080 -n cdrplatform ``` ### Step 2 - make a delete request on the exposed port For convenience, the ICAP-server ships with an API for maintaining the cache. the port that was opened in the last step can now be used to clear the cache. ```sh curl --request DELETE "http://localhost:/api/v1/cache" ``` ## Metadata_end Glasswall ICAP now makes use of mutual transport layer security (mTLS) to encrypt the traffic between your proxy and the ICAP server. This feature is also known as secure ICAP (s-ICAP). Supported TLS versions: - 1.2 - 1.3 Secure-ICAP will be enabled by default by the ICAP server as long as a TLS certificate is installed. Port 11344 will be opened, with the server ready to accept secure connections. When connecting to the ICAP server through a secure connection the URI scheme submitted in the ICAP client request should be `icaps`. This indicates that a secure connection is expected. Example secure-ICAP URL: ``` icaps://gw-icap-server.net/resp-cdr-service ``` ## Configuring certificate chain verification flags The ICAP server can be configured with verification flags dictating the conditions under which the certificate chain verification is performed. For information on the configuration and on changing the verification flags, please see the ICAP server section in [Configuration changes](/halo/glasswall-halo-configuration-changes). The config item is `certificate__vefificationflags`. ## Installing certificates TLS certificates are mounted to the ICAP server from external secrets. ### First time setup For first time TLS setup during installation of Glasswall ICAP, ensure the relevant `Create secrets` deployment step is followed and the certificates have been added to your external secrets secret manager - for example, in an AKS environment, [Step 3 - Add secrets in key vault](/halo/aks-step-3). Please ensure your proxy or ICAP client has access to corresponding certificates and a root certificate signed by the same certificate authority (CA) as the ICAP server. ### Updating certificates To update an existing certificate chain, locate the secrets in your environment's secret manager (from the `Create secrets` deployment step) and update the following fields: - `Tls-cafile` - the certificate authority (CA) root certificate. The rest of the certificates must correspond to this CA root certificate. - `Tls-server-cert` - the ICAP server's signed certificate. - `Tls-server-key` - the ICAP server's private key. Please ensure your proxy or ICAP client has access to updated certificates signed by the same certificate authority (CA) as the ICAP server. **Note:** The ICAP server should detect a change in its certificate configuration instantly, there may be a delay of 60 seconds however, while Kubernetes refreshes the volume attached to the server's pod. This guide explains how to replace a proxy **DNS hostname** (e.g.,`proxy.company.com`) with its **IP address** on Windows, using either the **GUI** or **PowerShell**. This approach is useful in scenarios where: - DNS resolution is blocked or unreliable - Network latency or certificate trust issues affect performance - Access must be restricted to specific IP addresses (egress control) --- ## Manual method (Internet Options) 1. Open **Control Panel** → **Internet Options** 2. Go to the **Connections** tab 3. Click **LAN Settings** 4. Under **Proxy server**, enable the checkbox: - Replace`proxy.company.com`with your proxy IP (e.g.,`192.168.1.10`) - Set the port (commonly`8080`) 5. Click **OK**, then **Apply** This will apply to most applications that use the **WinINET** proxy stack (e.g., Internet Explorer, Microsoft Edge Legacy, Office apps). --- ## Scripted method (PowerShell) Use this PowerShell script to configure both **WinHTTP** and **WinINET** proxy settings using an IP address: ```powershell # Define proxy IP and port $proxyIP = "http://192.168.1.10:8080" # Configure proxy for WinHTTP (used by system services like Intune, Windows Update) netsh winhttp set proxy $proxyIP ";$bypassList" # Configure proxy for WinINET (used by most user apps and browsers) $regPath = "HKCU:\Software\Microsoft\Windows\CurrentVersion\Internet Settings" Set-ItemProperty -Path $regPath -Name ProxyEnable -Value 1 Set-ItemProperty -Path $regPath -Name ProxyServer -Value $proxyIP Set-ItemProperty -Path $regPath -Name ProxyOverride -Value $bypassList ``` This guide explains how to replace a proxy **DNS hostname** with a **direct IP address** on macOS, using either the **network settings GUI** or a **shell script**. This is useful in environments where: - DNS resolution is unreliable or slow - You need to route traffic through a specific egress IP - Consistent IP usage is required for conditional access or firewall rules --- ## Manual method (System Settings) 1. Open **System Settings** → **Network** 2. Select your current network (e.g., Wi-Fi or Ethernet) 3. Click **Details** → **Proxies** 4. Under: - **Web proxy (HTTP)** and - **Secure web proxy (HTTPS)** Replace`proxy.company.com`with the IP address of the proxy (e.g.,`192.168.1.10`) 5. Set the port (usually`8080`) 6. Click **OK** and **Apply** --- ## Scripted method (bash) Use this shell script to apply proxy settings using a raw IP address across all network services: ```bash #!/bin/bash # Define proxy IP and port PROXY="192.168.1.10" PORT="8080" # Get list of all network services (excluding the header) services=$(networksetup -listallnetworkservices | tail +2) # Apply proxy settings to each network service for service in $services; do echo "Setting IP proxy for: $service" networksetup -setwebproxy "$service" "$PROXY" "$PORT" networksetup -setsecurewebproxy "$service" "$PROXY" "$PORT" networksetup -setproxybypassdomains "$service" $(echo $EXCLUSIONS | tr "," " ") done ``` This guide explains how to add your **proxy or outbound firewall IP address** to an **Azure conditional access** policy in Microsoft Entra ID (formerly Azure AD). This helps you: - Restrict access to corporate network locations - Bypass MFA for trusted IPs (such as proxy or VPN egress points) - Enforce stricter policies for external or untrusted sources --- ## Prerequisites - **Global administrator** or **security administrator** role in Azure AD - Knowledge of your **proxy/firewall's external (egress) IP address** - Azure AD Premium P1 or P2 license (conditional access requires this) --- ## Step 1 - Sign in to the Azure portal - Navigate to [https://portal.Azure.com](https://portal.Azure.com) - Launch the **Microsoft Entra ID** blade (formerly **Azure Active Directory**) --- ## Step 2 - Navigate to conditional access - From the side menu: - Click **Protection** → **Conditional access**. - Click an existing policy **or** click **+ New policy** to create one. --- ## Step 3 - Configure conditions based on locations - Under your policy: - Click **Conditions** → **Locations**. - Set the toggle to **Yes**. --- ## Step 4 - Define named locations - Click **Select locations** → then **+ Named location**. - Provide a meaningful name, e.g.,`Trusted Proxy IP`. - Under **IP ranges**, add your proxy or firewall's public IP(s). - Check the box if this IP should be marked as **trusted** (for MFA or device compliance rules). --- ## Step 5 - Apply the named location - Once the named location is saved: - Navigate back to the **Select locations** screen. - Select either: - **Include** → to apply the policy *only* when users are coming from that proxy IP - **Exclude** → to *bypass* the policy for that IP (e.g., exclude from MFA) --- ## Step 6 - Complete the policy - Under **assignments**, choose users/groups to apply the policy to. - Under **access controls**, choose: - **Grant** → block or allow access - **Session** → optional controls like sign-in frequency - Set **Enable policy** to **On**. - Click **Create** or **Save**. --- ## Example use cases | Scenario | Action in conditional access policy | |----------------------------------------|------------------------------------------------| | Bypass MFA for users behind proxy | Exclude proxy IP under "locations" condition | | Require MFA unless on proxy | Include all IPs, then exclude proxy IP | | Allow access only from proxy | Include only the proxy IP as named location | --- ## Testing and logs - Use **sign-in logs** in Microsoft Entra ID to verify: - The IP seen by Azure matches your proxy/firewall's egress IP - The conditional access policy result (`Success`,`Failure`,`Not Applied`, etc.) --- ## Tips - If you're behind multiple proxies or regional egress IPs, add them all to the named location. - Azure reads the **client public IP**, so NAT or forward proxies must expose the correct external address. - Be cautious when blocking access based on IP — always test with a break-glass account excluded from the policy. --- This guide explains how to configure proxy settings and exclusions on **MacOS** when using an IP-based proxy, including an exclusion list for applications that rely on **WebSockets**, such as: - **Slack** - **Lucidchart** - **Figma** These apps often fail behind proxies like **Squid**, which don't support **WebSockets** by default. ## Manual method (system settings) 1. Navigate to **System Settings** → **Network**. 2. Select your active interface (e.g., Wi-Fi). 3. Click **Details** → **Proxies**. 4. Under **web proxy (HTTP)** and **secure web proxy (HTTPS)**: - Replace`proxy.company.com`with your **proxy IP** (e.g.,`192.168.1.10`) - Set port (commonly`8080`) 5. In **bypass domains**, add the required domains (see below). 6. Click **OK**, then **Apply**. --- ## What this script does - Applies proxy settings to **all network interfaces** - Sets HTTP and HTTPS proxy - Configures a list of **bypass domains** that will not use the proxy - Ideal for use with`networksetup`and MDM-managed devices --- ## Scripted method (bash) ```bash #!/bin/bash # Define proxy server and port PROXY="192.168.1.10" PORT="8080" # Domains to exclude from proxy — especially for WebSocket-based apps EXCLUSIONS="localhost,127.0.0.1,*.microsoftonline.com,*.core.windows.net,*.slack.com,*.figma.com,*.lucidchart.com,*.lucid.app" # Get all active network services services=$(networksetup -listallnetworkservices | tail +2) # Apply settings per service for service in $services; do echo "Setting proxy for: $service" networksetup -setwebproxy "$service" "$PROXY" "$PORT" networksetup -setsecurewebproxy "$service" "$PROXY" "$PORT" networksetup -setproxybypassdomains "$service" $(echo $EXCLUSIONS | tr "," " ") done ``` --- This guide explains how to configure proxy settings and bypass exclusions on **Windows**, with attention to **IP-based proxy definitions** and **WebSocket app compatibility**. Common apps that rely on WebSockets: - **Slack** - **Figma** - **Lucidchart** These can fail when routed through proxies like **Squid**, which do not support`wss://`connections. --- ## What this script does - Sets a **system-wide proxy IP address** via PowerShell - Applies exclusions for **user apps (WinINET)** and **system services (WinHTTP)** - Prevents proxy routing for **WebSocket-based** apps and Microsoft services - Supports deployment via **Intune**, **GPO**, or manual script execution --- ## Manual configuration steps 1. Launch **Control Panel** → **Internet Options**. 2. Navigate to the **Connections** tab → **LAN Settings**. 3. Enable **Use a proxy server**. 4. Set: - Address = `192.168.1.10` - Port = `8080` 5. Click **Advanced**, then add the domains below to the **Exceptions** list. 6. Apply and save. --- ## Scripted method (PowerShell) ```powershell # Define proxy IP and port $proxyIP = "http://192.168.1.10:8080" # Define proxy bypass list $bypassList = "localhost;127.0.0.1;*.microsoftonline.com;*.core.windows.net;*.slack.com;*.figma.com;*.lucidchart.com;*.lucid.app;*.github.com;copilot-proxy.githubusercontent.com" # Apply proxy for system services netsh winhttp set proxy $proxyIP ";$bypassList" # Apply proxy for user apps $regPath = "HKCU:\Software\Microsoft\Windows\CurrentVersion\Internet Settings" Set-ItemProperty -Path $regPath -Name ProxyEnable -Value 1 Set-ItemProperty -Path $regPath -Name ProxyServer -Value $proxyIP Set-ItemProperty -Path $regPath -Name ProxyOverride -Value $bypassList ``` ## Overview During internal testing of our ICAP proxy implementation on macOS and Windows, we identified a key limitation in how browser-level and system-level proxy exclusions are handled: - **Character limits**: Both platforms impose a maximum character count for the local exclusion list (the "bypass proxy for" field). - **Impact**: When this limit is exceeded, some systems silently fall back to bypassing the proxy **entirely**, creating serious gaps in policy enforcement. To avoid this issue, we recommend managing proxy exclusions centrally using a **PAC (proxy auto-config) file** rather than relying on per-device configurations. --- ## Recommended approach: use a PAC file ### What is a PAC file? A **proxy auto-config (PAC)** file is a JavaScript-based configuration script that dynamically defines proxy behavior based on request URLs, domains, IPs, or other parameters. ### Benefits | Benefit | Description | |-----------------------|-----------------------------------------------------------------------------| | ✅ Centralized control | All clients reference one centrally hosted PAC file. Update once, and all clients receive the change. | | 🚫 No character limit | A PAC file can contain hundreds of exclusions and logic far beyond what's possible in a GUI field. | | 🔄 Dynamic routing | Define conditional logic (e.g. "use proxy for everything except these domains"). | | 🔐 Policy integrity | Ensures all clients apply the same exclusion logic without risking proxy bypass. | --- ## Implementation steps ### 1. **Create a PAC file** Use the following example as a base: ```javascript function FindProxyForURL(url, host) { // Domains to bypass proxy if (dnsDomainIs(host, "internal.glasswall.com") || shExpMatch(host, "*.corpnet.glasswall.local") || isInNet(host, "10.0.0.0", "255.0.0.0")) { return "DIRECT"; } // Everything else goes through ICAP proxy return "PROXY proxy.glasswall.com:3128"; } ``` Customize domain patterns and subnet IPs as needed. --- ### 2. **Host the PAC file** Place it in a **network-accessible location**: - Internal web server (e.g.`https://intranet.glasswall.com/proxy.pac`) - Network share (macOS-compatible SMB path or DFS) Ensure it is: - Secure (HTTPS preferred) - Readable by all endpoints - Version controlled --- ### 3. **Configure clients to use PAC** #### Windows - Use **Group Policy (GPO)** or **Intune**: - Set`Automatic proxy configuration`to the hosted PAC URL. - Disable manual exclusions to prevent character limit misuse. #### macOS - Use your Apple MDM provider - Enable proxy auto-config via`.mobileconfig`: ```xml ProxyAutoConfigURLString https://intranet.glasswall.com/proxy.pac ``` --- ### 4. **Test the configuration** - Open a browser and verify proxy behavior: - Access external sites (should route via ICAP). - Access excluded domains/IPs (should go direct). --- ## Notes from internal testing - Edge cases where browser extensions or third-party apps apply **additional proxy rules** may override the PAC file. - Browser hardcoded exclusions (e.g.`localhost`,`127.0.0.1`) are still respected. --- ## 🔧 Troubleshooting | Symptom | Likely cause | Resolution | |--------------------------------|---------------------------------------|------------------------------------------------| | All traffic bypasses proxy | Character limit reached in exclusion list | Use PAC file | | PAC file not respected | URL is unreachable or misconfigured | Confirm URL is accessible from client machine | | App ignores PAC | App does not use system proxy settings | Configure app separately or enforce via firewall rules | --- ## Summary Switching to a PAC file for managing proxy exclusions: - **Solves platform limits** - **Centralizes policy updates** - **Reduces misconfiguration risks** - **Ensures security controls stay intact** For help with deploying this at scale, contact the IT infrastructure or security engineering team. This guide explains how to replace a proxy **DNS hostname** (e.g.,`proxy.company.com`) with a **raw IP address** on Windows, both **manually** and **using PowerShell**. This is helpful in environments where: - DNS resolution is unreliable or blocked - Certificate trust or network latency issues occur - Explicit egress IP-based access control is needed --- ## Manual method (Internet Options) 1. Launch **Control Panel** → **Internet Options**. 2. Navigate to the **Connections** tab. 3. Click **LAN Settings**. 4. Under **Proxy server**, enable the checkbox: - Replace`proxy.company.com`with your proxy IP (e.g.,`192.168.1.10`) - Set the port (commonly`8080`) 5. Click **OK**, then **Apply**. This will apply to most applications that use the **WinINET** proxy stack (e.g., Internet Explorer, Microsoft Edge Legacy, Office apps). --- ## Scripted method (PowerShell) Use this PowerShell script to configure both **WinHTTP** and **WinINET** proxy settings using an IP address: ```powershell # Define proxy IP and port $proxyIP = "http://192.168.1.10:8080" # List of domains to bypass proxy $bypassList = "localhost;127.0.0.1;*.microsoftonline.com;*.core.windows.net" # Configure proxy for WinHTTP (used by system services like Intune, Windows Update) netsh winhttp set proxy $proxyIP ";$bypassList" # Configure proxy for WinINET (used by most user apps and browsers) $regPath = "HKCU:\Software\Microsoft\Windows\CurrentVersion\Internet Settings" Set-ItemProperty -Path $regPath -Name ProxyEnable -Value 1 Set-ItemProperty -Path $regPath -Name ProxyServer -Value $proxyIP Set-ItemProperty -Path $regPath -Name ProxyOverride -Value $bypassList ``` Reporting features within the Halo portal include ICAP, presenting summaries and trends of ICAP request details. ## Request summary The *Request summary* section displays the following visual reports: ### Request type * **Requested**: also known as respmod, such as when a user tries to download a file from the internet. * **Submitted**: also known as reqmod, such as when a user tries to upload a file into your organisation. ### Request status * **Adapted**: request was successfully adapted and contains sanitized content. * **Blocked**: request was blocked and content was replaced with report to inform user that type of content is not permitted. * **Errored**: request could not be processed and was replaced with an error report to inform user. * Bypassed content is not displayed in the list of ICAP requests. ### Top requested file types * Ordered list of file types within the ICAP requests content ## Request trends The request trends section displays the following trend charts, giving a view of how ICAP requests compare over time: - **Request status** - **Request type** The Glasswall Halo Portal allows you to view an audit trail of the ICAP requests sent to the ICAP server. Viewable in chronological order, the following key attributes are available for each request: - **Time completed**: the time the request was completed. - **Request ID**: a unique ID to identify the request. - **Request type**: the type of request: - *Requested*, also known as respmod, such as when a user tries to download a file from the internet. - *Submitted*, also known as reqmod, such as when a user tries to upload a file into your organisation. - **Request URL**: the URL of the request. - **Profile applied**: the profile used for the request, defining whether to process, block or bypass the content. - **Request status**: the status of the request: - *Adapted*: request was successfully adapted and contains sanitized content. - *Blocked*: request was blocked and content was replaced with report to inform user that type of content is not permitted. - *Errored*: request could not be processed and was replaced with an error report to inform user. - **Note:** bypassed content is not displayed in the list of ICAP requests. ## Filtering The list of ICAP requests can be filtered on all the possible values of both request type and status. ## ICAP request information For each request in the list, users can view further details about each piece of content the request contained under the *Content details* section: - **Transaction ID**: a unique ID to identify the content processing transaction. - **File type**: the type of file, if the content type could be mapped to one of Halo's [supported file types](/halo/glasswall-halo-supported-file-types). - **Content type**: the type of content, listed as a [media type](/halo/glasswall-supported-media-types). - **File name**: the name of the file, if available in the request. - **Content length**: the original length of the content. - **Processing rule applied**: derived from the profile provided with the request: - Process - Block - Bypass - While bypassed requests are not displayed, bypassed content may be displayed as part of a request containing multiple content items. - **Content status**: the status of the content: - Adapted: content was successfully sanitized. - Blocked: content was blocked from being processed by profile configuration. - Errored: content could not be processed. ICAP profiles provide a means to customize the processing of files by the ICAP server and manage CDR. Simultaneous activation of multiple ICAP profiles is supported but each incoming respmod or reqmod request is assigned precisely one profile. ## Profile ### Status The status property determines whether the ICAP server should proceed with utilizing the specified profile. When set to `false`, the profile is deactivated, and the request will result in a failure, accompanied by an error message. ### Profile settings #### Content management policy The `ProfileSettings.ContentManagementPolicy` property represents a named policy within Halo. This policy will be applied to all content within `RESPMOD`And`REQMOD` requests. If the specified `ContentManagementPolicy` does not exist during processing, requests will fall back to using the `default` profile. The content management policy labeled as 'Default' is allocated to the default profile. [Learn more about Policy Management](/rest-api/about-policy-management-api) #### Processingrulesdefault `ProfileSettings.ProcessingRulesDefault`Determines which action the ICAP server should take when it encounters an unrecognized mime/content-type. An unrecognized content type refers to one that is not part of the`ProcessingRules`Collection. #### Processingrules `ProfileSettings.ProcessingRules` is a collection of rules to be applied to a `RESPMOD`Or`REQMOD` request. The rules can contain custom mime/content-types. When ICAP server matches the content-type of a `REQMOD`Or`RESPMOD` request, it will decide what to do with the content depending on the value of the rule: | Rule | Description | |--|--| | `process` | CDR the file and replace the http content with the rebuilt file. | | `bypass` | Do nothing, pass the original file through. | | `block` | Do not CDR the file. Replaces the http content with an error report. | ### The default profile When ICAP is installed in Halo, a default ICAP profile will be created and persisted on first start-up. This profile is set to `process` all supported content types by default, and `block` unrecognised content types. Resetting your profile from release 2.5.4 ensures it will include the new media types. ### Example profile JSON ```json { "ProfileIdentity": "f49f7748-4f33-4638-8f24-bfd721bcd487", "ProfileName": "default", "ProfileVersion": 1, "IsDefaultProfile": true, "Status": "enabled", "ProfileSettings": { "ContentManagementPolicy": "default", "ProcessingRulesDefault": "block", "ProcessingRules": { "application/msword": "process", "application/vnd.openxmlformats-officedocument.wordprocessingml.document": "process", "application/vnd.ms-excel": "process", "application/vnd.openxmlformats-officedocument.spreadsheetml.sheet": "process", "application/vnd.ms-powerpoint": "process", "application/vnd.openxmlformats-officedocument.presentationml.presentation": "process", "application/pdf": "process", "image/jpeg": "process", "image/png": "process", "image/gif": "process", "image/tiff": "process", "image/bmp": "process", "image/svg+xml": "process", "image/webp": "process", "application/x-emf": "process", "application/x-wmf": "process", "audio/wav": "process", "audio/mpeg": "process", "audio/mp4": "process", "application/x-bzip2": "process", "application/gzip ": "process", "application/x-rar-compressed": "process", "application/x-tar": "process", "application/x-xz": "process", "application/zip ": "process", "application/x-7z-compressed": "process" } } } ``` ## Fallback mode If the policy API cannot be contacted for any reason during the retrieval and caching of profiles, the service can use a fallback profile until a time in which it can recover. This profile is configured to be the same as the default installed profile and as such uses a default content management policy and processing rules. To enable this, please refer to [Configuration changes](/halo/glasswall-halo-configuration-changes). When disabled the service will attempt to recover by restarting. ## Overview The [ICAP profile management API](https://API.docs.glasswall.com/profile-v1#ICAP-profile-management-API) allows for creation and modification of ICAP profiles. ## Creating a new ICAP profile ### 1. Make a post request to the [ICAP profile management API](https://API.docs.glasswall.com/profile-v1#ICAP-profile-management-API) The API can be reached through Glasswall Halo's entry point. This entry point shares the same IP address/url as the portal. * To submit a new ICAP profile, send a post request to the`/api/v1/profiles/`Endpoint. Include a JSON request body that contains the profile. **Example**: ```json { "status": "enabled", "profileSettings": { "contentManagementPolicy": "default", "processingRulesDefault": "block", "processingRules": { "application/msword": "process", "application/vnd.openxmlformats-officedocument.wordprocessingml.document": "process", ... } } } ``` If the request body is valid, the response will contain a summary of the newly created profile. ### 2. Use the profile in ICAP server For ICAP to find and use the profile for processing, configure your ICAP client to send the profile name via a`profile`Query param to the ICAP server. **Example:** `icap://icap-server.:1344/resp-cdr-service?profile=` For more information please refer to our [API documentation](https://API.docs.glasswall.com/profile-v1#ICAP-profile-management-API). The Glasswall Halo ICAP server can be configured with any ICAP-enabled network security appliance, which includes firewalls, vpns, proxies, gateways and load balancers. ## Firewalls * [Fortinet fortigate](/halo/fortigate) * [F5 big-ip ltm](/halo/f5) * [Squid 5.x](/halo/icap-squid) ## ICAP server integration For a complete step-by-step process of setting up the Glasswall Halo ICAP server, please refer to the [ICAP server integration](/halo/icap-server-integration) guide. To configure your F5 firewall and integrate it with the Glasswall Halo ICAP server, please follow this configuration guide. **Note:** these steps are documented as per F5 version BIG-IP 17.1.1.3, Build 0.0.5, Point Release 3. ## Configure interfaces and routes - Before configuring the interfaces, ensure you can see 3 interfaces under *Network* -> *Interfaces* -> *Interface list*. - **Note:** this shows the data plane interfaces and does not include management interface. - The names of the interfaces will be 1.1, 1.2, 1.3. - Based on the MAC address of each interface by comparing it with the MAC address on the interfaces attached to the firewall VM, note down which subnets each interface belongs to. - In the dev environment: - `1.1` Belongs to `inside subnet` - `1.2` Belongs to `outside subnet` - `1.3` Belongs to `icap subnet`. ### 1 - Configure management IP address Typically the first interface attached to the firewall is configured as the management interface. If for any reason the management UI is running on a different interface (IP address), follow these steps to change the management IP address. 1. Login to the F5 Admin Portal. 2. Navigate to *System* -> *Platform*. 3. Under the *Host IP address* drop-down, select **Custom host IP address** and enter the new IP. ### 2 - Configure inside interface 1. Navigate to *Network* -> *VLANs* -> *VLANs list* and click **Create**. 2. Enter the name as `inside-vlan`. 3. Select the inside interface from the drop-down and click **Add**. **Note:** the interface names will be `1.1`, `1.2` and `1.3`. In the previous step, the sub-net to which the interface belongs to, based on the MAC address is noted down. 4. Click **Finished**. 5. Navigate to *Network* -> *Self IPs* and click **Create**. 6. Enter the name as `inside-ip`, and enter the IP address and netmask of the inside interface. 7. Select the `inside-vlan` from the VLAN drop-down and click **Finished**. ### 3 - Configure outside interface 1. Navigate to *Network* -> *VLANs* -> *VLANs list* and click **Create**. 2. Enter the name as `outside-vlan`. 3. Select the outside interface from the drop-down and click **Add**. 4. Click **Finished**. 5. Navigate to *Network* -> *Self IPs* and click **Create**. 6. Enter the name as `inside-ip`, and enter the IP address and netmask of the outside interface. 7. Select the `outside-vlan` from the VLAN drop-down and click **Finished**. ### 4 - Configure ICAP interface 1. Navigate to *Network* -> *VLANs* -> *VLANs list* and click **Create**. 2. Enter the name as `icap-vlan`. 3. Select the ICAP interface from the drop-down and click **Add**. 4. Click **Finished**. 5. Navigate to *Network* -> *Self IPs* and click **Create**. 6. Enter the name as `icap-ip`, and enter the IP address and netmask of the ICAP interface. 7. Select the `icap-vlan` from the VLAN drop-down and click **Finished**. ## ICAP integration ### 1 - Create ICAP pool The ICAP pool represents a pool of ICAP servers. 1. Navigate to *Local traffic* -> *Pools* and click **Create**. 2. Enter an appropriate name and description; e.g. Name -`dev-aks-icap-pool`, Description -`ICAP Pool for Dev AKS env`. 3. Under *Health monitors*, select `tcp` from the *Available* column and move it to the *Active* column. 4. Enter an appropriate **Node name**. e.g. `dev-aks-icap-node`. 5. Enter the IP address of ICAP server, service port as `1344` and click **Add**. 6. Click **Finished** to save the configuration. If the health checks are successful, the status will appear in green, otherwise red. ### 2 - Create HTTP pool The HTTP pool represents a pool of web servers that needs to be protected by the F5. 1. Navigate to *Local traffic* -> *Pools* and click **Create**. 2. Enter an appropriate name and description. e.g. Name - `dev-http-pool`, Description -`Onboarding web app dev`. 3. Under *Health monitors*, select `http` from the *Available* column and move it to the *Active* column. 4. Enter an appropriate **Node name**. e.g. `dev-onboarding-app-node1`. 5. Enter the IP address and port of the web server, and click **Add**. 6. Click **Finished** to save the configuration. If the health checks are successful, the status will appear green, otherwise red. ### 3 - Create ICAP profile ICAP profile is used to send the traffic from HTTP virtual server to ICAP server to CDR the content. In this scenario as we want to process the files uploaded to web server, we need to use request modification service in the ICAP. 1. Navigate to *Local traffic* -> *Profiles* -> *Services* -> *ICAP* and click **Create**. 2. Enter an appropriate name. e.g. `dev-icap-req-mod`. 3. Select **ICAP** as the *Parent profile*. 4. Select the **Custom** box in *settings* and enter `icap://${SERVER_IP}:${SERVER_PORT}/req-cdr-service?profile=default` in the URL. You can use a custom ICAP profile if needed. 5. Complete the other fields as required and click **Finished**. ### 4 - Create HTTP profile A custom HTTP profile can be used to modify the HTTP traffic of the web server. 1. Navigate to *Local traffic* -> *Profiles* -> *Services* -> **HTTP** and click **Create**. 2. Enter an appropriate name. e.g. `dev-http-profile`. 3. Select **HTTP** as the *Parent profile*. 4. Select the **Custom** box on the *Settings* and customise the settings as required. 5. Click **Finished** to save the profile. ### 5 - Create ICAP internal virtual server An internal virtual server is a special type of virtual server used to send the traffic to ICAP server. The ICAP virtual server sits in front of the ICAP pool. 1. Navigate to *Local traffic* -> *Virtual servers* -> *Virtual servers list* and **Create**. 2. Enter an appropriate name and description. e.g. `dev-icap-vs`. 3. Select *Type* as `Internal` and enter `0.0.0.0/0` as the source address. 4. Select `Advanced` configuration and under ICAP profile, select the ICAP profile created in Step 3. 5. Under `Default Pool`, select the ICAP pool created in Step 1. 6. Click **Finished** to save the virtual server. ### 6 - Create request adapt profile 1. Navigate to *Local traffic* -> *Profiles* -> *Services* -> *Request adapt* and click **Create**. 2. Enter an appropriate name. e.g. `dev-icap-req-adapt`. 3. Under the *Parent* profile, select `requestadapt`. 4. Select the **Custom** box in *Settings*, and select the internal ICAP virtual server created in Step 6 in the internal virtual name drop-down. 5. Select the service down action as needed. This represents what action will be taken when the ICAP server is down. -`Drop`: To drop the traffic at the HTTP virtual server. -`Reset`: Resets the connection on the client side. -`Ignore`: Sends the original traffic to the web server. 6. Click **Finished** to save the request adapt profile. ### 7 - Create HTTP virtual server An HTTP virtual server sits in front of HTTP pool and provides a virtual IP address that can be used by the users to access the protected web server. 1. Navigate to *Local traffic* -> *Virtual servers* -> *Virtual servers list* and click **Create**. 2. Enter an appropriate name. e.g. `dev-onboarding-http-vs`. 3. Enter `0.0.0.0/0` in the *Source address*. 4. Choose an IP address for the HTTP virtual server that does not conflict with any other address space and enter it as the destination address. 5. Enter the web server port as the server port. 6. Select *Advanced* configuration and select HTTP profile created in Step 4 under `HTTP Profile(Client)`. 7. Select the request adapt profile created in Step 6 under request adapt profile. 8. Select **Auto map** under the *Source address translation* drop-down. 9. Select the HTTP pool under *Default pool* and click **Finished**. To configure your Fortigate firewall and integrate it with the Glasswall Halo ICAP server, please follow this configuration guide. **Note:** these steps are documented as per Fortigate v7.4. ## Configure interfaces and routes ### Configure static IP to inside interface Two network interfaces are attached to the VM: - **Management interface** - this is the primary network interface attached to the VM when it is created. By default the static private IP address of the interface is registered in the firewall. - **Inside interface** - this is an extra interface attached to VM after it is created. By default no IP address might be shown to this interface in the firewall. To configure the static IP to the inside interface in the firewall: 1. Login to the management portal and navigate to *Network* -> *Interfaces*. 2. Select and edit the inside interface. 3. Enter the IP/netmask and click **OK**. ### Create a static route A static route is needed in the firewall so that traffic from all ports will go to the internet via a given gateway IP address and interface To create a static route: 1. Navigate to *Network* -> *Static routes* and create a new route. 2. Select **Subnet** as the *Destination* and enter `0.0.0.0/0.0.0.0`. 3. Enter the subnet's gateway IP address under *Gateway address*. Typically the first number of the subnet is the gateway. e.g. 192.168.xx.1. 4. Select the interface that should be used to reach the gateway IP address and click **OK**. ## Configure ICAP By default the ICAP feature in the Fortigate firewall is disabled and needs to be enabled. 1. Login to the management portal and navigate to *System* -> *Feature visibility* and enable *ICAP* under additional feature. 2. Create an ICAP server by navigating to *Security profiles* -> *ICAP servers* and create a new server. 3. Enter the name, IP address and port(default 1344) of the ICAP server, and click **OK**. ### Create ICAP profile 1. Navigate to *Security profiles* -> *ICAP* and create a new profile. 2. Enter a name for the profile. 3. Enable **Request processing** if you want to process files sent in the requests. 4. Select the *Server* created in the previous step from the drop-down. 5. Under *Path* enter `req-cdr-service`. Optionally pass an ICAP profile e.g. `req-cdr-service?profile=test-profile1`. 6. Select one of the options when ICAP fails to process the files - **error** or **bypass**. 7. Enable **Response processing** if you want to process the files being received in the response. 8. Select the *Server* created in the previous step from the drop-down. 9. Under *Path* enter `resp-cdr-service`. Optionally pass an ICAP profile e.g. `resp-CDR-service?profile=test-profile1. 10. Select one of the options when ICAP fails to process the files - `Error`Or`Bypass`. 11. Enable **Streaming media bypass**. ### Create or update firewall policy 1. Navigate to *Policy & objects* -> *Firewall policy* and create or edit a firewall policy to enable ICAP integration. 2. For *Inspection mode*, select the **Proxy-based** option. 3. From the *SSL inspection* drop-down, select **Deep-inspection** or **Custom-deep-inspection**. 4. Enable the **ICAP** option and select the ICAP profile created in the previous step. 5. Click **OK**. ## Download CA certificate As we have now enabled the SSL inspection in the firewall, we need to download the CA certificate and import it in the user's machine or browsers. This ensures that users won't see SSL errors while accessing internet. 1. Navigate to *Security profiles* -> *SSL/SSH inspection* and select the profile used in the firewall policy. 2. Click **Download** beside *CA certificate*. 3. Copy it to user's machines and add it to the trusted list under CA certificates. ## Exempt websites from SSL inspection We need the ability to exempt certain websites from SSL inspection for various reasons, such as compatibility issues that arise when SSL inspection is enabled. By default, specific addresses are already exempted from SSL inspection. 1. Navigate to *Security profiles* -> *SSL/SSH inspection*. 2. Select **Log SSL exemptions** so the firewall will log any such exemptions. 3. If required, add specific web categories in this section to exempt from SSL inspection. This guide explains how to configure **Squid 6.x** with the **Glasswall Halo ICAP server**, enabling advanced threat protection through our advanced Content Disarm and Reconstruction (CDR) technology via Glasswall Halo. **Note:** instructions are based on **Squid 6.x built with OpenSSL**. for a full list of configuration options, please refer to the [Squid documentation](https://www.squid-cache.org/Doc/). ## Squid Integration Guide * [Step 1 - Basic configuration](/halo/icap-squid#step-1-basic-configuration) * [Step 2 - Persistent connections](/halo/icap-squid#step-2---persistent-connections) * [Step 3 - Restart squid](/halo/icap-squid#step-3---restart-squid) * [Step 4 - Troubleshooting (Optional)](/halo/icap-squid#step-4---troubleshooting-optional) * [Step 5 - Enabling https scanning](/halo/icap-squid#step-5---scanning-https-content) * [Step 6 - Final restart](/halo/icap-squid#step-6---final-restart) ### Step 1: Basic configuration Locate and edit your Squid configuration file, typically at: * `/etc/squid/squid.conf` * `/etc/squid5/squid.conf` #### 1A - Define access control lists Uncomment or add the following ACLs for local networks and safe ports: ```bash acl localnet src 10.0.0.0/8 acl localnet src 172.16.0.0/12 acl localnet src 192.168.0.0/16 acl localnet src fc00::/7 acl localnet src fe80::/10 acl SSL_ports port 443 acl Safe_ports port 80 acl Safe_ports port 21 acl Safe_ports port 443 acl Safe_ports port 70 acl Safe_ports port 210 acl Safe_ports port 1025-65535 acl Safe_ports port 280 acl Safe_ports port 488 acl Safe_ports port 591 acl Safe_ports port 777 acl CONNECT method CONNECT ``` #### 1B - Allow local network access Under recommended minimum access permission configuration, add: ```bash http_port 3128 http_access allow localnet http_access allow localhost http_access deny all ``` #### 1C - Enable ICAP Add the following to turn on ICAP support: ```bash icap_enable on icap_send_client_ip on icap_preview_enable on icap_preview_size 0 icap_service_failure_limit -1 ``` ### 1D - Configure request modification (uploads) ```bash icap_service glasswall_req reqmod_precache bypass=0 icap://:1344/req-cdr-service adaptation_access glasswall_req allow all ``` ### 1E - Configure response modification (downloads) ```bash icap_service glasswall_resp respmod_precache bypass=0 icap://:1344/req-cdr-service adaptation_access glasswall_resp allow all ``` ### Step 2 - Persistent connections Squid 6.x supports persistent ICAP connections by default. To enforce: ```bash icap_persistent_connections on ``` **Tip:** ensure the ICAP server is also configured for persistent connections. Otherwise, disable them (off) to prevent errors. ### Step 3 - Restart squid Apply changes: ```bash sudo systemctl restart squid ``` ### Step 4 - Troubleshooting (optional) If you see errors like: ```bash essential ICAP service is down after an options fetch failure: icap://:1344/req-cdr-service [down,!opt] ``` Check the following: * Confirm you are running the latest stable Squid 6.x * If persistent connections cause issues, disable them: ```bash icap_persistent_connections off ``` ### Step 5 - Scanning HTTPs content To scan encrypted https traffic, Squid must be compiled with SSL and configured for SSL bumping. **Requirements** * Squid 6.x with openssl * A root certificate for SSL interception #### 5A - Update listening port ```bash http_port 3128 ssl-bump generate-host-certificates=on dynamic_cert_mem_cache_size=4MB cert=/etc/ssl/myc.pem ``` #### 5B - Configure SSL bump ```bash sslcrtd_program /lib/squid/ssl_crtd -s /var/cache/squid_ssldb -M 4MB sslcrtd_children 5 ssl_bump server-first all acl step1 at_step SslBump1 ssl_bump peek step1 ssl_bump bump all sslproxy_cert_error deny all ``` #### 5C - Generate and install root certificate ```bash openssl req -new -newkey rsa:2048 -days 1000 -nodes -x509 -keyout myc.pem -out myc.pem ``` Initialize certificate storage: ```bash rm -rf /var/cache/squid_ssldb /lib/squid/ssl_crtd -c -s /var/cache/squid_ssldb ``` **Important:** install myc.pem as a trusted root certificate in your browser. ### Step 6 - Final restart Restart Squid to finalize all changes: ```bash sudo systemctl restart squid ``` To configure FileCloud and integrate it with the Glasswall Halo ICAP server, follow this configuration guide. **Note:** This guide is based on FileCloud version 23.1 and outlines the essential steps for integrating it with the Glasswall Halo ICAP server. For more detailed information, refer to the FileCloud documentation. ## Prerequisites Before beginning the integration process, ensure the following prerequisites are met. ### 1. FileCloud Ensure that FileCloud is installed and that the license is activated. ### 2. Glasswall Halo ICAP server Confirm that the Halo ICAP server is installed and properly licensed. For guidance on installation and configuration, refer to: * Quick start guide for Glasswall Halo ICAP server * Instructions for installing or upgrading the Halo ICAP server * Configuration guide for Halo ICAP server Verify that the Halo ICAP server has a valid license to operate effectively. For information on license configuration, refer to the licensing section in the Halo ICAP server documentation. ### 3. Web browser access Ensure you have web browser access to the FileCloud admin portal. This guide assumes that FileCloud has already been set up and initially configured. Refer to the FileCloud documentation for instructions on accessing the admin portal. ## Integration steps To integrate FileCloud with the Glasswall Halo ICAP server, follow these steps. ### 1. Access the FileCloud admin portal * Open your preferred web browser and navigate to the FileCloud admin portal. ### 2. Navigate to settings * In the left hand navigation panel, select **Settings**. ### 3. Select third party integrations * Click the *Third Party Integrations* tab. ### 4. Configure ICAP antivirus settings * Under the *Anti-virus* section, choose **ICAP A** from the drop down menu in the *Anti-virus Type* field. ### 5. Fill in the required fields * **Server local IP address:** Typically, you can leave this as the default value 0.0.0.0. If you have a dedicated FileCloud policy with ICAP, enter the private LAN IP address of your FileCloud server. Refer to the FileCloud documentation for more details. * **ICAP remote hostname:** Enter the IP address or hostname of your Halo ICAP server. * **ICAP port:** The default port for ICAP traffic is 1344. Ensure this is set correctly. ### 6. Enable secure ICAP if applicable * If your ICAP server uses SSL or TLS, select the **Secure ICAP** checkbox to enable secure communication. ### 7. Set file size limit * The default file size limit for scanning is set to 100 MB. Adjust this value if you need to scan larger files. ### 8. Specify the ICAP service name * Enter `req-CDR-service` as the ICAP service name. Requests follow the following URL structure: ``` icap://${SERVER_IP}:${SERVER_PORT}/req-cdr-service?profile=default ``` You can use a custom ICAP profile if required. ### 9. Configure debug logging * If you require debug logging, enable this option for troubleshooting purposes. ### 10. Save and test the configuration * Click **Save** to apply the settings. * To verify the integration, click the "ICAP Test" button at the top of the page. By following these steps, you can successfully integrate FileCloud with the Glasswall Halo ICAP server, enabling advanced file scanning and security capabilities. For additional configuration options and troubleshooting guidance, refer to the Glasswall Halo ICAP server documentation. ## Enhance your Glasswall Halo protection with ReversingLabs Spectra Intelligence integration Glasswall is committed to continuously advancing and innovating to secure your organization's digital infrastructure. In this constantly evolving cybersecurity landscape, safeguarding your data from threats is essential. By integrating ReversingLabs' Spectra Intelligence with Glasswall Halo, we have enhanced file security by combining advanced threat insights with cutting-edge Content Disarm and Reconstruction (CDR) technology. With real-time file reputation intelligence that classifies billions of files, Glasswall Halo helps your organization stay ahead of cyber threats, leveraging ReversingLabs' constantly updated malware database to identify and address file-based security risks. Glasswall Halo leverages file reputation to support both its synchronous and asynchronous APIs, ensuring that even unsupported file types by the Glasswall Embedded Engine are evaluated for potential threats. This integration enables organizations to adopt a Zero Trust security approach while maintaining the flexibility to handle files that may not fully align with Glasswall CDR specifications. ## Why choose Glasswall Halo with Spectra Intelligence and CDR? * **Advanced threat intelligence** * ReversingLabs evaluates the risk of over 40 billion files, augmenting the security of Glasswall’s Zero Trust threat removal with enhanced protection. * **Comprehensive file type support** * ReversingLabs provides real-time insights on over 4,800 file types, enabling organizations to swiftly assess risk, mitigate evolving threats, and prevent data breaches. * **Automated, customizable protection** * Admins can automate file handling by leveraging advanced malware detection and classification. safe files are automatically approved, while suspicious ones are blocked. ## The largest repository of goodware & malware ReversingLabs Spectra Intelligence provides the industry’s fastest and most precise malware insights. Customers can access the reputations of over 40 billion files in the threat repository, with millions of new samples added daily. Utilizing ReversingLabs’ advanced binary analysis and multi-factor classification algorithm, files are systematically analyzed and accurately classified as either goodware or malware. ## How it works When a file enters your network, Glasswall Halo analyzes it using ReversingLabs Spectra Intelligence. The file’s 'reputation' is checked against ReversingLabs’ extensive database, evaluating its behavior and connections to known malware. Simultaneously, Glasswall Halo’s CDR technology sanitizes the file, ensuring only safe content reaches your endpoints. Our CDR technology works by disassembling files, such as documents and images, and removing any potentially malicious elements. Glasswall CDR retains the file’s original functionality and appearance, providing seamless protection for the end user. [Learn more about ReversingLabs' Spectra Intelligence](https://www.ReversingLabs.com/products/spectra-intelligence) or [contact us](https://www.glasswall.com/contact) for more information. #### Learn more * [Getting started](/halo/threat-intelligence-getting-started) * [About Glasswall Halo](/halo/about-glasswall-halo) * **What is Glasswall CDR?** - CDR stands for Content Disarm & Reconstruction. Glasswall's CDR technology works by inspecting a file's digital DNA, cleaning risky content according to a set policy, and rebuilding to a 'known good' standard, delivering a safe & visually identical file. Once your Halo order has been processed, if you’ve opted to include the File Reputation feature, we will set up a new ReversingLabs account for you. **Note:** you can also use an existing ReversingLabs account, provided you have the required credentials. Please follow the steps below to configure Glasswall Halo with your ReversingLabs account. ## Prerequisite - Minimum Halo version: [v2.4.15](/halo/v2415) ## Configure Glasswall Halo with ReversingLabs First, follow the steps in the [deployment](/halo/glasswall-halo-deployment-overview) section to deploy Halo according to your cloud provider. How you then supply your ReversingLabs credentials depends on the deployment: - **Cloud (AKS, EKS, GKE, OKE)** — store the credentials in your secret manager and set the remaining fields on the Engine's ConfigMap. See the *Enable ReversingLabs* step on the **Install CDR components** page for your provider. - **OVA and other self-hosted appliances** — configure it at runtime from the Portal or the Halo API. See [Enable ReversingLabs](/halo/enable-reversing-labs). The ReversingLabs endpoint itself is fixed by the Engine and is not configurable; you supply only the credentials and, optionally, a lookup timeout. ## Retrieve threat intelligence data for your files ### Single files #### In the Sync API In this example, we process a single file on the CDR-file endpoint on the Sync API. 1. Make a POST request with a single file to the sync API, for example the CDR-file endpoint: `/api/v3/cdr-file`. 2. Observe the response headers. They should contain the following: | Header | Description | Values | |------|-------------|--------| | `x-filereputation-responsecode` | HTTP response code from the file reputation service | `200 OK`, `400 Bad Request`, `401 Unauthorized`, `403 Forbidden`, `404 Not Found`, `405 Method Not Allowed`, `409 Conflict`, `413 Request Too Large`, `429 Too Many Requests`, `500 Internal Server Error`, `502 Bad Gateway`, `503 Service Unavailable`, `UnknownError`, `Timeout` | | `x-filereputation-status` | Malware presence status | `unknown`, `known`, `suspicious`, `malicious` | | `x-filereputation-threatname` | Detected threat name for the requested sample | Example: `win32.trojan.nsis` | | `x-filereputation-threatlevel` | Threat severity calculated by a proprietary ReversingLabs algorithm | Value from `0` to `5`, where `5` indicates the highest severity | | `x-filereputation-trustlevel` | Confidence that a known sample is goodware | Value from `0` to `5`, where `0` represents the highest confidence | ### Archives In this example, we process a ZIP file on the CDR-file endpoint on the Sync API. 1. Make a POST request with a supported archive file to the Sync API, for example the CDR-file endpoint: `/api/v3/cdr-file`. Ensure an analysis report will be generated. 2. Download the composite archive result containing the `/clean` and `/report` folders. 3. Open the report archive file under the `/report` directory. 4. Observe the contents of the `manifest.cdr-json` file. The JSON should contain a new `fileReputation` section: ```json { "fileReputation": { "response": "200", "fileStatus": "SUSPICIOUS", "threatName": "Win32.Trojan.Nsis", "threatLevel": "4", "trustLevel": "5" } } ``` ### Compressed files For compressed files (`.bz2`And`.gzip`), The file reputation data is returned via response headers (similar to single files). **Note:** that the file reputation results are for the underlying file and not the parent compressed type. ## Objective Set up automated file sanitization in Amazon S3 buckets using Glasswall Halo API via event notification functionality and lambda functions. ## Amazon S3 Integration Guide * **[Prerequisites](/halo/amazon-s3-integration#prerequisites)** * **[Step 1 - Create an S3 bucket](/halo/amazon-s3-integration#step-1---create-an-s3-bucket)** * **[Step 2 - Create SQS queue](/halo/amazon-s3-integration#step-2---create-sqs-queue)** * **[Step 3 - Turn on event notification](/halo/amazon-s3-integration#step-3---turn-on-event-notifications)** * **[Step 4 - Create a Lambda function](/halo/amazon-s3-integration#step-4---create-a-lambda-function)** * **[Step 5 - Configure execution role](/halo/amazon-s3-integration#step-5---configure-execution-role)** * **[Step 6 - Upload Lambda code](/halo/amazon-s3-integration#step-6---upload-lambda-code)** * **[Step 7 - Add trigger](/halo/amazon-s3-integration#step-7---add-trigger)** * **[Step 8 - Test](/halo/amazon-s3-integration#step-8---test)** *** ## Prerequisites * AWS account and IAM (Identity and Access Management) role with the following permission policies: * Amazons3fullaccess: to create and configure the S3 buckets. * Amazonsqsfullaccess: to create and configure the SQS queue used to trigger. * Awslambda_fullaccess: to create and configure the lambda function that executes a call to Glasswall Halo. * (Optional) iamfullaccess: required to create and configure a default execution role for the new function. If this role cannot be assigned, follow the relevant [steps in this guide](/halo/amazon-s3-integration#optional-iamfullaccess) to request creation of a role and then use this pre-created role. * Authenticated access to Glasswall Halo's synchronous API. * * * ### Step 1 - Create an S3 bucket First, you need to create an S3 bucket where the source files are added, and this will be the source of the events which trigger the workflow. 1. Log in to the [AWS management console](https://console.aws.amazon.com). 2. Navigate to the Amazon S3 console by entering “S3” in the search bar or by selecting **S3** under the *Services* > *Storage* menu. 3. In the left navigation pane, select **Buckets**. 4. Click **Create bucket**. The *Create bucket* page opens. 5. For bucket name, enter a name for your bucket. The bucket name must: - Be unique within a partition. A partition is a grouping of regions; AWS currently has three partitions:`aws`(Standard regions),`aws-cn`(China regions), and`aws-us-gov`(AWS govcloud (us) regions). - Be between 3 and 63 characters long. - Consist only of lowercase letters, numbers, dots (.), and hyphens (-). For best compatibility, we recommend that you avoid using dots (.) in bucket names, except for buckets that are used only for static website hosting. - Begin and end with a letter or number. Bucket names are globally unique, the AWS console validation will advise you if the entered bucket name is already in use. **Note:** after you create the bucket, you cannot change its name. For more information about naming buckets, see [bucket naming rules](https://docs.aws.amazon.com/AmazonS3/latest/userguide/bucketnamingrules.html). 6. For **Region**, choose the AWS region where you want the bucket to reside. **Note:** to minimize latency and costs and address regulatory requirements, choose a region close to you. Objects stored in a region never leave that region unless you explicitly transfer them to another region. For a list of Amazon S3 AWS regions, see [AWS service endpoints](https://docs.aws.amazon.com/general/latest/gr/rande.html#s3_region) in the amazon web services general reference. If you require any additional settings for your bucket, such as access control, logging, versioning, or encryption, you can configure them accordingly. Additionally, under the set permissions section, you can define who has access to the bucket and its objects by choosing from options like bucket policies, access control lists (ACLs), or block all public access. Ensure that you review and set these configurations as per your requirements before proceeding. For more detailed information about creating and configuring S3 buckets please refer to the [AWS S3 user guide](https://docs.aws.amazon.com/AmazonS3/latest/userguide/create-bucket-overview.html). * * * ## Step 2 - Create SQS queue After the S3 bucket is created, the Amazon SQS (Simple Queue Service) queue needs to be created. This is where the event notifications from your source S3 bucket will be sent, and picked up by the Lambda function. 1. Navigate to the Amazon SQS console by searching for “SQS” in the search bar or by selecting **Simple Queue Service** under the *Services* > *Application integration* menu. 2. Click **Create queue**. 3. Choose between the two types of SQS queues: **Standard** or **FIFO**. Standard queues provide high throughput and best-effort ordering, while FIFO queues guarantee “exactly once” processing and strict ordering based on message group ID. The standard queue type is set by default. **Note:** you cannot change the queue type once it has been created. 4. Enter a unique **Name** for your queue; e.g. *s3cdrevents*. The name of a FIFO queue must end with the *.fifo* suffix. The suffix counts towards the 80-character queue name quota. To determine whether a queue is [fifo](https://docs.aws.amazon.com/AWSSimpleQueueService/latest/SQSDeveloperGuide/fifo-queues.html), you can check whether the queue name ends with the suffix. 5. The console sets default values for the queue [configuration parameters](https://docs.aws.amazon.com/AWSSimpleQueueService/latest/SQSDeveloperGuide/sqs-configure-queue-parameters.html). If you are familiar with SQS, under **Configuration** you can set new values for the parameters. For the purposes of this use case, we will leave most of these as default. 6. Scroll to the **Access policy** section. 7. Select the **Advanced** option to edit the policy JSON via the advanced editor to allow the S3 service to publish messages to the queue. 8. Add a comma after the “__owner_statement” element and paste the following JSON snippet: ``` { "Sid": "AllowS3ToPublish", "Effect": "Allow", "Principal": { "Service": "s3.amazonaws.com" }, "Action": "sqs:SendMessage", "Resource": "" } ``` **Note:** ensure that you replace **arn-of-queue** in the snippet with the ARN (Amazon Resource Name) of the queue; this will be the same as the resource ARN of the owner statement in the existing JSON. 8. Review the configuration values you have entered and click **Create queue**. Make a note of the ARN for the queue for use later. For more detailed information about creating and configuring Amazon SQS please refer to the [AWS S3 user guide](https://docs.aws.amazon.com/AWSSimpleQueueService/latest/SQSDeveloperGuide/sqs-getting-started.html). * * * ## Step 3 - Turn on event notifications The next step is to turn on the event notification, so that your SQS queue receives a notification every time a file is placed in the source S3 bucket. 1. Navigate to the [Amazon S3](https://console.aws.amazon.com/s3/) console and from the **Buckets** list, select the bucket you created earlier. 2. From the *Bucket overview* page, select the **Properties** tab. 3. Scroll down to the *Event notifications* section and click **Create event notification**. 4. In the *General configuration* section, specify a descriptive **Event name** for your event notification. Optionally, you can also specify a **Prefix** and a **Suffix** to limit the notifications to objects with keys ending in the specified characters. 5. In the *Event types* section, you can select one or more event types that you want to receive notifications for. In this example we are focusing on the **Object creation** events so that new or copied objects result in notifications that result in CDR being actioned on the object. Select the **All object create events** checkbox. 6. Lastly, in the *Destination* section, choose the event notification destination – in this case the SQS queue we have created and configured. Select **SQS queue** as your destination and pick the SQS queue we created and configured previously. 7. Select **Save changes**, and Amazon S3 sends a test message to the event notification destination. The event notification is created, and you are returned to the S3 bucket properties. For more detailed information about event notifications please refer to the [AWS S3 user guide](https://docs.aws.amazon.com/AmazonS3/latest/userguide/notification-how-to-event-types-and-destinations.html). * * * ## Step 4 - Create a Lambda function Once the event notification has been turned on, you need to create a new Lambda function. 1. Navigate to the Lambda service by searching for “Lambda” in the search bar or by selecting **Lambda** under the *Services* > *Compute* menu. 2. In the left navigation pane, choose **Functions**. 3. Click **Create function**. 4. At the *Create function* page you are presented with three options; select **Author from scratch**. 5. Enter an appropriate **Function name** for your Lambda function; e.g. *CDR-file*. 6. Next, select a **Runtime** that matches your intended language. **Note:** You will be able to upload code in a later step. 7. (Optional) If you are familiar with Lambda functions, you can set additional configurations via the **General configuration** option under the *Configurations* tab. 8. Click **Create function**. You arrive at the *Function overview* page. 9. Assigning the execution role can be done in two ways: * Requesting and assigning a new role using [iamfullaccess](/halo/amazon-s3-integration#optional-iamfullaccess) * Assigning an already existing role * This can be done via the Lambda service: *Function overview* > *Configuration* > *Permissions* > *Edit*. ### [Optional] iamfullaccess If you are unable to be assigned *iamfullaccess* permissions, please request a role be created with the following permissions: Logs:createlogstream Logs:putlogevents Provide the completed JSON from the [Configure execution role](/halo/amazon-s3-integration#step-5---configure-execution-role) step to your privileged users to create a role. This new role can be selected from the Lambda creation user interface in place of allowing a new default role to be created. * Scroll to the *Change default execution role* section. * Choose **Use an existing role** and select the pre-created role. The next step of configuring the execution role can now be skipped and you can proceed to [Step 6 - Upload lambda code](/halo/amazon-s3-integration#step-6---upload-lambda-code). * * * ## Step 5 - Configure execution role To allow the Lambda function access to the AWS services (S3, SQS), you need to assign additional permissions to the Lambda’s execution role. 1. In the [AWS Lambda](https://console.aws.amazon.com/lambda/) console, choose **Functions** in the left navigation pane. 2. On your Lambda function's details page, choose the **Configuration** tab, and then click **Permissions** in the left navigation pane. 3. Under **Execution role**, choose the link of the role name. The IAM console opens. 4. On the IAM console's *Summary* page for your Lambda function's execution role, choose the **Permissions** tab. 5. From the **Add permissions** menu, select **Create inline policy**. 6. Switch to the JSON editor. 7. In the snippet below, replace the below section (resource) with the ARN of the SQS queue you created and noted down from a previous step: **Arn:AWS:sqs:\{region\}:\{account\}:\{queuename\}"** ```json { "Version": "2012-10-17", "Statement": [ { "Sid": "UseCaseStatement0", "Effect": "Allow", "Action": [ "sqs:DeleteMessage", "sqs:GetQueueUrl", "sqs:ReceiveMessage", "sqs:GetQueueAttributes", "sqs:ListQueueTags" ], "Resource": [ "arn:aws:sqs:{Region}:{Account}:{QueueName}" ] }, { "Sid": "UseCaseStatement1", "Effect": "Allow", "Action": [ "sqs:ListDeadLetterSourceQueues", "sqs:ListMessageMoveTasks", "sqs:ListQueues", "s3:GetObject", "s3:CreateBucket", "s3:PutObject" ], "Resource": "*" } ] } ``` 8. Paste the snippet into the JSON editor with the updated resources. 9. Click **Next**. 10. Give the policy a **Name**; e.g. *usecasepolicy*. 11. Click **Create policy**. Your Lambda function now has permissions to interact with the services you require. * * * ## Step 6 - Upload Lambda code When files are uploaded to the source S3 bucket, you require some code to be run in order to process them when the Lambda is invoked. Depending on your framework there are multiple ways to get the Lambda function running the intended code, but this guide focuses on the zip upload functionality. The Lambda function is going to have the following logic regardless of framework: * Retrieve file: interpret the SQS message to retrieve the file from the source bucket. * CDR file: make an authenticated request to the Glasswall Halo Rest API. * Handle responses: handle responses both successful and unsuccessful. For more information on deploying Lambda functions, refer to the [AWS Lambda documentation](https://docs.aws.amazon.com/lambda/latest/dg/gettingstarted-package.html). * * * ## Step 7 - Add trigger In this step you configure the Lambda function to be invoked when SQS messages are published to the queue by adding a trigger. 1. In the **Function** overview pane of your function’s console page, choose **Add trigger**. 2. From the list of available triggers select **SQS**. 3. Select the SQS queue you created previously and click **Add**. The process is now complete and ready to be tested. * * * ## Step 8 - Test Now that everything is in place, upload a file to the source S3 bucket and within a few seconds a file with the same name will appear in the destination S3 bucket. This file is a visually identical protected copy of the original file, without any of the risky content or structural defects which could include malware. #### Confirm via Glasswall Halo Portal * Process the original file using Glasswall's [Clean a file](https://halo.glasswall.com/clean-a-file) application. * Your file will be given a [risk level](/halo/original-malware-status) along with any risky content items or structural defects listed. * Process the protected file from the destination S3 bucket in the [Clean a file](https://halo.glasswall.com/clean-a-file) application. * The file should be returned as clean with no risky content or structural defects. ## Objective Set up automated file sanitization in Azure Blob Storage using Glasswall Halo API via the event grid functionality and service bus queues. ## Azure Blob Storage Integration Guide * **[Prerequisites](/halo/azure-blob-storage-integration#prerequisites)** * **[Step 1 - Create a service bus namespace](/halo/azure-blob-storage-integration#step-1---create-a-service-bus-namespace)** * **[Step 2 - Create a service bus queue](/halo/azure-blob-storage-integration#step-2---create-a-service-bus-queue)** * **[Step 3 - Set up source blob storage account events](/halo/azure-blob-storage-integration#step-3---set-up-source-blob-storage-account-events)** * **[Step 4 - Create a function app](/halo/azure-blob-storage-integration#step-4---create-a-function-app)** * **[Step 5 - Create a function](/halo/azure-blob-storage-integration#step-5---create-a-function)** * * * ## Prerequisites * Azure Portal account * Source Azure Blob Storage account * Destination Azure Blob Storage account **Notes:** * Ensure that all the configuration parameters are correctly named and added to the Azure Function App. Any mismatches will lead to failures. * This guide covers flat files only and does not include detailed logging. * * * ### Step 1 - Create a Service Bus namespace First, you need to create a Service Bus namespace and a queue so that you can receive the event grid messages. **Note:** the tier and settings are related to your load or it. 1. Sign in to [Azure Portal](https://portal.azure.com/). 2. Navigate to *All services* > *Integration* > *Service Bus*. 3. In the **Basics** tab of the *Create namespace* page, create a Namespace. **Note:** the sample code provided will work with all tiers so select the settings appropriate for your usage, and enter all mandatory configuration. For more detailed information about creating a Service Bus namespace and queue, please refer to [Azure documentation](https://learn.Microsoft.com/en-us/Azure/service-bus-messaging/service-bus-quickstart-portal). * * * ### Step 2 - Create a Service Bus queue Once the namespace is created, we need to create a queue to post the events to. 1. On the *Service Bus namespace* page, select **Queues** in the left navigational menu. 2. On the *Queues* page, select **+ queue** on the toolbar. **Note:** as with the namespace, the settings are down to preference – for this sample we are simply taking the defaults. 3. Enter a name for the queue, and leave the other values with their defaults. The name you choose doesn't matter as the sample code allows it to be configured to whatever is set. 4. Select **Create**. For more detailed information about creating a Service Bus namespace and queue, please refer to [Azure documentation](https://learn.Microsoft.com/en-us/Azure/service-bus-messaging/service-bus-quickstart-portal). * * * ### Step 3 - Set up source Blob Storage account events After creating the Service Bus namespace and queue, configure the source Blob Storage account to emit events as messages to the queue. Although these events can be configured to trigger an Azure Function directly, each event must be acknowledged within 30 seconds. While most files are processed quickly, larger files or periods of high load can exceed this window. To avoid this limitation, we recommend configuring Event Grid to deliver events to the Service Bus queue and allowing the Azure Function to consume messages from the queue. This approach provides greater resilience and flexibility by decoupling event ingestion from processing. 1. To set up events, navigate to the source storage account and select *Events* > *More options* > *Service Bus queue*. 2. Enter a *Name* under event subscription details. 3. Enter a *System topic name* under topic details. 4. From the *Filter to event type* dropdown, select **Blob created**. This means there will be a message for every new file in Blob Storage across the storage account. 5. Finally, choose the queue you created in the previous step under endpoint detail. A new Service Bus message is now placed in the queue for every file uploaded to that storage account, which will be consumed via a function we will create next. * * * ### Step 4 - Create a Function App You must have a Function App to host the execution of your functions. The Azure function that consumes the Service Bus messages you just created will be able to use the event grid details to identify the new file. Your function should perform the following: * Identify the new file from the Service Bus message. * Access the file from the source storage account. * Send it in a request to Glasswall Halo’s Synchronous API. * Create (if it doesn’t already exist) a container of the same name in the destination storage account. * Write either the protected file (as the same name as the original) or a text file if it couldn’t be protected. 1. To create a Function App, navigate to the *Create a resource* page. 2. In the *New* page, select *Compute* > *Function app*. 3. Adjust the settings to suit your function code and hosting option: For more detailed information about creating a Function App, please refer to [Azure documentation](https://learn.Microsoft.com/en-us/Azure/Azure-functions/functions-create-function-app-portal). * * * ### Step 5 - Create a function Once the Function App is created, you can create the function. 1. In the Function App overview, it provides guidance for various methods of getting your code deployed. > Choose the method most suitable for your selected codebase. 2. The following configuration needs to be added to the Function App via *Settings* > *Configuration*. All values can be sourced from the relevant resources in the Azure Portal. For access to Glasswall Halo either use your instance and authentication, or for trial purposes [contact our sales team](https://www.glasswall.com/contact) for basic authentication credentials to use our public endpoint. #### Configuration | Key | Value | | --- | --- | | Azure_servicebus_connectionstring | Connection string for service bus | | Azure_servicebus_newfiles_queuename | Queue name for new blob events | | Azure_storage_account_name | Source storage account name | | Azure_storage_account_key | Source storage account key | | Azure_storage_destination_connectionstring | Destination storage account connection string | | cdr_username | Username for Halo API basic auth | | cdr_password | Password for Halo API basic auth | | cdr_url | URL for CDR-file endpoint (e.g. `https://api.glasswall.com/api/v3/cdr-file`) | Once all the steps are complete, any file placed in the source Azure Blob Storage account is automatically protected by Glasswall Halo and written to the destination storage account. If a file cannot be protected, a text file containing an error message is written instead. ## Objective Set up automated file sanitization in Google Cloud Storage using the Glasswall Halo API via a Google Cloud Run service and Eventarc. ## Google Cloud Storage Integration Guide - [Prerequisites](/halo/google-cloud-storage-integration#prerequisites) - [Step 1 – Setup](/halo/google-cloud-storage-integration#step-1--setup) - [Step 2 – Deploy Google Cloud Run](/halo/google-cloud-storage-integration#step-2--deploy-google-cloud-run) - [Step 3 – Set up storage events](/halo/google-cloud-storage-integration#step-3--set-up-storage-events) - [Step 4 – Test application](/halo/google-cloud-storage-integration#step-4--test-application) * * * ## Prerequisites - Google Cloud Console account - Source Google Cloud bucket - Destination Google Cloud bucket - [Google Cloud CLI](https://cloud.google.com/sdk/docs/install) **Notes:** - Before you begin, ensure that all configuration parameters are correctly named and added to the Google Cloud Run application, as any mismatches will lead to failures. - This guide covers flat files only and does not include detailed logging. * * * ## Step 1 – Setup Google provides a similar scenario that can be used to get started. Follow the [Before you begin guide](https://cloud.google.com/run/docs/tutorials/eventarc#before-you-begin) for permissions, service accounts, and command-line setup. Once completed, you should be ready to deploy your application. Also ensure that you have completed the following steps: 1. Sign in to the [Google Cloud Console](https://console.cloud.google.com). 2. Prepare your application code in a working directory. 3. Run the following commands: ```shell gcloud auth login gcloud config set project "${PROJECT_ID}" ``` `PROJECT_ID` can be found on the welcome page of the Google Cloud Platform console. * * * ## Step 2 – Deploy Google Cloud Run In the following steps, replace `${SERVICE_NAME}` with the desired name of the deployed Cloud Run application. 1. To deploy from the command line, use the `gcloud run deploy` command: ```shell gcloud run deploy "${SERVICE_NAME}" --source . --region "${SERVICE_REGION}" --set-env-vars OutputBucket="${OUTPUT_BUCKET_NAME}" --set-env-vars HALO_URL="${HALO_URL}" --set-env-vars HALO_USERNAME="${HALO_USERNAME}" --set-env-vars HALO_PASSWORD="${HALO_PASSWORD}" ``` 2. Configuration is performed during deployment. Replace the variables enclosed in `${}` with the appropriate values. | Key | Value | | --- | --- | | Service_name | Name of the Cloud Run service to be created | | Service_region | Region where the Cloud Run service will be deployed | | Output_bucket_name | Destination storage bucket name (must already exist) | | Halo_url | URL for the CDR-file endpoint (for example, `https://api.glasswall.com/api/v3/cdr-file`) | | Halo_username | Username for Halo API basic authentication | | Halo_password | Password for Halo API basic authentication | Refer to the [Google Cloud Run deploy documentation](https://cloud.google.com/sdk/gcloud/reference/run/deploy) for more details. * * * ## Step 3 – Set up storage events To configure storage events, create a Google Eventarc trigger using the following command: ```shell gcloud eventarc triggers create "${SERVICE_NAME}" --destination-run-service="${SERVICE_NAME}" --destination-run-region="${SERVICE_REGION}" --destination-run-path="/" --location="${SERVICE_REGION}" --event-filters="type=google.cloud.storage.object.v1.finalized" --event-filters="bucket=${INPUT_BUCKET_NAME}" --service-account="${PROJECT_NUMBER}-compute@developer.gserviceaccount.com" ``` * * * ## Step 4 – Test application To test the application, upload supported files to the input bucket. Successfully processed files will appear in the output bucket. ## Objective Set up the Glasswall Halo ICAP server to sanitize or block files as they pass through your security infrastructure. ## Glasswall Halo’s ICAP Server Integration Guide * [Introduction](/halo/integrations/icap-server-integration#introduction) * [Step 1 - Deploy Glasswall Halo](/halo/integrations/icap-server-integration#step-1---deploy-glasswall-halo) * [Step 2 - Deploy the ICAP server](/halo/integrations/icap-server-integration#step-2---deploy-the-icap-server) * [Step 3 - Configure your ICAP server client](/halo/integrations/icap-server-integration#step-3---configure-your-icap-server-client) * [Step 4 - Configure the default ICAP profile](/halo/integrations/icap-server-integration#step-4---configure-the-default-icap-profile) * [Step 5 - Test by downloading files](/halo/integrations/icap-server-integration#step-5---test-by-downloading-files) * * * ### Introduction Our ICAP server mediates between ICAP clients (such as proxy servers, firewalls, or gateways) and the Glasswall Halo service. when a file passes through the ICAP client, it is sent to Glasswall Halo for sanitization (CDR) via ICAP before returning to the client. This ensures complete control of content encountered by members of your organization as they browse the web. you will be able to define the type of files allowed into your organization and what risky content should be sanitized from them to meet your risk profile needs. #### What is CDR? CDR (Content Disarm and Reconstruction) uses a unique method to keep organizations and individuals safe from file-based threats. Instead of trying to detect threats like most cybersecurity solutions, Glasswall CDR follows a Zero Trust approach. This means that only files that have been checked and cleaned by Glasswall are considered safe because all potential threats have been removed. We don't try to find malicious code; we simply take away its ability to harm the document. Learn more about Glasswall CDR in the main documentation #### What is ICAP? The Internet Content Adaptation Protocol (ICAP) is a lightweight protocol designed to enhance transparent proxy servers by offloading tasks like virus scanning and content filtering to dedicated ICAP servers. This allows web servers to focus on http traffic, improving efficiency. ICAP leverages edge devices to deliver specialized services such as ad insertion, content translation, and multi-av scanning. Our ICAP server mediates between ICAP clients (such as proxy servers, firewalls, or gateways) and the Glasswall Halo service. When a file passes through the ICAP client, it is sent to Glasswall Halo for sanitization via ICAP before returning to the client. This ensures complete protection of content traversing your zero-trust boundaries, whether incoming or outgoing from your environments. [Learn more about our ICAP Server](/halo/about-glasswall-icap) * * * ### Step 1 - Deploy Glasswall Halo Before you can begin the ICAP server integration, you’ll need to ensure that Glasswall Halo is deployed and configured. This is because our ICAP server is integrated with Glasswall Halo. Please refer to our easy-to-follow [deployment guides](/halo/glasswall-halo-deployment-overview), or skip this step if you’ve already got Halo setup. * * * ### Step 2 - Deploy the ICAP server If you didn't deploy the ICAP server during your Halo deployment, please refer to our step-by-step [deployment guide](/halo/icap-deployment). * * * ### Step 3 - Configure your ICAP server client After the ICAP server is operational, you'll need to configure your ICAP client to route its requests through the server. Our ICAP server is compatible with any ICAP-enabled web security appliance. For additional support, please see our [integration guides](/halo/icap-configuration-overview) for popular clients and our [proxy configuration guides](/halo/proxy-configuration) for help setting up proxy environments. * * * ### Step 4 - Configure the default ICAP profile Next, we'll set up the default ICAP profile to manage how different file types are handled. ICAP server requests use a content type known as media types, which can be tricky to interpret. To make it easier, we've matched these media types to the file types supported by Glasswall Halo and other common formats - please refer to the [supported media types](/halo/glasswall-supported-media-types) for more information. **Note:** you can create multiple ICAP profiles and assign one to each ICAP request. If none is specified, this default profile will be used. * Configure the ICAP profile through the [ICAP profile management API](https://api.docs.glasswall.com/profile-v1#ICAP-profile-management-API) endpoint or directly via [Halo’s ICAP settings](https://halo.glasswall.com/). * For each mapped file type, you can define what action the ICAP server should take. * Here, we'll process Word, Excel, and PowerPoint files, block PDFs, and bypass all other file types. * Next, configure your policy settings. * You may leave Glasswall's default content management policy which sanitizes all risky content from processed files to ensure maximum protection or amend them as required. * * * ### Step 5 - Test by downloading files You're now ready to browse securely. * To test the file sanitization, download a Word, Excel, or PowerPoint file. * As the file downloads, your web security appliance will intercept it and send it to Glasswall Halo via ICAP. Here it is sanitized in under a second with no noticeable change in user experience. #### Error handling * Since PDFs are set to be blocked by this default profile, test it by trying to view or download a PDF. * The file should be replaced with a Halo protection report explaining why the content is unavailable, preventing the potentially dangerous file from reaching the user. ## Conclusion With all steps completed, your ICAP server is now ready to handle and sanitize files using Glasswall Halo’s Content Disarm and Reconstruction (CDR) process. The ICAP server mediates between clients and Glasswall Halo, enforcing security policies, automating file handling, and enhancing protection while ensuring minimal disruption to user experience. Introducing the XML Validation API, a new tool designed to facilitate XML validation by ensuring that your XML markup is secure and well-formed in accordance with the specified schema. The `/api/v1/validate-xml` endpoint accepts an XML `file` and an XSD `schema`. If any errors are detected when parsing the XML or schema, a `400 Bad Request` response will be returned. Upon validation, a `200 OK` response will be issued along with any validation errors that were detected against the provided schema. **Note:** DTD processing is disabled. When a DTD is encountered, validation will fail with a message stating that DTDs are prohibited. Minimum Halo version: [v2.6.0](/halo/v260) [API documentation](https://api.docs.glasswall.com/#synchronous-API) --- ## Licensing ### Accessing the XML Validation API A new license is required to access the XML Validation API. If you would like to add XML validation to your Halo deployment, please [contact us](https://www.glasswall.com/support) to obtain a license that includes this functionality. ### Daily file limit and daily data limit Utilizing the XML Validation API counts toward your daily file and data usage. If your Halo file processing limit is exceeded, any new requests to the XML Validation API will fail in the same manner as synchronous or asynchronous API requests. **Note:** only the XML portion of the request contributes to daily file and data usage. The XSD schema is not counted. ## Policy settings The default policy contains recommended settings offering a balance of usability and XML attack mitigation. Here's an example XML policy: ```json { "policySettings": { "EnableWarningHandling": false, "BlockedTags": [ "/XInclude:include", "/XSL/Transform:stylesheet" ], "AllowedEncodings": [ "utf-8", "utf-16", "iso-8859-1" ], "AllowEmptyEncoding": true, "AllowDoctypeInCData": false } } ``` ### EnableWarningHandling `PolicySettings.EnableWarningHandling`Tells the XML validation API to return warnings in the response when validating the input XML. Warnings are disabled by default. The warnings are returned in a list. if none are found the list is empty: ```json { "ValidationWarnings": [ "The optional field Test is missing." ] } ``` ### BlockedTags Custom tags are allowed in XML documents. Specifying potentially risky tags in `PolicySettings.BlockedTags` allows the XML Validation API to fail if it encounters any of these tags while validating the input XML. The `ValidationResults` object on the response identifies which tag caused validation to fail. These blocked tags match tags in the XML document by an expanded tag name that includes the namespace. This ensures the correct tag is blocked and aliasing namespaces differently cannot circumvent the check. Tags are matched by how they end so that a single rule can block namespaces defined for different years, e.g. `http://www.w3.org/2001/XInclude` and `http://www.w3.org/1999/XML/xinclude`. For example in the following XML: ```XML ... ``` The `xi` namespace is set to`http://www.w3.org/2001/XInclude` and is used to prefix the `xi:include` tag. The validation of this tag is therefore evaluated to `http://www.w3.org/2001/XInclude:include`. This expanded tag name now ends with one of the default blocked tags`/XInclude:include` and is added to the validation results. These checks are also case insensitive. By default, the following XML tags are blocked: - XInclude - XSL ### AllowedEncodings XML documents may specify a character encoding standard.`PolicySettings.AllowedEncodings` is a list of strings, allowing users to specify which character encoding standard to allow in the input XML. If the XML Validation API encounters a specified encoding, validation will fail and the encoding standard is identified in the `ValidationResults` object on the response. By default, the following character encoding standards are allowed: - UTF-8 - UTF-16 - ISO-8859-1 ### AllowEmptyEncoding `PolicySettings.AllowEmptyEndoding` configures the ability to allow validation to pass if no character encoding is specified in the input XML. **Note:** even with this is enabled, validation will still fail if the input XML specifies an encoding standard that is not included in `PolicySettings.AllowedEncodings`. This is set to true by default. ### AllowDoctypeInCData CDATA sections can be used to embed entire XML documents within other XML documents.`PolicySettings.AllowDoctypeInCData` configures the XML Validation API to fail validation if it encounters a doctype declaration in the CDATA section. Alternatively this can be set to true to allow doctype declarations. This is set to false by default. The XML Policy Management API allows users to create new XML validation policies in Glasswall Halo. ## Creating a new XML policy ### 1. Make a post request to the XML Policy Management API To submit a new ICAP profile, send a post request to the `api/v1/xml-policies/` endpoint. Include a JSON request body that contains the policy. The policy name is specified as a path parameter. **Example:** ```json { "policySettings": { "EnableWarningHandling": false, "BlockedTags": [ "/XInclude:include", "/XSL/Transform:stylesheet" ], "AllowedEncodings": [ "utf-8", "utf-16", "iso-8859-1" ], "AllowEmptyEncoding": true, "AllowDoctypeInCData": false } } ``` If the request body is valid, the response will contain a summary of the newly created policy. ### 2. Use the policy in the XML Validation API Heading back to the XML Validation API, the saved policy can be utilised via the `policyName` query parameter. Example post request:`/api/v1/validate-xml?policyName=`. If desired, a dynamic policy can be used. Provide a JSON value to the multipart/form field labelled`policy`. If no policy name and no dynamic policy is specified, the default policy is used. #### Learn more [XML Policy Management API documentation](https://api.docs.glasswall.com/xml-validation-policy-v1#XML-policy-management-API) [XML Validation API documentation](https://api.docs.glasswall.com/xml-validation-v1#XML-validation-API) Glasswall Halo provides the ability to export and import content items for [supported file types](/halo/glasswall-halo-supported-file-types). This facilitates the option for additional processing outside of Glasswall Halo, before being imported back into Glasswall Halo to be recomposed for usage. Minimum required Halo version: [v2.8.0](/halo/v280) ## Exportable content - Users can export either a binary file or a base64 encoded file. Once complete, the response is a standard zip archive or base64 encoded zip archive respectively. - The maximum file size for export is 1 GB. **Note:** archives are not supported. These endpoints accept only a single file for processing. This means that the uploaded file, or base64 encoded file, should not be an archive (e.g. zip, tar, rar) or contain multiple files. ## Importable content - Users can import either a standard zip archive or a base64 encoded zip archive. Once complete, the response is a binary file or base64 encoded file respectively. - The maximum zip archive size for import is 3 GB. **Note:** these endpoints accept only a single flat zip file for processing. This means that the uploaded zip file, or base64 encoded zip file, should not contain any nested zip files within it. ## Additional information For more information on how to use this API, see our [API documentation](https://API.docs.glasswall.com/#synchronous-API-import-and-export). Glasswall Halo is built around the Glasswall Embedded Engine, if you wish to learn more about import and export, please visit our [Embedded Engine documentation](/embedded-engine/embedded-engine-content-export-and-import) for an in-depth explanation. ## Licensing ### Accessing the Import and Export API A license for Glasswall Embedded Engine import and export is required to access the API. If you would like to add export and import functionality to your Halo deployment, please [contact us](https://www.glasswall.com/support) to obtain a new license that includes this. ### Daily file limit and daily data limit Utilising the Import and Export API will count towards your daily file and data usage. If your Halo file processing limit is exceeded, any new requests to the import and export API will fail in the same manner as they would with the sync or async APIs. **Preview feature** Glasswall Conform can automatically repair PDF files that are unable to be processed due to malformed or corrupt structures. It extracts visible content, like text, images, and graphics, and rebuilds the file to meet PDF standards, making it compatible with Glasswall's CDR technology. [Learn more about Glasswall Conform](/embedded-engine/glasswall-conform-overview) ## Constraints and limitations While Glasswall Conform is a powerful tool, certain constraints and limitations should be considered: - Image handling: some image colour spaces are unsupported and may be ignored. additionally, image processing may convert compressed images to a lossless format, which can increase file size. - Font handling: Glasswall Conform supports base 14 and many Microsoft fonts, but unsupported custom fonts are replaced to mitigate potential risks. - PDF structure: PDFs missing essential structural elements (e.g., root catalog, cross-reference tables) may not be recoverable. - Memory usage: PDFs with many images may consume significant memory. While the tool has been tested with files up to 50 MB, larger files may experience performance issues. - Color spaces: the CalRGB colour space is not supported. - Graphics handling: support for complex graphics, such as shapes, charts, and graphs, is limited. This version prioritises text integrity. - Document recovery: severely corrupted pdfs or those with missing structural elements may be unrecoverable. - Blocked content items: malformed PDF files containing content items blocked by policy may reach the end user because conform might remove the blocked content items before processing by the Engine. ## Using Glasswall Conform to process irregular PDFs Glasswall Conform enhances PDF processing by allowing irregular or malformed documents to be safely cleaned when standard validation would normally fail. It is available across both the Sync API and the Halo Portal. ### Prerequisite * Halo version **2.10.0** or later ### Supported endpoints Conform is currently supported on the following sync API endpoints: * `/api/v3/cdr-file` * `/api/v3/cdr` ## How to enable Conform ### Option 1: Sync API To process a file using conform via the API: 1. Send a`POST`Request to the sync API endpoint with the`pdf-conform=true`Query parameter: ``` /api/v3/cdr-file?pdf-conform=true ``` 2. After processing, check the response headers for: * `x-pdf-conform` This indicates whether Glasswall Conform was used during processing. ### Option 2: Halo Portal In the **Clean a file** section of the Halo Portal, enable the **Process irregular pdfs** toggle. If Conform is applied during file cleaning, the analysis report will include additional details indicating why it was used. **Halo Storage Monitoring** is a powerful service that enables automatic sanitization of [supported file types](/halo/glasswall-halo-supported-file-types) at the point of upload. It can continuously monitor a wide range of cloud storage resources and sanitize files in real time. When a file is uploaded to a monitored resource, storage monitoring automatically: 1. Downloads the file. 2. Sends it to the Glasswall Embedded Engine for sanitization. 3. Re-uploads a clean version of the file back to the original location. --- ## Prerequisites - Halo [v2.11.0](/halo/v2110) or later - A valid [Glasswall Halo license](https://www.glasswall.com/support) - Publicly reachable cluster ingress address - Alternatively, you may whitelist the IPs of your chosen storage solution to ensure proper connectivity - Monitor must be configured for each storage resource - For SharePoint integration, storage monitoring must be [registered with Microsoft graph API](https://learn.Microsoft.com/en-us/graph/auth-register-app-v2) --- ## Using storage monitoring To get started, follow our [Storage Monitoring Setup Guide](/halo/storage-monitoring-setup). The setup process differs for new and existing Halo installations. More technical details are available in the [Storage Monitoring API documentation](https://api.docs.glasswall.com/storage-monitoring-V1#storage-monitoring-API). --- ## Supported cloud storage solutions Currently, Halo storage monitoring supports: - [Microsoft SharePoint](/halo/storage-monitoring-sharepoint) - [Microsoft OneDrive](/halo/storage-monitoring-onedrive) - [Microsoft Outlook](/halo/storage-monitoring-outlook) --- ## Data visibility When storage monitoring is active, system administrators have visibility of the filenames of all files processed by the service. This includes files from SharePoint document libraries, OneDrive user drives, and Outlook email attachments. Organizations should review their internal data handling and privacy policies before enabling storage monitoring and ensure that users are informed that file and attachment names will be visible to administrators through the Halo portal and API. --- ## Additional information Glasswall Halo is powered by the [Glasswall Embedded Engine](/embedded-engine/about-embedded-engine), which rebuilds files to a safe standard while preserving usability. Learn more about how it works and why it's different from traditional detection-based tools. --- ## Licensing and limits ### Accessing storage monitoring A valid Glasswall Halo license is required to access the Storage Monitoring API. Please [contact us](https://www.glasswall.com/support) to obtain or upgrade your license. ### Daily usage limits Use of Storage monitoring contributes to your **daily file** and **data usage** limits within Halo. - If the file processing limit is exceeded the original file is re-uploaded **unchanged**. ## Enabling storage monitoring By default, storage monitoring is **disabled** (replicas set to 0). Administrators can enable it by scaling the deployment to 1 replica using the following `kubectl` command: ```bash kubectl scale deployment/storage-monitor --namespace cdrplatform --replicas=1 ``` To enable StorageMonitoring in the Portal, set the `EnabledPages` config in the `portal-configuration` configmap. ```bash enabled_pages="SystemSettings\,PolicySettings\,SharePointMonitoring" helm get values -n cdrplatform cdrplatform-portal > portal.yaml helm upgrade --install cdrplatform-portal cdrplatform-portal --set configuration.EnabledPages="${enabled_pages}" \ -f portal.yaml -n cdrplatform ``` ### Integration requirements Once enabled, to ensure storage monitoring integrates correctly with external cloud storage services, one of the following conditions must be met: - The cluster must have a **publicly accessible ingress address**, > **or** - The **IP addresses** of your storage service must be **whitelisted** to allow inbound connectivity. For example, integration with **SharePoint** requires access to specific **Microsoft Graph API IP ranges**, which must be reachable by the storage monitoring component. ## Required app registration To enable Halo to monitor M365 cloud storage, an application must be [registered in Microsoft Entra ID](https://learn.Microsoft.com/en-us/graph/auth-register-app-v2) with the appropriate Microsoft graph API permissions. **Note:** you’ll need access to a Microsoft Entra tenant and an account with at least the **Cloud application administrator** role. You can either follow the steps below to manually create the app registration or use this provided [shell script](/.attachments/create-azure-app-registrations-storage-monitor.sh) to automate the process. ### Register the application 1. Sign in to the [Microsoft Entra Admin Center](https://entra.Microsoft.com/). 2. If needed, switch to the desired tenant via **Settings > Directories + subscriptions**. 3. Navigate to **Identity > Applications > App registrations** and click **New registration**. 4. Enter a **Name** for the application. 5. Under **Supported account types**, click **Accounts in this organizational directory only**. 6. Leave the **Redirect URI** blank. 7. Click **Register**. After registration, make note of the following from the **Overview** pane: - **Application (client) ID** - **Directory (tenant) ID** These values are required during Halo setup. ### Generate a client secret 1. Navigate to **Certificates & secrets** > **Client secrets**. 2. Click **New client secret**. 3. Provide a description and select an expiration period (or specify a custom lifetime). 4. Click **Add**. 5. **Copy and save the client secret value immediately.** You won’t be able to view it again after leaving the page. **Note:** client secrets expire after 24 months or less. If expired, repeat the steps above to generate a new one. ### Configure API permissions 1. In the app's **API permissions** pane, click **Add a permission**. 2. Click **Microsoft graph** > **Application permissions**. 3. Grant the following permissions: - For SharePoint: - `Files.ReadWrite.All` - `Sites.Read.All` - For OneDrive: - `Files.ReadWrite.All` - `User.ReadBasic.All` - For Outlook: - `User.ReadBasic.All` - `Mail.ReadWrite` 4. Click **Add permissions**. 5. If permissions show as **Not granted**, click **Grant admin consent**. ### Additional permissions for Outlook email banner (optional) If you plan to use the [Outlook email disclaimer banner](/halo/storage-monitoring-outlook#email-disclaimer-banner), the app registration also requires Exchange Online permissions: 1. In the app's **API permissions** pane, click **Add a permission**. 2. Click **APIs my organization uses** and search for **Office 365 Exchange Online**. 3. Click **Application permissions**. 4. Grant the following permission: - `Exchange.ManageAsApp` 5. Click **Add permissions** and **Grant admin consent**. 6. Navigate to **Microsoft Entra ID** > **Roles and administrators**, find the **Exchange Administrator** role, and assign it to your app registration. For full certificate setup instructions, see [Setup Exchange Online for the Outlook banner](/halo/storage-monitoring-banner-setup-exchange-online). Once these steps are complete, you can proceed with deploying Glasswall Halo with storage monitoring enabled. ## Storage monitoring OneDrive integration The Glasswall Halo Storage Monitoring service integrates seamlessly with OneDrive user drives, automatically sanitizing [supported file types](/halo/glasswall-halo-supported-file-types) as they are uploaded to monitored drives. --- ## Prerequisites Before configuring OneDrive monitoring, ensure the following: - The **Storage monitoring service** is deployed and running. - An **application is registered** in Microsoft Entra ID with the appropriate permissions. - The application’s **Client ID**, **Tenant ID**, and **Client Secret** are stored in Azure key vault. - You have access to the **Halo Storage Monitoring API** For guidance, refer to the [Halo Storage Monitoring Setup Guide](/halo/storage-monitoring-setup). --- ## Monitoring a user's drive To set up monitoring, you’ll need: - **User ID** of the OneDrive owner You can retrieve the user ID using an endpoint provided by the Halo Storage Monitoring API. see the [Storage Monitoring API documentation](https://api.docs.glasswall.com/storage-monitoring-V1) for details. ### Optional configuration You can optionally customize each monitor with: - A **custom policy** If no custom settings are applied, the default policy is used. Monitors can be updated or removed at any time to change policies. **Note:** policies linked to active monitors are locked and cannot be deleted. To remove a policy, first reassign or delete any associated monitors. --- ## How monitoring works Once activated, the monitor automatically performs the following for each supported file uploaded to the monitored drive: 1. Checks out the file 2. Downloads the file 3. Applies sanitization 4. Re-uploads the cleaned file 5. Checks the file back in If Halo cannot process an uploaded file, the original file remains unchanged. **Note:** OneDrive retains version history for all files, allowing users to view, download, or restore earlier versions — including the original, pre-sanitized file. ## Storage monitoring SharePoint integration The Glasswall Halo storage monitoring service supports integration with SharePoint document libraries, allowing [supported file types](/halo/glasswall-halo-supported-file-types) to be sanitized automatically when uploaded to a monitored library. --- ## Monitoring a SharePoint document library To configure monitoring, users must supply: - The **Site ID** - The **Drive ID** (i.e. the document library ID) These details can be retrieved using endpoints provided by the Halo Storage Monitoring API. For more information, refer to [Storage Monitoring API](https://api.docs.glasswall.com/storage-monitoring-V1). Each monitor can optionally be configured with: - A **custom policy** If no custom configuration is provided, the monitor will use the default policy. Monitors can be updated or removed at any time to adjust policies. **Note:** policies used by monitors are locked and cannot be deleted until they are unlinked. To unlock a policy, reassign affected monitors to a different policy or delete the monitors. Once a monitor is active, it automatically performs the following for each supported file uploaded to the monitored document library: 1. Checks the file out 2. Downloads the file 3. Applies sanitization 4. Re-uploads the clean file 5. Checks the file back in If Halo cannot process an uploaded file, the original file remains unchanged. **Note:** SharePoint maintains version history for all files, enabling users to view or restore earlier versions — including the original (pre-sanitized) upload. --- ## SharePoint metadata considerations When files, particularly OOXML documents, are uploaded to SharePoint, SharePoint may append a metadata package to the file. This is influenced by the document library and content type configuration. **Important:** this added metadata may cause sanitized files to appear “risky” if downloaded again and re-analyzed. The metadata package may include: - Tag fields, OCR text, media dates, structured search properties - SharePoint form templates for user interaction - Structures supporting taxonomy and enterprise metadata - InfoPath and Office integration components Depending on your organization’s SharePoint setup, **sensitive metadata** might also be included, such as: - Internal content type structures - Custom fields or taxonomies - User-specific metadata (e.g.,`dc:creator`,`lastModifiedBy`, Workflow IDs) - `GUIDs`Or`itemIDs`Which, while not inherently dangerous, may expose internal system patterns useful to attackers. ## Storage monitoring Outlook integration The Glasswall Halo Storage Monitoring service integrates with Microsoft Outlook mailboxes, automatically sanitizing [supported file types](/halo/glasswall-halo-supported-file-types) attached to inbound emails. --- ## Prerequisites Before configuring Outlook monitoring, ensure the following: - The **Storage monitoring service** is deployed and running. - An **application is registered** in Microsoft Entra ID with the appropriate permissions. If you already have an app registration for SharePoint or OneDrive monitoring, you can reuse it — just add the Outlook permissions (`User.ReadBasic.All` and `Mail.ReadWrite`). - The application's **Client ID**, **Tenant ID** and **Client Secret** are stored in Azure key vault. - You have access to the **Halo Storage Monitoring API**. For guidance, refer to the [Halo Storage Monitoring Setup Guide](/halo/storage-monitoring-setup). :::info If you are adding Outlook permissions to an existing app registration that is already in use for SharePoint or OneDrive, you must **restart the Storage Monitor service** after granting the new permissions. The service caches its permissions on startup, so a restart is required for the new permissions to take effect. ::: --- ## Monitoring a user's mailbox To set up monitoring, you'll need: - **User ID** of the mailbox owner You can retrieve the user ID using an endpoint provided by the Halo Storage Monitoring API. See the [Storage Monitoring API documentation](https://api.docs.glasswall.com/storage-monitoring-V1) for details. ### Optional configuration You can optionally customize each monitor with: - A **custom policy** If no custom settings are applied, the default policy is used. Monitors can be updated or removed at any time to change policies. :::note Policies linked to active monitors are locked and cannot be deleted. To remove a policy, first stop or delete any associated monitors. ::: --- ## How monitoring works Once activated, the monitor automatically performs the following for each supported file attached to an inbound email: 1. Detects the new email via a webhook notification 2. Downloads each attachment 3. Applies sanitization 4. Re-uploads the cleaned attachment, prefixed with `GW-` 5. Removes the original unsanitized attachment If the filetype of the attachment is unsupported, the original file remains unchanged **without** the `GW-` prefix. :::note Files embedded directly in the HTML body of an email (such as inline images and email signatures) are **not** processed by Halo. Only discrete file attachments are sanitized. ::: --- ## Identifying processed and unprocessed attachments After Halo processes an email, you can determine the status of each attachment by its filename: - **Processed (safe):** attachments prefixed with `GW-` (e.g. `GW-report.docx`) have been successfully sanitized by Glasswall. - **Unprocessed:** attachments **without** the `GW-` prefix remain in their original, unsanitized state. This may occur if the file type is unsupported or if processing failed. Exercise caution when opening these files. --- ## Blocking unprocessed attachments By default, Halo will **remove supported attachments that fail processing** from the email rather than leaving the original unsanitized file. This ensures that for supported file types only successfully sanitized files remain in the user's inbox. This behavior is controlled by the `MONITORING__BlockUnprocessedAttachments` configuration setting, which defaults to `true`. When disabled (`false`), attachments that Halo cannot process are left in the email in their original state **without** the `GW-` prefix. --- ## Email disclaimer banner (optional) {#email-disclaimer-banner} Outlook monitoring works without the email banner — it is an optional feature that can be enabled separately. Halo will still sanitize attachments regardless of whether the banner is configured. The banner is a notice prepended to the top of inbound emails that contain attachments, informing the recipient that Glasswall is processing their attachments. It helps users identify which emails have attachments that are being or have been processed by Halo. There are two options for setting up the banner: ### Option 1 — Halo manages the banner automatically When Exchange Online certificate authentication is configured, Halo manages the banner lifecycle automatically — creating the distribution group and transport rule on startup and keeping membership in sync with active monitors. See [Enabling the Outlook email banner via Helm](/halo/Deployment/Storage-Monitor-Outlook-Banner/Enabling-the-Email-Banner-via-Helm) for deployment instructions. ### Option 2 — Manual setup with PowerShell scripts If you prefer to manage the banner externally without granting Halo access to Exchange Online, you can run the PowerShell scripts standalone. A certificate is **not required** for standalone use — you can authenticate interactively via a browser login prompt. See [Enabling the Outlook email banner via PowerShell](/halo/Deployment/Storage-Monitor-Outlook-Banner/Enabling-the-Email-Banner-via-PowerShell) for step-by-step instructions. --- ## Known limitations - **Distribution group caching:** when removing the banner from a user by updating the distribution group, changes can take several hours to take effect. Exchange Online caches the distribution group membership used by transport rules, so the banner may continue to appear for users who have been removed until the cache refreshes. - **Mobile and web clients may require a manual refresh:** users on the Outlook mobile app or Outlook on the web may need to manually refresh their inbox to see rebuilt attachments. The desktop client typically reflects changes automatically. ## Prerequisites - An app registration for Exchange Online access (you will need its **Application (Client) ID** and your tenant's **Organization** domain, e.g. `contoso.onmicrosoft.com`) - This can be created after following [Registering with Microsoft Entra ID](/halo/registering-with-microsoft-entra-id) - Azure CLI (ensure the CLI is logged in with `az login`) - OpenSSL (`openssl`) - (Optional) PowerShell Core (`pwsh`) - To skip the requirement for `pwsh` and `Connect-ExchangeOnline` validation, use the `--skip-verify` option ## (Automated) Setting up the app registration to connect to Exchange Online The StorageMonitor connects to Exchange Online using app-only (certificate-based) authentication via the `Connect-ExchangeOnline` cmdlet. This requires a `.pfx` certificate registered against an Azure AD (Entra ID) app registration. :::note You’ll need access to a Microsoft Entra tenant and an account with at least the **Privileged Role Administrator** role. ::: You can use [setup-exchange-online-certificate.sh](/.attachments/setup-exchange-online-certificate.sh) to automate the process of setting up the app registration and generating the certificate. Usage: ```sh ./setup-exchange-online-certificate.sh --app-id --organization .onmicrosoft.com ``` :::note The `organization` should be your tenant name, suffixed with `.onmicrosoft.com`, but may be your primary custom domain instead. You can find the primary domain using the following commands: ```powershell Connect-ExchangeOnline # If not already connected Get-AcceptedDomain | Where-Object { $_.Default -eq $true } ``` ::: The script is configurable with some options, to view them run with `--help`. ```sh ./setup-exchange-online-certificate.sh --help ``` The following are configurable: - Certificate name prefix (`StorageMonitorExchangeOnline` by default) - Certificate validity period (1 year by default) - Output directory for the `.cer` and `.pfx` files (./ by default) - Option to skip PowerShell stages, removing `pwsh` as a requirement The email banner is an **optional** feature for Outlook monitoring. Halo will sanitize email attachments regardless of whether the banner is configured. When enabled, Storage Monitor will: - mount an Exchange Online certificate from the `cdrplatform-secrets` secret using the key `exchange-online-cert` - set `EXCHANGEONLINE__CertificatePath=/mnt/exchange/exchange-certificate.pfx` - use `EXCHANGEONLINE__AppId` and `EXCHANGEONLINE__Organization` from chart configuration - create the Exchange Online distribution group and transport rule on startup - keep the distribution group synchronized with Outlook monitors in the background ## Prerequisites - You have Outlook monitoring already configured for the service - You have [registered your application with Microsoft Entra ID](/halo/registering-with-microsoft-entra-id) with the required Exchange Online permissions - You have a `.pfx` certificate with private key for the Exchange Online app registration — see [Setup Exchange Online for the Outlook banner](/halo/storage-monitoring-banner-setup-exchange-online) for certificate creation instructions - You know the Exchange Online app registration `AppId` - You know the Exchange Online `Organization` value, for example `glasswall.com` - You have version 2.4.0+ of the cdrplatform-storage-monitor Helm chart ## Deployment Add the certificate to the existing `cdrplatform-secrets` secret in the target namespace: ```bash # Linux-only kubectl -n cdrplatform patch secret cdrplatform-secrets \ --type merge \ -p "{\"data\":{\"exchange-online-cert\":\"$(base64 -w0 " \ --set configuration.EXCHANGEONLINE__Organization="" -f cdrplatform-storage-monitor/values.yaml \ --atomic ``` ## Notes - To disable the feature again, set `ENABLE_OUTLOOK_BANNER` back to `false` and redeploy through Helm. If you prefer to manage the Outlook email banner externally rather than granting Glasswall Halo access to Exchange Online, PowerShell scripts are provided for standalone use: - Install-HaloOutlookBanner.ps1 - Invoke-HaloOutlookBannerSynchronization.ps1 - Remove-DistroGroupMember.ps1 ## Prerequisites - Have your Exchange Online authentication details ready — either a username/email (for interactive login) or a PFX certificate - Install the `ExchangeOnlineManagement` module if it is not already present (the scripts will install it automatically if missing) --- ## Step 1 — Install the Outlook banner Run `Install-HaloOutlookBanner.ps1` to create the Distribution Group and Transport Rule. ### Authenticating with Exchange Online - Using interactive browser login (recommended) `powershell .\Install-HaloOutlookBanner.ps1 -UserPrincipalName "admin@contoso.onmicrosoft.com"` OR - Using a PFX certificate `powershell .\Install-HaloOutlookBanner.ps1 -ExchangeCertificatePath "C:\path\to\certificate.pfx" -ExchangeAppId "your-azure-app-id" -ExchangeOrganization "contoso.onmicrosoft.com"` --- ## Step 2 — Synchronize the distribution group with Halo monitors ### Transport rules are cached :::note The Transport Rule will cache the Distribution Group, which expires after 4 hours. It could take up to 4 hours for changes to take effect. ::: Run `Invoke-HaloOutlookBannerSynchronization.ps1` whenever monitor changes are made, to keep the Distribution Group in sync. ### Authenticating with Exchange Online - Using interactive browser login (recommended) `powershell .\Invoke-HaloOutlookBannerSynchronization.ps1 -HaloUri "" -NoAuth -UserPrincipalName ""` OR - Using a PFX certificate `powershell .\Invoke-HaloOutlookBannerSynchronization.ps1 -HaloUri "https://your-halo-instance.com" -NoAuth -ExchangeCertificatePath "C:\path\to\certificate.pfx" -ExchangeAppId "your-azure-app-id" -ExchangeOrganization "contoso.onmicrosoft.com"` :::tip Replace `-NoAuth` if needed, with: `-HaloToken` (bearer token) **or** `-HaloCredential` (basic auth) ::: **Glasswall Halo** can be configured to enable **Single Sign-On** (SSO) with OpenID Connect ([OIDC](https://www.microsoft.com/en-us/security/business/security-101/what-is-openid-connect-oidc)). SSO is an authentication method that allows users to sign in using one set of credentials to multiple independent software systems. Using SSO means a user doesn't have to sign in to every application they use. Below is information on how to configure SSO within your Glasswall Halo solution using [Microsoft Entra ID](https://learn.microsoft.com/en-us/entra/identity/enterprise-apps/what-is-single-sign-on) as the identity and access management (IAM) provider. In principle, *any* IAM solution that offers OIDC may be configured to support SSO for Glasswall Halo. ## Supported identity providers Because Halo portal sign-in is built on OpenID Connect (OIDC), any OIDC-compliant identity provider (IdP) can be used. Glasswall provides step-by-step guidance for the following providers: | Identity provider | Guidance | Typical use | | --- | --- | --- | | **Microsoft Entra ID** (formerly Azure AD) | This page, including [Azure Government (GovCloud)](#azure-government-govcloud-deployments) | Cloud deployments using Microsoft 365 / Entra ID | | **Keycloak** | [Keycloak SSO integration](/halo/keycloak-sso-integration) | Air-gapped and on-premises deployments | | **Okta** | [Okta authorization server](/halo/okta-authorization-server), [Okta portal SSO](/halo/okta-portal-sso), [Okta API bearer authentication](/halo/okta-api-bearer-authentication) | Organizations standardized on Okta | Other OIDC providers (for example, Ping Identity or Google Workspace) are **not officially supported**, but can often be configured using the same pattern: point the portal and access services at your provider's OIDC authority and create the equivalent application registrations/clients. Because these providers are untested, you may encounter unknown issues—if you are planning to use another OIDC provider, [contact Glasswall Support](https://www.glasswall.com/support) first. ### SSO applies to portal sign-in, not the REST API SSO (OIDC) governs how **users sign in to the Halo portal**. The **Halo REST API authenticates separately**—when API authentication is enabled it uses **Basic authentication** (an organization ID and token) or a **Bearer token**, and it never uses the interactive OIDC sign-in flow. Enabling SSO on a deployment does not change how the API is called. This is why API requests against an SSO-enabled deployment still require Basic (or Bearer) credentials whenever API authentication is enabled, and why a `401 Unauthorized` on `/api/...` usually means the API credentials were never provisioned rather than an SSO problem. See: - [Authentication overview](/halo/glasswall-halo-authentication-overview)—how to authenticate API requests. - [Enable API authentication](/halo/eks-step-7#73c---enable-api-authentication)—how API Basic-auth credentials (organization ID and tokens) are provisioned and rotated. :::warning Avoid being locked out of the portal If SSO is the **only** configured sign-in method, administrators can be locked out if the identity provider becomes unreachable or misconfigured. Keep a recovery path available: - Retain a working set of **API Basic-authentication credentials** (organization ID and token). These are independent of SSO and can be used to call the API directly. - To regain administrative access to the portal without SSO, redeploy the `cdrplatform-portal` chart with SSO disabled (`--set configuration.OIDC=null`) or with `--set configuration.AutoAdmin=true`. See [Enable admin access without authentication](/halo/eks-step-7#75c---enable-admin-access-without-authentication). ::: :::note An expired license does not lock you out An expired or over-limit license disables the affected Halo features, but it does **not** lock administrators out of the portal. You can still sign in (using your configured sign-in method) to view and update the license—see [License management](/halo/glasswall-portal-license-management). ::: ## Prerequisites for Microsoft Entra ID integration - A Microsoft enterprise agreement for a Microsoft Entra ID account is required to enable this SSO option. - A domain name must be configured for the Halo portal (*portal service*) to use SSO authentication. - Azure App Registrations are required whenever SSO authentication using Microsoft Entra ID needs to be enabled. - Three App Registrations must be created: - `Cdrplatform-API-access` - `Cdrplatform-portal-access` - `Cdrplatform-portal-client` - The [attached shell script](/.attachments/create-azure-app-registrations.sh) helps create these app registrations. Enter the domain name you would like to use for the portal as an argument to the script. Example: ```bash bash create-azure-app-registrations.sh halo.glasswall.com ``` By default the script creates all three registrations. To set up only one type of authentication, pass `--skip-api` to create just the Portal registrations, or `--skip-portal` to create just the API registration: ```bash # Portal SSO only (skips api-access) bash create-azure-app-registrations.sh --skip-api halo.glasswall.com # API authentication only (skips portal-access and portal-client) bash create-azure-app-registrations.sh --skip-portal ``` Prefer to set them up by hand? Follow [Creating the Azure app registrations manually](/halo/creating-azure-app-registrations-manually) for the equivalent steps in the Microsoft Entra admin center. - The enterprise application `ar-Halo-portal-client` must be granted admin consent. --- ## Configuration ### Enable Microsoft Entra ID API authentication To enable Microsoft Entra ID based authentication, set `configuration.authenticationscheme=bearer`. Also set `tenant_id`, `api_domain` and `valid_audiences` variables while deploying the Helm chart. **Note:** Set `valid_audiences` to the **Application (client) ID** of the `cdrplatform-api-access` app registration (found in the Azure portal under **App registrations** > **Overview**). ```sh tenant_id="" api_domain="" valid_audiences="" # Application (client) ID of the cdrplatform-api-access app registration (copy from Azure portal) ``` ```sh helm upgrade cdrplatform-api-access cdrplatform-api-access \ --reuse-values \ --set ingress.tls.enabled=true \ --set ingress.tls.domain=${api_domain} \ --set ingress.tls.secretName=tls-secret \ --set configuration.AuthenticationScheme=Bearer \ --set configuration.Authentication__Schemes__Bearer__ValidAudiences__0=${valid_audiences} \ --set configuration.Authentication__Schemes__Bearer__ValidIssuer=https://sts.windows.net/${tenant_id}/ \ --set configuration.Authentication__Schemes__Bearer__Authority=https://login.microsoftonline.com/${tenant_id}/v2.0/ \ --atomic ``` --- ### Configure SSO login for Glasswall Halo Set `portal_domain`, `portal_client_id`, `tenant_id` variables in the commands below. `portal_domain` is the domain used for TLS on the portal service and is also the domain used in the `cdrplatform-portal-client` App Registration. `portal_client_id` is the application (client) ID of the `cdrplatform-portal-client` App Registration. `enabled_pages` includes additional pages based on which services are deployed: - **ICAP Server**: add `IcapSettings,IcapRequests,IcapReporting` - **Storage Monitor**: add `SharePointMonitoring,OneDriveMonitoring,OutlookMonitoring` ```sh portal_domain="" portal_client_id="" tenant_id="" # Add pages based on deployed services: # + ICAP Server: IcapSettings,IcapRequests,IcapReporting # + Storage Monitor: SharePointMonitoring,OneDriveMonitoring,OutlookMonitoring enabled_pages="SystemSettings\,PolicySettings\,ValidationSettings" ``` ```sh helm upgrade cdrplatform-portal cdrplatform-portal \ --reuse-values \ --set ingress.tls.enabled=true \ --set ingress.tls.domain=${portal_domain} \ --set ingress.tls.secretName=tls-secret \ --set configuration.BackendUrl="https://${portal_domain}" \ --set configuration.EnabledPages=${enabled_pages} \ --set configuration.OIDC.ProviderOptions.Authority="https://login.microsoftonline.com/${tenant_id}/v2.0" \ --set configuration.OIDC.ProviderOptions.RedirectUri="https://${portal_domain}/authentication/login-callback" \ --set configuration.OIDC.ProviderOptions.ClientId="${portal_client_id}" \ --set configuration.OIDC.ProviderOptions.PostLogoutRedirectUri="https://${portal_domain}/authentication/logout-callback" \ --atomic ``` Next, the *portal access* service needs to enable SSO authentication using Microsoft Entra ID. Set `tenant_id`, `portal_domain` and `valid_audiences` variables below. **Note:** Set `valid_audiences` to the **Application (client) ID** of the `cdrplatform-portal-access` app registration (found in the Azure portal under **App registrations** > **Overview**). ```sh tenant_id="" portal_domain="" valid_audiences="" # Application (client) ID of the cdrplatform-portal-access app registration (copy from Azure portal) ``` ```sh helm upgrade cdrplatform-portal-access cdrplatform-portal-access \ --reuse-values \ --set ingress.tls.enabled=true \ --set ingress.tls.domain=${portal_domain} \ --set ingress.tls.secretName=tls-secret \ --set configuration.AuthenticationScheme=Bearer \ --set configuration.Authentication__Schemes__Bearer__ValidAudiences__0=${valid_audiences} \ --set configuration.Authentication__Schemes__Bearer__ValidIssuer=https://sts.windows.net/${tenant_id}/ \ --set configuration.Authentication__Schemes__Bearer__Authority=https://login.microsoftonline.com/${tenant_id}/v2.0/ \ --atomic ``` ## Azure Government (GovCloud) deployments If your tenant is in **Azure Government** rather than the commercial Azure cloud, the Microsoft Entra ID endpoints differ. Replace the commercial endpoints used in the commands above with their Azure Government equivalents: | Purpose | Commercial Azure | Azure Government | | --- | --- | --- | | Login / OIDC authority | `https://login.microsoftonline.com` | `https://login.microsoftonline.us` | | Microsoft Graph (used by `create-azure-app-registrations.sh`) | `https://graph.microsoft.com` | `https://graph.microsoft.us` | For example, the portal `Authority` becomes: ```sh --set configuration.OIDC.ProviderOptions.Authority="https://login.microsoftonline.us/${tenant_id}/v2.0" ``` and the access services' `Authority` becomes: ```sh --set configuration.Authentication__Schemes__Bearer__Authority=https://login.microsoftonline.us/${tenant_id}/v2.0/ ``` **Note:** Confirm the exact `issuer`, `authorization_endpoint` and `jwks_uri` for your tenant by retrieving the OIDC metadata document from your authority, for example `https://login.microsoftonline.us/${tenant_id}/v2.0/.well-known/openid-configuration`, and use those values in the `ValidIssuer` and related settings. When running `create-azure-app-registrations.sh` against an Azure Government tenant, sign in to the Government cloud first (`az cloud set --name AzureUSGovernment`) before running the script. ## User roles ### To assign roles to portal users 1. Navigate to the enterprise application titled `ar-cdrplatform-portal-access` within Microsoft Entra ID. **Note:** The screenshots use `ar-cdrplatform-perf-portal-access` as an example. 2. Navigate to the *Users and groups* section and select **Add user/group**. 3. Select the desired user or group, then choose the appropriate role. 4. Click **Assign**. --- ### To assign roles to API users - Navigate to the enterprise application titled `ar-cdrplatform-API-access` in Microsoft Entra ID. - From the *Users and groups* section, click **Add user/group**. - Select the desired user or group, then choose the appropriate role. - [Learn more via Microsoft application management documentation](https://learn.microsoft.com/en-us/azure/active-directory/manage-apps/what-is-single-sign-on) - Click **Assign**. Glasswall Halo uses [Microsoft Entra ID](https://learn.microsoft.com/en-us/entra/identity/) (formerly Azure AD) app registrations to enable Single Sign-On (SSO) for the Portal and bearer-token authentication for the API. The [provided shell script](/.attachments/create-azure-app-registrations.sh) automates the creation of these registrations using the Azure CLI. If you would rather create them by hand—for example, if you do not have access to the Azure CLI, or your organization requires changes to be made through the portal—this page walks through the equivalent steps in the [Microsoft Entra admin center](https://entra.microsoft.com/). The result is identical to running the script. ## Which registrations do I need? There are up to three app registrations. Which ones you create depends on what you are enabling: | App registration | Purpose | Required for | | --- | --- | --- | | `cdrplatform-api-access` | Protects the Halo API with bearer-token authentication | API authentication | | `cdrplatform-portal-access` | Protects the Portal's backend (*portal access*) service | Portal SSO | | `cdrplatform-portal-client` | The Portal single-page application that users sign in to | Portal SSO | - For **Portal SSO**, create `cdrplatform-portal-access` and `cdrplatform-portal-client` (this is the equivalent of running the script with `--skip-api`). - For **API authentication only**, create `cdrplatform-api-access` (the equivalent of `--skip-portal`). - For a **full deployment**, create all three. Create `cdrplatform-portal-access` before `cdrplatform-portal-client`, because the client registration references a scope that you expose on the access registration. ## Prerequisites - Access to a Microsoft Entra tenant, with an account that has at least the **Cloud Application Administrator** role. - The **domain name** you intend to use for the Halo Portal (for example, `halo.glasswall.com`). This is referred to below as ``. - Your **Directory (tenant) ID**. You can find it on the Microsoft Entra admin center **Overview** page. It is referred to below as ``. :::note The script optionally appends a *suffix* to the registration names and identifier URIs (for example, to host more than one environment in the same tenant). If you need a suffix, append it to every name and Application ID URI below—for example, `ar-halo-api-access-dev` and `api:///cdrplatform-api-access-dev`. ::: --- ## 1. `cdrplatform-api-access` This registration protects the Halo API. Skip this section if you are only enabling Portal SSO. ### Register the application 1. Sign in to the [Microsoft Entra admin center](https://entra.microsoft.com/). 2. In the left-hand menu, expand **Entra ID** and select **App registrations**, then click **New registration**. 3. Enter the **Name** `ar-halo-api-access`. 4. Under **Supported account types**, select **Single tenant only - ``**. 5. Leave the **Redirect URI** blank. 6. Click **Register**. ### Set the Application ID URI 1. In the registration, go to **Expose an API**. 2. Next to **Application ID URI**, click **Add** (or **Set**). 3. Replace the default value with: ``` api:///cdrplatform-api-access ``` 4. Click **Save**. ### Configure API permissions 1. Go to **API permissions**. 2. Ensure **Microsoft Graph > User.Read** (delegated) is present. It is added by default on new registrations; if it is missing, click **Add a permission > Microsoft Graph > Delegated permissions**, search for `User.Read`, select it and click **Add permissions**. ### Add app roles Add two app roles so that users and applications can be granted **Admin** or **User** access. 1. Go to **App roles** and click **Create app role**. 2. Create the **Admin** role with these values: - **Display name:** `Admin` - **Allowed member types:** **Both (Users/Groups + Applications)** - **Value:** `Admin` - **Description:** `Admins can manage anything on Halo` - **Do you want to enable this app role?** selected 3. Click **Apply**. 4. Click **Create app role** again and create the **User** role: - **Display name:** `User` - **Allowed member types:** **Both (Users/Groups + Applications)** - **Value:** `User` - **Description:** `Users can read policies and rebuild files` - **Do you want to enable this app role?** selected 5. Click **Apply**. :::note Registering an application in the portal automatically creates the matching **Enterprise application** (service principal) in your tenant, so there is no separate step for that. You assign users and groups to the **Admin** and **User** roles from the Enterprise application—see [Single Sign-On > User roles](/halo/glasswall-halo-sso#user-roles). ::: --- ## 2. `cdrplatform-portal-access` This registration protects the Portal's backend *portal access* service. Skip this section (and section 3) if you are only enabling API authentication. ### Register the application 1. In **App registrations**, click **New registration**. 2. Enter the **Name** `ar-halo-portal-access`. 3. Under **Supported account types**, select **Single tenant only - ``**. 4. Leave the **Redirect URI** blank. 5. Click **Register**. ### Set the Application ID URI 1. Go to **Expose an API**. 2. Next to **Application ID URI**, click **Add** (or **Set**) and enter: ``` api:///cdrplatform-portal-access ``` 3. Click **Save**. ### Expose the `PortalUserScope` scope The Portal client registration (created next) needs a delegated scope to call this service on behalf of the signed-in user. 1. Still on **Expose an API**, click **Add a scope**. 2. Enter the following: - **Scope name:** `PortalUserScope` - **Who can consent?** **Admins and users** - **Admin consent display name:** `PortalUserScope` - **Admin consent description:** `Allow the application to access Halo on behalf of the signed-in user.` - **State:** **Enabled** 3. Click **Add scope**. ### Configure API permissions Ensure **Microsoft Graph > User.Read** (delegated) is present, as in the API access registration above. ### Add app roles Add the same **Admin** and **User** app roles described in [section 1](#add-app-roles), using the identical display names, values, descriptions and **Both** member types. --- ## 3. `cdrplatform-portal-client` This is the single-page application (SPA) that users sign in to. Create it after `cdrplatform-portal-access`. ### Register the application 1. In **App registrations**, click **New registration**. 2. Enter the **Name** `ar-halo-portal-client`. 3. Under **Supported account types**, select **Single tenant only - ``**. 4. Leave the **Redirect URI** blank for now (it is configured below). 5. Click **Register**. ### Set the Application ID URI 1. Go to **Expose an API**. 2. Next to **Application ID URI**, click **Add** (or **Set**) and enter: ``` api:///cdrplatform-portal-client ``` 3. Click **Save**. ### Configure the redirect URI and logout URL The Portal is a single-page application (SPA), so its redirect URI must be registered under the **Single-page application** platform. Replace `` with your Portal domain (for example, `halo.glasswall.com`). 1. Go to **Authentication**. 2. On the **Redirect URI configuration** tab, click **Add Redirect URI**. 3. On the **Select a platform to add redirect URI** pane, select **Single-page application** and enter the redirect URI: ``` https:///authentication/login-callback ``` Click **Configure**. 4. Open the **Settings** tab and set the **Front-channel logout URL** to: ``` https:///authentication/logout ``` Click **Save**. ### Configure API permissions The Portal client needs to sign users in and call the *portal access* service on their behalf. 1. Go to **API permissions**. 2. Ensure **Microsoft Graph > User.Read** (delegated) is present. 3. Click **Add a permission > APIs my organization uses**, then search for and select **ar-halo-portal-access**. 4. Choose **Delegated permissions**, select **PortalUserScope** and click **Add permissions**. ### Grant admin consent 1. Still on the **API permissions** page, click **Grant admin consent for ``** and confirm. 2. Confirm that the **Status** column shows a green checkmark for each permission. :::note Admin consent for the Portal client (`ar-halo-portal-client`) is required for users to sign in without being prompted to consent individually. ::: --- ## Record the values for deployment Once the registrations exist, make a note of the following. These are used when you configure SSO and API authentication in the Halo Helm charts (see [Single Sign-On](/halo/glasswall-halo-sso#configuration)). - **Directory (tenant) ID**—`` - **`cdrplatform-portal-client` Application (client) ID**—used as `portal_client_id` - **`cdrplatform-portal-access` Application (client) ID**—used as the `valid_audiences` value for the *portal access* service - **`cdrplatform-api-access` Application (client) ID**—used as the `valid_audiences` value for the API - The Application ID URIs you set: - `api:///cdrplatform-api-access` - `api:///cdrplatform-portal-access` - `api:///cdrplatform-portal-client` You can find each **Application (client) ID** on the **Overview** page of the corresponding app registration. --- Return to Single Sign-On Get help ## Overview This guide provides step-by-step instructions for integrating Keycloak Single Sign-On (SSO) with the Glasswall Halo platform in an offline (air-gapped), on-premises environment. It assumes you've already deployed the Single Node Halo platform (e.g., on VMs via Proxmox or ESXi) and now want to enable OIDC-based SSO using Keycloak. You'll learn how to: - Set up a Keycloak server with HTTPS in an offline environment (using a preloaded container image or manual VM installation). - Import and customize the provided `glasswall-realm.json` Keycloak configuration. - Configure TLS certificates and DNS/host resolution for both Keycloak and Halo. - Deploy the Halo portal and related services: - `cdrplatform-portal` - `cdrplatform-portalaccess` - `cdrplatform-api-access` ## Glasswall Halo's Keycloak SSO Integration Guide * [Prerequisites](/halo/keycloak-sso-integration#prerequisites) * [Step 1 - Configuring keycloak for Glasswall Halo SSO](/halo/keycloak-sso-integration#step-1---configuring-keycloak-for-glasswall-halo-sso) * [Step 2 - TLS certificates and DNS considerations](/halo/keycloak-sso-integration#step-2---tls-certificates-and-dns-considerations) * [Step 3 - Deploying Halo services with Keycloak SSO (Helm configurations)](/halo/keycloak-sso-integration#step-3---deploying-halo-services-with-keycloak-sso-helm-configurations) * [Step 4 - Validation and testing SSO integration](/halo/keycloak-sso-integration#step-4---validation-and-testing-sso-integration) * [Step 5 - Troubleshooting](/halo/keycloak-sso-integration#step-5---troubleshooting) ## Prerequisites ### Environment requirements - This setup is intended for air-gapped or strictly on-premises environments. - The Single Node Halo cluster must already be deployed. - Internal DNS resolution must be in place, either via a local DNS server or manually through `/etc/hosts` entries. - Keycloak must be installed locally on a host or virtual machine. → [Keycloak documentation & installation](https://www.keycloak.org/downloads) ### Required resources - A pre-configured realm file (`glasswall-realm.json`), provided by [Glasswall Support](mailto:support@glasswall.com). - Local access to Halo Helm charts for deployment within the Single Node Halo cluster. ### DNS/hosts configuration Ensure the hostname for your Keycloak server is resolvable by both client machines and Halo platform components. - **With internal DNS**: create an A record for `keycloak.dev.local` pointing to the Keycloak server’s IP address. - **Without internal DNS**: add the following entry to the `hosts` file on each machine that needs access: ## Step 1 - Configuring Keycloak for Glasswall Halo SSO Glasswall provides a Keycloak realm configuration (as a JSON file) that defines the realm, clients, roles, and scopes needed for Halo's SSO. You can import this configuration to avoid manual setup. In an offline setup, do this via the Keycloak Admin UI. ### 1.1 - Importing the Glasswall realm configuration A. **Obtain the realm export**: you should have a file named `glasswall-realm.json`. This file contains the realm settings for Halo's SSO. **Glasswall realm summary** The Glasswall realm includes the following pre-configured items: - **Realm ID**: Glasswall - **Clients** - Halo-portal-client - Halo-API-access - Halo-portal-access - **Key roles** - Realm-level: admin, user - Client-level: Halo client admin and user roles. - Client scopes (pre-configured with mappers): - Portaluserscope: necessary claims (roles, email) to tokens. - **Token lifespan** - SSO session idle timeout: 30 mins - **Authentication**: standard username/password. B. **Log into Keycloak Admin** - Using a web browser, navigate to the Keycloak Admin Console at `https://:8443/` and log in with the master admin account. C. **Import the realm**. - In the admin console select the dropdown that currently shows "Master" and click **Create realm**. - From the *Create realm* screen, select **Import**, and upload the `glasswall-realm.json` file. - Keycloak will parse the file and populate the realm name (e.g.,`glasswall`). - Confirm the import. D. **Verify realm settings** - After import, review key settings and ensure: - Realm name is`glasswall`. - Realm is enabled. - SSL required: likely set to`external`. - Clients should include: - `halo-portal-client` - `halo-portal-access` - `halo-api-access` ### 1.2 Customizing client settings (hostnames and redirect URIs) Next, for each of the Halo clients, update the configuration to match your environment. #### Halo-portal-client (portal frontend) **Valid redirect URIs**: replace the placeholder domain in the realm JSON with your actual portal domain. The sample configuration uses a test domain (for example,`https://pvetest.cdr.glasswall.dev/authentication/login-callback`). Update this value to match the URL users will use to access your Halo portal. For instance, if your Halo portal is available at `https://halo.dev.local/` and the application’s login callback path is `/authentication/login-callback`, set: - **Valid redirect URIs**:`https://halo.dev.local/authentication/*` - **Web origins**:`https://halo.dev.local` - **Root URL** (optional):`https://halo.dev.local` - **Valid post logout redirect URIs**:`https://halo.dev.local/authentication/logout-callback` #### Halo-portal-access (portal backend) - **Valid redirect URIs**: (e.g.,`https://pvetest.cdr.glasswall.dev/*`) - Update it to your portal domain. - **Web origins**:`https://halo.dev.local` #### Halo-API-access - Leave redirect settings as-is, and save each client’s settings. ### 1.3 Creating users and assigning roles Now you'll create users and assign Halo roles. A. In the `glasswall` realm, navigate to **Users** → **Add user**. B. Fill in the Username and details. C. In **Credentials** section, set a password and uncheck "Temporary". D. In **Role mappings**, assign: - Realm roles: - `Admin` - `User` - Client roles: - `halo-portal-client`[Admin, user] - `halo-api-access`[Admin, user] - `halo-portal-access`[Admin, user] An Admin user sees: And can manage: Including: --- ## Step 2 - TLS certificates and DNS considerations Ensure proper TLS trust and DNS for all components. ### 2.1 TLS trust Halo platform services (`Portal`, `Portal-Access`, `API-Access`) must trust Keycloak’s certificate. During the OIDC handshake these services call Keycloak over HTTPS to fetch the discovery document and signing keys. If they cannot validate the Keycloak certificate chain, the handshake fails and users receive `401` errors at sign-in even though Keycloak itself is healthy. This is the most common cause of SSO failures in air-gapped deployments, where Keycloak typically uses a **self-signed certificate** or a certificate issued by a **private/internal Certificate Authority (CA)** that is not in any public trust store. :::warning Self-signed and private-CA certificates The Halo services validate the full certificate chain. A self-signed leaf certificate, or one issued by an internal CA, will be rejected unless the issuing CA certificate is explicitly trusted. The recommended approach is to issue the Keycloak certificate from an **internal CA** and make that **CA certificate** trusted by the Halo services, rather than trusting individual leaf certificates. ::: **Make the Keycloak CA trusted by the Halo services** 1. Obtain the CA certificate that signed Keycloak's TLS certificate in PEM format (for a self-signed setup this is the Keycloak certificate itself). 2. Provide that CA certificate to the Halo services so it is mounted into the trusted CA bundle the services use. The supported mechanism depends on your chart version—contact [Glasswall Support](https://www.glasswall.com/support) to confirm the exact value for your release before deploying, and to obtain the CA-trust values if they are not exposed in your chart. 3. Restart (or `helm upgrade`) the `cdrplatform-portal`, `cdrplatform-portal-access` and `cdrplatform-api-access` deployments so they pick up the updated trust bundle. 4. Validate the trust chain **before** testing sign-in. The Halo service containers are minimal (distroless) images with no shell, so run this check from an admin workstation or jump host that can reach Keycloak—not from inside a service pod—using the same CA certificate you mounted into the services: ```bash # Run from a host that can reach Keycloak (not from inside a service pod). # Replace with your Keycloak host and the path to the CA certificate. openssl s_client -connect "${keycloak_domain}:8443" -CAfile /path/to/keycloak-ca.crt -servername "${keycloak_domain}" `API` and create a new Authorization server. - Give an appropriate name. e.g. `Glasswall Halo` - Enter a common audience value that will be shared across all Halo clients (Portal and API). e.g. `api://halo` - Give an appropriate description. e.g. `Glasswall Halo Authorization server` - Once saved, make a note of the `Issuer Metadata URI`. Remove `/.well-known/oauth-authorization-server` from the end of the URI. e.g. ```sh export ORIGINAL_ISSUER_URI=https://your-org.okta.com/oauth2/aus1234567890abcdef/.well-known/oauth-authorization-server export OKTA_ISSUER_URI=https://your-org.okta.com/oauth2/aus1234567890abcdef export VALID_AUDIENCE="api://halo" ``` > **Note:** The audience `api://halo` is shared across all Halo clients. Both Portal SSO and API Bearer authentication validate tokens against this audience. This allows a single authorization server to serve both use cases. ## Add custom scope for Halo API access 2. Navigate to the `Scopes` tab on the authorization server and add a new scope: - Name: `halo.api` - Display Phrase: Halo API - Description: `Access to Halo API` - Check `Set as a default scope` > **Note:** The OIDC scopes (e.g. `openid`, `profile`, `email`) are user-context only and cannot be used with the Client Credentials flow. A custom scope is required for machine-to-machine API access. ## Set up claims 3. Configure the claims needed by Halo services. Navigate to the `Claims` tab on the authorization server. - Full name: - Name: `name` - Include in token type: `Access Token` - Value type: `Expression` - Value: `user.firstName + " " + user.lastName` - Include in: `Any scope` - Family name: - Name: `family_name` - Include in token type: `Access Token` - Value type: `Expression` - Value: `user.lastName` - Include in: `Any scope` - Given name: - Name: `given_name` - Include in token type: `Access Token` - Value type: `Expression` - Value: `user.firstName` - Include in: `Any scope` - Email ID: - Name: `unique_name` - Include in token type: `Access Token` - Value type: `Expression` - Value: `user.email` - Include in: `Any scope` ## Define roles > **Note:** Halo uses two separate role claims: > - `roles`: used by the Portal (via Portal-Access) to control UI access > - `role`: used by the API-Access service for endpoint authorization > > If the role claims are not configured, the Portal defaults to a read-only role and the API defaults to the `User` role. See [Portal Roles to Action Mapping](/halo/portal-roles-to-action-mapping) and [API Roles to Action Mapping](/halo/api-roles-to-action-mapping) for details on what each role can access. 4. Halo recognizes two role values: `User` and `Admin` (case-insensitive). Okta doesn't have a native way of defining roles in Authorization servers. Role claims need to be added to pass the user/client roles in the access token. ### Portal SSO roles (`roles` claim) This claim is used by the Portal to determine which pages and features are available to the user. We recommend using Halo-specific group names rather than generic `User` or `Admin` groups, which may conflict with other applications in your Okta organization. 1. Create the groups in Okta. Navigate to `Directory` -> `Groups` and create: - `Halo_User` - `Halo_Admin` 2. Assign users to the appropriate group. 3. Navigate to the authorization server's `Claims` tab and create a `roles` claim that translates group names to Halo role values: - Name: `roles` - Include in token type: `Access Token` - Value type: `Expression` - Value: ``` isMemberOfGroupName("Halo_Admin") ? "Admin" : isMemberOfGroupName("Halo_User") ? "User" : "" ``` - Include in: `Any scope` > **Note:** The `isMemberOfGroupName()` function only works in user-context flows (e.g., Authorization Code). It does not apply to Client Credentials (machine-to-machine) flows. ### API roles (`role` claim) This claim is used by the API-Access service to authorize operations. Since API clients use the Client Credentials flow (no user context), group membership expressions don't apply. Instead, map the `client_id` to a role value. 4. Navigate to the authorization server's `Claims` tab and create a `role` claim: - Name: `role` - Include in token type: `Access Token` - Value type: `Expression` - Value: ``` (app.clientId == "") ? "Admin" : "User" ``` - Include in: `Any scope` > **Note:** Replace `` with the `client_id` of your API Services application. Add additional conditions for multiple API clients, e.g.: > ``` > (app.clientId == "") ? "Admin" : (app.clientId == "") ? "Admin" : "User" > ``` ## Next steps With the authorization server configured, proceed to set up one or both of: - [Halo Portal SSO](/halo/okta-portal-sso) - configure Okta as the identity provider for the Halo Portal - [Halo API Bearer Authentication](/halo/okta-api-bearer-authentication) - configure Bearer token authentication for the Halo API Each guide includes configuring an access policy on this authorization server for its respective client. Use Okta as Identity Provider to login to Halo portal using SSO. ## Prerequisites - Okta authorization server configured for Halo (see [Okta Authorization Server](/halo/okta-authorization-server)) ## Register your app in Okta ### Create app integration 1. Login to Okta Admin Console → Navigate to `Applications` -> `Applications`. 2. Click Create App Integration → Choose OIDC - OpenID Connect. 3. Click Single-Page Application. ### Configure app integration 4. Configure the following settings: - App integration name: e.g. `Glasswall Halo Portal` - Grant type: Choose below 3 options - Authorization Code - Refresh Token - Implicit (hybrid) - Sign-in redirect URIs: e.g. `https:///authentication/login-callback` - Sign-out redirect URIs: e.g. `https:///authentication/logout-callback` - Controlled access: Choose `Allow everyone in your organization to access`. Or select only specific groups if you want to limit the access to few people. - Unselect `Enable immediate access with Federation Broker Mode`. ### Save application and note client ID 5. Save the Application and note the `Client ID`. ```sh export PORTAL_CLIENT_ID="" ``` ### Grant API scopes 6. Navigate to `Okta API Scopes` tab in the application and grant below scopes. - `okta.myAccount.email.read` - `okta.myAccount.profile.read` - `okta.users.read` - `okta.users.read.self` ### Note issuer URI and audience 7. From the authorization server configured in the [prerequisite step](/halo/okta-authorization-server), note the `Issuer Metadata URI` and `VALID_AUDIENCE`: ```sh export OKTA_ISSUER_URI="https:///oauth2/" export OKTA_ORIGIN="https://" export VALID_AUDIENCE="api://halo" ``` ### Add access policy for portal 8. Navigate to the authorization server's `Access Policies` tab (`Security` -> `API` -> select your authorization server -> `Access Policies`). 9. Add a new access policy: - Name: e.g. `Portal SSO Access` - Description: e.g. `Access policy for Halo Portal SSO users` - Assign to: the Portal SPA client created above (search by name `Glasswall Halo Portal`) 10. Add a rule: - Name: e.g. `Allow Portal Users` - Grant type: `Authorization Code` - User is a member of: `Everyone` (or restrict to a specific group, e.g., `Halo-Admin`, if you want to limit Portal access) - Leave other settings as defaults or adjust as needed. ## Update Portal service ```sh export HALO_DOMAIN= helm upgrade --install cdrplatform-portal cdrplatform-portal -n cdrplatform --reuse-values \ --set configuration.AutoAdmin=false \ --set configuration.BackendScope="email openid profile" \ --set configuration.BackendUrl="https://${HALO_DOMAIN:?}" \ --set configuration.EnabledPages="SystemSettings\,PolicySettings\,IcapSettings\,IcapRequests\,IcapReporting" \ --set configuration.OIDC.ProviderOptions.Authority="${OKTA_ISSUER_URI:?}" \ --set configuration.OIDC.ProviderOptions.ClientId="${PORTAL_CLIENT_ID:?}" \ --set configuration.OIDC.ProviderOptions.PostLogoutRedirectUri="https://${HALO_DOMAIN:?}/authentication/logout-callback" \ --set configuration.OIDC.ProviderOptions.RedirectUri="https://${HALO_DOMAIN:?}/authentication/login-callback" \ --set appenvironment.HTTP_CSP_CONNECT_SRC="'self' ${OKTA_ORIGIN:?}" \ --set appenvironment.HTTP_CSP_FRAME_SRC="'self' ${OKTA_ORIGIN:?}" \ --set appenvironment.HTTP_CSP_FRAME_ANCESTORS="'self' ${OKTA_ORIGIN:?}" \ --set ingress.enabled=true \ --set ingress.tls.domain="${HALO_DOMAIN:?}" \ --set ingress.tls.enabled=true \ --atomic ``` ## Update Portal-Access service ```sh helm upgrade --install cdrplatform-portal-access -n cdrplatform cdrplatform-portal-access --reuse-values \ --set configuration.Authentication__Schemes__Bearer__Authority="${OKTA_ISSUER_URI:?}" \ --set configuration.Authentication__Schemes__Bearer__ValidAudiences__0="${VALID_AUDIENCE:?}" \ --set configuration.Authentication__Schemes__Bearer__ValidIssuer="${OKTA_ISSUER_URI:?}" \ --set configuration.AuthenticationScheme="Bearer" \ --set configuration.CORSDOMAIN="'*'" \ --set ingress.enabled=true \ --set ingress.tls.domain="${HALO_DOMAIN:?}" \ --set ingress.tls.enabled=true \ --atomic ``` ## Provide access to users There are 2 roles in Halo - `User` and `Admin`. Roles are configured via group membership on the Okta authorization server (see [Define Roles](/halo/okta-authorization-server#define-roles)). - To provide user role, add the user to the `Halo_User` group. - To provide admin role, add the user to the `Halo_Admin` group. Authenticate programmatic access to the Halo API using OAuth 2.0 Bearer tokens issued by Okta. ## Overview Halo exposes a REST API through the API-Access service. All API requests must include a valid Bearer token in the `Authorization` header. Tokens are obtained from Okta using the OAuth 2.0 Client Credentials flow (machine-to-machine). This guide walks through configuring Okta as the Identity Provider for the Halo API-Access service, then testing the integration with Bearer token authentication. ## Prerequisites - A running Halo deployment with API-Access service configured - An Okta organization with admin access - Okta authorization server configured for Halo (see [Okta Authorization Server](/halo/okta-authorization-server)) - Network access from the API client to both Okta and the Halo API endpoint --- ## Setup ### Register an API client in Okta #### Create a machine-to-machine application 1. Login to the Okta Admin Console. 2. Navigate to `Applications` -> `Applications` -> `Create App Integration` -> `API Services`. 3. Provide an appropriate App integration name: e.g. `Glasswall Halo API` #### Disable DPoP (proof of possession) 4. Navigate to the application's `General` tab -> `General Settings` section -> click `Edit`. 5. Under **Proof of possession**, untick `Require Demonstrating Proof of Possession (DPoP) header in token requests`. 6. Save. > **Note:** Okta enables DPoP by default for API Services applications. DPoP binds tokens to the client's cryptographic key, preventing token replay attacks. However, the Halo API-Access service does not currently support DPoP token validation, so this must be disabled for Bearer token authentication to work. #### Configure client credentials 7. Note the `Client ID` and generate a `Client Secret`. ```sh export CLIENT_ID="" export CLIENT_SECRET="" ``` > **Security Note:** Store client secrets securely using a secrets manager (e.g., Azure KeyVault). Never commit secrets to source control. #### Configure roles (optional) 8. If your API client needs write access to license management endpoints, ensure the `role` claim is configured on the authorization server with the `Admin` value mapped to your client (see [API Roles](/halo/okta-authorization-server#api-roles-role-claim)). > **Note:** Without the `role` claim, the API defaults to the `User` role. See [API Roles to Action Mapping](/halo/api-roles-to-action-mapping) for details on what each role can access. #### Note issuer URI and audience 9. From the authorization server configured in the [prerequisite step](/halo/okta-authorization-server): ```sh export OKTA_ISSUER_URI="https:///oauth2/" export VALID_AUDIENCE="api://halo" ``` #### Add access policy for API client 10. Navigate to the authorization server's `Access Policies` tab (`Security` -> `API` -> select your authorization server -> `Access Policies`). 11. Add a new access policy: - Name: e.g. `API Client Access` - Description: e.g. `Access policy for Halo API clients` - Assign to: the API Services client created above (search by application name `glasswall-halo-api` from step 2) 12. Add a rule: - Name: e.g. `Allow API Clients` - Grant type: `Client Credentials` - Leave other settings as defaults or adjust token lifetime as needed. > **Note:** Access policies for the Client Credentials grant control which **clients** (by `client_id`) can request tokens. Group membership does not apply because there is no user context in this flow. Secure access to the API by restricting who has the `client_id` and `client_secret` via your secrets manager. ### Configure API-Access service for Okta The `cdrplatform-api-access` Helm chart must be configured to validate Bearer tokens issued by your Okta authorization server. The chart exposes authentication settings under the `configuration` values key. #### Deploy the configuration > **Note:** The `Authority` and `ValidIssuer` must both be set to the Okta issuer URI. The API-Access service uses `Authority` to discover the JWKS (JSON Web Key Set) for token signature validation, and `ValidIssuer` to verify the `iss` claim in the token. ```sh export HALO_DOMAIN= helm upgrade --install cdrplatform-api-access cdrplatform-api-access --reuse-values \ --set configuration.AuthenticationScheme="Bearer" \ --set configuration.Authentication__Schemes__Bearer__Authority="${OKTA_ISSUER_URI:?}" \ --set configuration.Authentication__Schemes__Bearer__ValidIssuer="${OKTA_ISSUER_URI:?}" \ --set configuration.Authentication__Schemes__Bearer__ValidAudiences__0="${VALID_AUDIENCE:?}" \ --set strategy.type="Recreate" \ --set ingress.enabled=true \ --set ingress.tls.enabled=true \ --set ingress.tls.domain="${HALO_DOMAIN:?}" \ --atomic ``` #### Multiple audiences (optional) If multiple API clients or applications need to access the Halo API with different audiences, add additional `ValidAudiences` entries: ```sh --set configuration.Authentication__Schemes__Bearer__ValidAudiences__0="api://halo-api" \ --set configuration.Authentication__Schemes__Bearer__ValidAudiences__1="" \ --set configuration.Authentication__Schemes__Bearer__ValidAudiences__2="" ``` --- ## Halo API usage ### Obtain a bearer token #### Client Credentials flow (machine-to-machine) Use this flow for automated systems, CI/CD pipelines, and service-to-service integrations where no user interaction is required. ```sh export TOKEN_ENDPOINT="${OKTA_ISSUER_URI}/v1/token" ``` Request a token: ```sh curl -s -X POST "${TOKEN_ENDPOINT}" \ -H "Content-Type: application/x-www-form-urlencoded" \ -d "grant_type=client_credentials" \ -d "client_id=${CLIENT_ID}" \ -d "client_secret=${CLIENT_SECRET}" \ -d "scope=halo.api" ``` Example response: ```json { "access_token": "eyJhbGciOiJSUzI1NiIsInR5cCI6Ikp...", "token_type": "Bearer", "expires_in": 3600, "scope": "halo.api" } ``` Extract the token: ```sh export ACCESS_TOKEN=$(curl -s -X POST "${TOKEN_ENDPOINT}" \ -H "Content-Type: application/x-www-form-urlencoded" \ -d "grant_type=client_credentials" \ -d "client_id=${CLIENT_ID}" \ -d "client_secret=${CLIENT_SECRET}" \ -d "scope=halo.api" | jq -r '.access_token') ``` ### Authenticate API requests Include the Bearer token in the `Authorization` header of every API request: ```sh export HALO_API_URL="https://" ``` #### Verify authentication Test with a protected endpoint. A request without a token should return `401`, and a request with a valid token should return `200`: ```sh # Without token - expect 401 curl -s -o /dev/null -w "%{http_code}" \ "${HALO_API_URL}/api/v1/policies" # With token - expect 200 curl -s -o /dev/null -w "%{http_code}" \ -H "Authorization: Bearer ${ACCESS_TOKEN}" \ "${HALO_API_URL}/api/v1/policies" ``` #### Example: Submit a file for processing ```sh curl -s -X POST \ -H "Authorization: Bearer ${ACCESS_TOKEN}" \ -F "file=@/path/to/document.pdf" \ "${HALO_API_URL}/api/v1/rebuild/file" \ --output rebuilt_document.pdf ``` ## Troubleshooting | Symptom | Cause | Resolution | |---|---|---| | `401 Unauthorized` | Token expired or invalid | Request a new token. Verify `CLIENT_ID` and `CLIENT_SECRET` are correct. | | `401 Unauthorized` with valid token | Audience mismatch | Ensure `ValidAudiences` in API-Access matches the `aud` claim in the token. | | `403 Forbidden` with `AdminRoleMissing` | Missing `Admin` role in token | Configure the `role` claim on the authorization server (see [API Roles](/halo/okta-authorization-server#api-roles-role-claim)). Required for license management write operations. | | Token request fails | Incorrect token endpoint URL | Verify the authorization server ID and Okta domain. Check `/.well-known/openid-configuration`. | | `invalid_scope` error | OIDC scopes used with Client Credentials | Use the custom `halo.api` scope instead of `openid`, `profile`, or `email`. | | `invalid_dpop_proof` error | DPoP not disabled | Disable DPoP on the API Services application (see [Disable DPoP](#disable-dpop-proof-of-possession)). | | Network timeout | Firewall or proxy blocking | Ensure the API client can reach both the Okta token endpoint and the Halo API endpoint. | The Halo portal is your centralized hub for deploying and managing Glasswall’s next-generation Content Disarm and Reconstruction (CDR) technology. Through this enterprise portal you can: * Dive into in-depth reporting to monitor file processing, threats detected, and system usage trends. * Customize protection policies at a granular level, aligning file safety with your organization’s risk appetite. * Seamlessly access key features such as file cleaning, ICAP integration, XML validation, and API configuration. In essence, the Halo portal empowers you to control, monitor, and tailor Glasswall’s sophisticated file defense capabilities from one unified interface. ## Minimum requirements All you need to get started is a good internet connection, and an appropriate up to date browser on your device. ## Supported browsers The portal can be accessed and used directly via the following popular web browsers: * Edge * Chrome * Firefox ## Accessing Halo Portal To access the Halo portal, simply enter **[halo.glasswall.com](https://halo.glasswall.com/)** in your browser’s address bar and wait for the page to load. Once the portal launches, you are ready to explore and make use of Glasswall’s advanced Content Disarm and Reconstruction (CDR) technology. Access levels within the Halo portal vary by user type: * **Admin users** have full access to all portal features, including reporting, file cleaning, license management, protection settings, and API configuration. * **Standard users** have access to core features such as file cleaning, API information, and view-only access to reporting and protection settings. ## Getting started #### Admin users * [Reporting](/halo/glasswall-portal-risk-report) * [Risk report](/halo/glasswall-portal-risk-report) * [Processed files](/halo/glasswall-portal-processed-files) * [ICAP report](/halo/glasswall-portal-icap-report) * [ICAP requests](/halo/glasswall-portal-icap-requests) * [Clean a file](/halo/glasswall-portal-clean-a-file) * [Glasswall APIs](/halo/glasswall-portal-apis) * [Protection settings](/halo/glasswall-portal-protection-settings) * [License management](/halo/glasswall-portal-license-management) * [Help](/halo/glasswall-portal-technical-help) #### Standard users * [Reporting](/halo/glasswall-portal-risk-report-standard) * [Risk report](/halo/glasswall-portal-risk-report-standard) * [Processed files](/halo/glasswall-portal-processed-files-standard) * [ICAP report](/halo/glasswall-portal-icap-report-standard) * [Clean a file](/halo/glasswall-portal-clean-a-file-standard) * [Glasswall APIs](/halo/glasswall-portal-apis-standard) * [Protection settings](/halo/glasswall-portal-protection-settings-standard) * [Help](/halo/glasswall-portal-technical-help-standard) In the Halo portal, the *Reporting* section provides users with both a high-level overview and in-depth breakdown of file data processed and protected by your organization. ## Risk report ### Protected file overview This section provides a high level overview of the total files processed by your organisation including: * Number of total files protected * Total size of protected files * Total number of risky items removed * Average time to protect a file ### Risk summary The *Risk summary* section shows a more thorough breakdown of your organization's processed files data including: #### Original file risk level After a file has been rebuilt, Glasswall Halo assigns a risk level to the original file. #### File protection level File protection level represents a file’s status after it’s been rebuilt. #### Original malware status Glasswall has partnered with ReversingLabs to uncover details on how many of your original files contained malware. ReversingLabs runs the hash for each file through its file reputation database and classifies how malicious it’s likely to be. #### Severity of malicious files Original files that have been confirmed as malicious by ReversingLabs each receive a threat severity score of 1-5. #### Risky content removed This is a list of risky content that has been removed, based on your organization’s policy settings. This list only includes risky content removed from protected files. This is a list of risky content that has been removed, based on your organization’s policy settings. #### Risky content allowed by your policies This is a list of risky content that has not been removed from your processed files. * Your organization has chosen to allow this content through your policy settings. #### Top protected file types This is a list of file types that have been protected by Glasswall CDR. #### Use of Glasswall's products and APIs This is a breakdown of the Glasswall products that were used to rebuild your organization’s files. ### Risk trends In the *Risk trends* section, you can see trend charts for the following: **File protection level** This is the file count by protection level over time. A file protection level represents a file’s status after it’s been rebuilt. **Original file risk level** This is the file count by original file risk level over time. A file's original risk level represents a file's risk before it has been rebuilt. Within the reporting section of the Halo portal, the *Processed files* page provides a comprehensive overview of files that have been analyzed by Glasswall Halo within the last 30 days. Users can review key information for each file, including its name and format, the date and time it was processed, the product or service that performed the analysis, a unique transaction id for reference, and detailed results showing the processing outcome and overall protection status. In the reporting section of the Halo portal, users can access detailed insights into requests processed through the Internet Content Adaptation Protocol (ICAP). This feature showcases how Glasswall CDR rapidly analyzes and sanitizes files to remove potential threats, ensuring secure file access across different use cases and environments. The ICAP reporting view allows users to monitor the number of files that were blocked and examine a detailed breakdown of request activity by request type, request status, media type, and blocked media type. In addition, the dashboard displays visual graphs highlighting request trends over time, organized by request status and request type, helping users identify patterns, assess system performance, and track overall file protection activity. Within the reporting section of the Halo portal, the processed files page provides a detailed record of all ICAP requests handled by Glasswall Halo within the last 30 days. Each entry includes key information such as the request ID, request type, and request URL, along with the specific profile that was applied during processing. Users can also view the request’s current status and access the full analysis report to review how the file was evaluated and protected. To make it easier to focus on specific data, the page includes filtering options that allow users to refine results by request type and request status, helping them quickly identify trends, investigate specific requests, or monitor system activity in real time. The Halo portal gives users the opportunity to explore and experience Glasswall’s advanced Content Disarm and reconstruction (CDR) technology first-hand. Through the *Clean a file* tool, users can easily upload and sanitize files using an intuitive, user-friendly interface. This demo tool provides a fast and reliable way to remove hidden threats that traditional antivirus solutions often miss, ensuring files are safe to open and share. During the file sanitization process, Glasswall Halo deconstructs each file into its individual components and examines its underlying structure; often referred to as its *Digital DNA*. Any elements that deviate from the file’s original manufacturer specifications or pose potential security risks are removed. The file is then meticulously rebuilt to its intended format, producing a clean, secure, and visually identical version that preserves usability while eliminating malicious content. #### What would you like to do? * [Clean a file](glasswall-portal-clean-a-file#clean-a-file) * [View details](/halo/glasswall-portal-clean-a-file#viewing-details) * [Download a cleaned file](/halo/glasswall-portal-clean-a-file#downloading-a-cleaned-file) * [Sanitize multiple files](glasswall-portal-clean-a-file#clean-and-download-multiple-files) * [Sanitize archive files](glasswall-portal-clean-a-file#clean-archive-files) --- ## Clean a file Glasswall Halo's file cleaning application allows you to input files in two ways, as explained below. ### Drag and drop The quicker and more intuitive way of using this application simply involves dragging and dropping your required file into the drop zone area. * The file is then instantly processed, rebuilt, and made available for download. * A detailed report of the file changes is also created for your reference. ### Uploading a file Users can also input files via a simple upload process. * Click **Browse** from the drop zone area and select a file from your device. * Halo will then process the file and rebuild it to its 'known good' standards. ## Viewing details After your file has been processed and rebuilt by Halo, a summary of the changes made is compiled for your reference. To find out how Glasswall made your file safe, click **View details**. This will display the *Original file risk level* along with a description of what it means. ### File analysis The system breaks the file down into its core components, examining its structure, content, and potential threats. Once processing is complete, the* File analysis* view provides a detailed technical breakdown of what was discovered and the actions taken to secure the file. This includes information on any content that was removed during sanitization, the file’s reputation as assessed by *ReversingLabs*, a comprehensive risk analysis highlighting potential vulnerabilities, and details of any structural repairs applied to restore the file to a safe, functional state. By presenting this level of detail, users gain full visibility into the protection process, allowing them to understand exactly how the file was cleaned and reinforced while maintaining its original usability. ## Downloading a cleaned file Once Halo has processed your file, the rebuilt clean file will be made available to you for download. Click **Download clean file** and save it to your device. Alternatively, if you processed an entire folder of files at once, you could download the sanitized files together by clicking **Download all clean files**. ## Clean and download multiple files Users can sanitize not just individual files but also multiple files or entire folders in a single, streamlined process. There are two ways to do this: either select the files you want to process using the upload option or drag and drop an entire folder into the drop zone. **Note:** you can process up to 100 files at a time. Once processing is complete, all sanitized files can be downloaded together by clicking download all clean files. For a deeper understanding of what was done to each file, you can view detailed analysis information for individual files by selecting view details. ## Clean archive files Archive files are quite often used as a quicker and easier way to transfer multiple files. For this reason, we have enabled Halo to process archive files and folders to ensure enterprise users can quickly check if a large number of incoming or outgoing files are a threat. You can process archive files in the same way as a single file, by either uploading the archive folder or dragging and dropping the folder into the drop zone. *Clean a file* can process an archive that contains folders and files inside, but not other archives. Once the files have been cleaned, you can download each file individually or altogether by clicking **Download all clean files**. In the *APIs* section of the Halo portal, users can explore the full range of Glasswall’s API offerings and gain a clear understanding of their capabilities. this section provides guidance on how to access the APIs, get started with integration, and effectively deploy Glasswall’s intelligent file protection technology within your existing infrastructure. Our APIs are designed to deliver Zero Trust Content Disarm and Reconstruction (CDR) capabilities wherever files exist, whether in motion across networks or at rest in storage. they enable seamless protection against complex and zero-day file based threats at critical points in document workflows. [Learn more about Glasswall APIs](/rest-api/about-glasswall-apis) When using the Halo portal, you can customize your protection preferences by adjusting settings across four main tabs: * [**Policy settings**](/halo/glasswall-portal-protection-settings#policy-settings) – configure how each of the [supported file types](/halo/glasswall-halo-supported-file-types) is processed. * [**ICAP settings**](/halo/glasswall-portal-protection-settings#icap-settings) – modify how files are handled over http within your organization. * [**Validation settings**](/halo/glasswall-portal-protection-settings#validation-settings) – set up XML validation options for enhanced file protection. * **File reputation** – check file hashes against ReversingLabs. See [Enable ReversingLabs](/halo/enable-reversing-labs) for the configuration steps. These options allow users to tailor their organization’s security approach to match their specific risk tolerance, ensuring files are protected in a way that aligns with internal policies and operational needs. ## Policy settings Users can configure and adjust their file protection preferences through the *Policy settings* tab. This section allows the Halo portal to understand how you want to handle potentially risky content within files, giving you control over how threats are managed during processing. By default, recommended settings are configured to sanitize, providing an optimal balance between security and usability. This ensures that malicious or suspicious content is safely removed while preserving the file’s original functionality and appearance. However, organizations can modify these settings to align with their own security policies, risk tolerance, or operational requirements, allowing a flexible approach to file protection across different departments and use cases. [Learn more about policy management](/rest-api/about-policy-management-api) ## ICAP settings When using the Halo portal, users can customize their ICAP preferences by adjusting the settings for each of the [supported media types](/halo/glasswall-supported-media-types) through the *ICAP settings* tab. These settings determine how each media type is processed over http within your organization, giving you control over file handling and security policies. You can create one or more ICAP profiles to meet different organizational requirements. For example, you might configure a profile that restricts standard users’ access to certain MIME types when uploading or downloading files over http, while allowing administrators broader access to manage files more freely. ### Select ICAP profile Select the right ICAP profile via the **ICAP profile** drop-down. **Note:** as default, the **Profile status** is set to "Enabled" if you want to allow the ICAP profile to be used in requests from your ICAP client. ### Apply policy settings You need to select the right policy settings via the **Policy to apply** drop-down. ## Other media types You can also set preferences for other media types via the API - media types set to "bypass" will be allowed through the ICAP server without being protected. [Learn more](/halo/icap-profiles-overview) ## Validation settings The *Validation settings* tab allows users to validate XML files against defined schemas and policies to ensure structural consistency, security, compliance, and accurate error detection. This process helps verify that XML content adheres to approved standards and organizational requirements before it is processed or shared. While Glasswall Halo permits the use of custom tags in XML documents, certain tags may introduce potential security risks. To manage or update tags that are not already listed within the schema, users can do so through the Glasswall API. For more details on XML schema validation and how to configure these options, refer to the [XML schema validation overview](/halo/glasswall-xml-schema-validation-overview). The Glasswall Portal allows Admin users to upload their Glasswall license via the *License management* page, as well as retrieve information about an existing license, and download a license usage report if required. If you don't have a Glasswall license but would like to obtain one, please [contact us](https://www.glasswall.com/contact/). ## Prerequisites - Required services: - `Cdrplatform/portal` - `Cdrplatform/portal-access` - Configuration of SSO login in Glasswall portal - An admin SSO login to access the *System settings* page - Valid Glasswall license in **.lic** format #### What would you like to do? * [Upload a new license](license-management-portal-setup-guide#uploading-a-new-license) * [Update an existing license](license-management-portal-setup-guide#updating-an-existing-license) * [Remove an active license](/halo/glasswall-portal-license-management#remove-an-active-license) * [License usage reports](/halo/glasswall-portal-license-management#license-usage-reports) * * * ## Upload a new license 1. Navigate to the Glasswall Portal at **[Halo.glasswall.com](https://halo.glasswall.com/)**. 2. Select *System settings* from the left hand menu. **Note:** this page is only visible if you are logged in via an Admin account. 3. Click **Browse** and upload your valid Glasswall license. Your license details are displayed, indicating that your license is ready to use. ## Update an existing license To update an existing license, navigate to the *System settings* page and click **Change license**. ## Remove an active license Please follow the steps from the [API setup guide](/halo/license-management-api-setup-guide) to remove an active license. ## License usage reports Users can generate a *License usage report* to show overall system utilisation. This report is mainly used by systems running in passive licensing mode to share license consumption data with Glasswall. Usage data is updated daily, so recent activity may not appear immediately if the report is created soon after files are processed in Halo. ### Downloading usage reports Downloading can be done from the *System settings* page by an Admin user. When downloaded via the portal, the report is provided as a `.json` file, with a name based on the license ID and the report creation date. To download a usage report, scroll down to the *License usage* section and click **Download usage report**. A file containing the report will be downloaded automatically. For more information about usage reports and their contents, see [Creating a license usage report](/halo/creating-a-license-usage-report). From the portal’s *Help* page, you can access direct links to our user documentation and technical integration guides, as well as contact details for our [customer support](https://www.glasswall.com/support). In the Halo portal, the *Reporting* section provides users with both a high-level overview and in-depth breakdown of file data processed and protected by your organization. ## Risk report ### Protected file overview This section provides a high level overview of the total files processed by your organisation including: * Number of total files protected * Total size of protected files * Total number of risky items removed * Average time to protect a file ### Risk summary The *Risk summary* section shows a more thorough breakdown of your organization's processed files data including: #### Original file risk level After a file has been rebuilt, Glasswall Halo assigns a risk level to the original file. #### File protection level File protection level represents a file’s status after it’s been rebuilt. #### Original malware status Glasswall has partnered with ReversingLabs to uncover details on how many of your original files contained malware. ReversingLabs runs the hash for each file through its file reputation database and classifies how malicious it’s likely to be. #### Severity of malicious files Original files that have been confirmed as malicious by ReversingLabs each receive a threat severity score of 1-5. #### Risky content removed This is a list of risky content that has been removed, based on your organization’s policy settings. This list only includes risky content removed from protected files. This is a list of risky content that has been removed, based on your organization’s policy settings. #### Risky content allowed by your policies This is a list of risky content that has not been removed from your processed files. * Your organization has chosen to allow this content through your policy settings. #### Top protected file types This is a list of file types that have been protected by Glasswall CDR. #### Use of Glasswall's products and APIs This is a breakdown of the Glasswall products that were used to rebuild your organization’s files. ### Risk trends In the *Risk trends* section, you can see trend charts for the following: **File protection level** This is the file count by protection level over time. A file protection level represents a file’s status after it’s been rebuilt. **Original file risk level** This is the file count by original file risk level over time. A file's original risk level represents a file's risk before it has been rebuilt. Within the reporting section of the Halo portal, the *Processed files* page provides a comprehensive overview of files that have been analyzed by Glasswall Halo within the last 30 days. Users can review key information for each file, including its name and format, the date and time it was processed, the product or service that performed the analysis, a unique transaction id for reference, and detailed results showing the processing outcome and overall protection status. In the reporting section of the Halo portal, users can access detailed insights into requests processed through the Internet Content Adaptation Protocol (ICAP). This feature showcases how Glasswall CDR rapidly analyzes and sanitizes files to remove potential threats, ensuring secure file access across different use cases and environments. The ICAP reporting view allows users to monitor the number of files that were blocked and examine a detailed breakdown of request activity by request type, request status, media type, and blocked media type. In addition, the dashboard displays visual graphs highlighting request trends over time, organized by request status and request type, helping users identify patterns, assess system performance, and track overall file protection activity. To demo and trial our advanced CDR technology, you can try Glasswall Halo's file cleaning application - a simple and user-friendly tool for removing threats from your everyday files. It's a quick and easy way to eliminate malware that could be left behind by traditional antivirus software. The file sanitization process involves deconstructing a file down to its many layers and inspecting its digital DNA. risky content is removed from the file, according to the file's 'known good' manufacturer's settings. It is then rebuilt, and a safe and visually identical file is delivered. #### What would you like to do? * [Sanitize a file](/halo/glasswall-portal-clean-a-file-standard#sanitize-a-file) * [View details](/halo/glasswall-portal-clean-a-file-standard#viewing-details) * [Download cleaned file(s)](/halo/glasswall-portal-clean-a-file-standard#download-cleaned-files) * [Sanitize multiple files](/halo/glasswall-portal-clean-a-file-standard#sanitize-multiple-files) * [Sanitize archive files](/halo/glasswall-portal-clean-a-file-standard#sanitize-archive-files) --- ## Sanitize a file Glasswall Halo's file cleaning application allows you to input files in two ways, as explained below. ### Drag and drop The quicker and more intuitive way of using this application simply involves dragging and dropping your required file into the drop zone area. * The file is then instantly processed, rebuilt, and made available for download. * A detailed report of the file changes is also created for your reference. ### Uploading a file Users can also input files via a simple upload process. * Click **Browse** from the drop zone area and select a file from your device. * Halo will then process the file and rebuild it to its 'known good' standards. ## Viewing details After your file has been processed and rebuilt by Halo, a summary of the changes made is compiled for your reference. To find out how Glasswall made your file safe, click **View details**. This will display the *Original file risk level* along with a description of what it means. ### File analysis The system breaks the file down into its core components, examining its structure, content, and potential threats. Once processing is complete, the *File analysis* view provides a detailed technical breakdown of what was discovered and the actions taken to secure the file. This includes information on any content that was removed during sanitization, the file’s reputation as assessed by ReversingLabs, a comprehensive risk analysis highlighting potential vulnerabilities, and details of any structural repairs applied to restore the file to a safe, functional state. By presenting this level of detail, users gain full visibility into the protection process, allowing them to understand exactly how the file was cleaned and reinforced while maintaining its original usability. ## Download cleaned file(s) Once Halo has processed your file, the rebuilt file will be made available to you for download. Click **Download clean file** and save it to your device. Alternatively, if you processed an entire folder of files at once, you could download the sanitized files together by clicking **Download all clean files**. ## Sanitize multiple files Along with single files, users can also sanitize multiple files and folders in one quick process. This can be achieved in 2 ways; by selecting your required files using the upload method or by dragging and dropping an entire folder into the drop zone. **Note:** a maximum of 100 files can be processed in one go. Once your files have been processed, you are able to download them all together by clicking download all clean files, as well as view their individual file analysis details. ## Sanitize archive files Archive files are quite often used as a quicker and easier way to transfer multiple files. For this reason, we have enabled Halo to process archive files and folders to ensure enterprise users can quickly check if a large number of incoming or outgoing files are a threat. You can process archive files in the same way as a single file, by either uploading the archive folder or dragging and dropping the folder into the drop zone. Halo can process an archive that contains folders and files inside, but not other archives. Once the files have been cleaned, you can download each file individually or altogether by clicking **Download all clean files**. In the *APIs* section of the Halo portal, users can explore the full range of Glasswall’s API offerings and gain a clear understanding of their capabilities. This section provides guidance on how to access the APIs, get started with integration, and effectively deploy Glasswall’s intelligent file protection technology within your existing infrastructure. Our APIs are designed to deliver Zero Trust Content Disarm and Reconstruction capabilities wherever files exist, whether in motion across networks or at rest in storage. They enable seamless protection against complex and zero day file-based threats at critical points in document workflows. [Learn more about Glasswall APIs](/rest-api/about-glasswall-apis) When using the Halo portal, you can explore your organization’s protection preferences across four main tabs: * [**Policy settings**](/halo/glasswall-portal-protection-settings-standard#policy-settings) – view how each of the [supported file types](/halo/glasswall-halo-supported-file-types) is processed. * [**ICAP settings**](/halo/glasswall-portal-protection-settings-standard#icap-settings) – see how files are handled over http within your organization. * [**Validation settings**](/halo/glasswall-portal-protection-settings-standard#validation-settings) – review the XML validation options in place for enhanced file protection. * **File reputation** – see whether file hashes are checked against ReversingLabs. Read more about [file reputation](/halo/threat-intelligence-overview). These sections provide visibility into how Glasswall Halo protects your files according to your organization’s security policies and risk tolerance. **Note:** these tabs are **view only**. As a standard user you can review the current configurations, but any updates or changes must be requested through your administrator. --- ## Policy settings The *Policy settings* tab shows how your organization manages potentially risky content within files. these settings define how threats are processed, helping maintain both security and usability. Recommended policies are usually set to **Sanitize**, which removes malicious or suspicious elements while keeping the file’s original appearance and functionality. this ensures files remain safe to use without disrupting workflow. If changes are needed to these settings, please contact your administrator. [Learn more about Policy management](/rest-api/about-policy-management-api) --- ## ICAP settings The *ICAP settings* tab displays how your organization’s ICAP profiles are configured to process files over http. These profiles determine how different media types are handled, ensuring files are analyzed and sanitized before being shared or downloaded. You can review the settings and see which ICAP profiles and policies are active, including how media types are managed within your environment. All settings in this section are view only. To request updates or changes, please get in touch with your administrator. [Learn more about ICAP profiles](/halo/icap-profiles-overview) --- ## Validation settings The *Validation settings* tab shows how XML files are validated against approved schemas and policies to ensure consistency, security, and compliance. This process confirms that XML content meets organizational standards and helps detect structural errors before files are processed or shared. These settings are view only. If any adjustments are required, please reach out to your administrator. For more information on XML schema validation and how it works, visit the [XML schema validation overview](/halo/glasswall-xml-schema-validation-overview) . From the portal’s *Help* page, you can access direct links to our user documentation and technical integration guides, as well as contact details for our [customer support](https://www.glasswall.com/support). ## Overview Each service emits a number of log statements when certain events within that service occur. When these events are logged they contain the relevant amount of written data and values to help you understand and trace that log but they also contain an event id as part of the log statement. This event ID allows you to find a series of log statements or use queries to help better understand what events have occurred within a service. You can collect and centralise the logs of Glasswall Halo using a log aggregation and monitoring tool, such as Elk stack, Splunk, or Datadog. These tools provide powerful search and visualisation capabilities, allowing you to query logs based on event IDs, timestamps, and other attributes. ## Log levels Log levels indicate the severity or importance of a log entry. This will help you prioritise and filter log messages. Here's how to understand and utilise log levels effectively: ### Log level hierarchy Log levels typically follow a hierarchy, with each level indicating increasing severity. Common log levels include: - **Information**: Provides general information about the application's operation. - **Warning**: Highlights potential issues or unexpected behaviour that may not be critical. - **Error**: Indicates errors that require immediate attention. - **Critical**: Reserved for the most severe issues that may lead to application failure. ### Consumer actions When you consume the log data you can take specific actions based on log levels: - **Info**: These logs are valuable for monitoring the regular operation of an application. Operations teams may use them to ensure the system is running smoothly. - **Warning**: Potential issues or irregularities are highlighted. Operations teams can set up alerts to proactively address these. - **Error**: Immediate attention is needed. Operations teams should investigate and address errors promptly to prevent service disruptions. - **Critical**: Indicates a severe issue that requires immediate action. Operations teams should be alerted, and incident response processes should be initiated. **The events emitted by Glasswall Halo are as follows:** ## Global event IDs | Event | ID | Description | Log level | |----------------------------|--------|------------------------------------------------------------------------------|-------------| | MongoDB-healthcheckfailed | 100100 | When the MongoDB database health check fails | Error | | MongoDB-documentsretrieved | 100101 | When a query to the MongoDB database successfully returns documents | Information | | MongoDB-documentinserted | 100102 | When documents are successfully inserted into the MongoDB database | Information | | MongoDB-documentsdeleted | 100103 | When documents are successfully deleted from the MongoDB database | Information | | MongoDB-sumofcalculated | 100104 | When a `sumof` query is successfully executed against the MongoDB database | Information | | MongoDB-documentsupdated | 100105 | When documents are successfully updated in the MongoDB database | Information | | MongoDB-documentsmatched | 100106 | When an `exists` query is successfully executed against the MongoDB database | Information | | CosmosDB-healthcheckfailed | 100150 | When the cosmos database health check fails | Error | | CosmosDB-itemsretrieved | 100151 | When a query to the cosmos database successfully returns items | Information | | CosmosDB-iteminserted | 100152 | When items are successfully inserted into the cosmos database | Information | | CosmosDB-itemsdeleted | 100153 | When items are successfully deleted from the cosmos database | Information | | CosmosDB-sumofcalculated | 100154 | When a `sumof` query is successfully executed against the cosmos database | Information | | CosmosDB-itemsupdated | 100155 | When items are successfully updated in the cosmos database | Information | | CosmosDB-itemsmatched | 100156 | When an `exists` query is successfully executed against the cosmos database | Information | ## Sync API | Event | Id | Description | Log level | | ---------------------------- | --- | ------------------------------------------------------------------------------------------------------------------------ | ----------- | | Receivedauthresponse | 5 | When the API received the response from the Engine service | Information | | Subscribedtoqueue | 6 | When the API has subscribed to the response queue | Information | | Createdqueue | 7 | When the API creates the correct response queue | Information | | Sentmessage | 8 | When the API successfully submits a message to the rabbit mq | Information | | Filewritten | 9 | When the API successfully writes a file to the file storage | Information | | Fileread | 10 | When the API successfully reads the rebuilt file or analysis from storage | Information | | Messagepollerstarted | 11 | When the rate limiting polling has started | Information | | Incorrectresponsemessage | 12 | When the API received a response from the Engine service but there is no task waiting to return the response to the user | Warning | | Operationfailed | 13 | When an operation within the API fails | Warning | | Retryingpolicy | 14 | When an operation retries based on the defined retry policy | Warning | | Failedrebuildarchiveentry | 15 | When the API fails to rebuild the clean files sections of an archive response | Warning | | Failedanalysisarchiveentry | 16 | When the API fails to rebuild the analysis section of an archive response | Warning | | Ratelimited | 17 | When a request is not allowed due to rate limiting being reached | Warning | | Errorhandlingrpcresponse | 18 | When the API fails to handle a response message from the Engine | Error | | Failedtosendtransactionevent | 19 | When the API fails to send transaction events used for dashboards | Error | | Requestfailed | 20 | When the request to the API fails for a known reason | Error | | Unknownrequestfailure | 21 | When the request to the API fails for an unknown reason | Error | | Pipelinestarted | 22 | When an action within the API begins | Information | | Pipelinecompleted | 23 | When an action within the API has completed | Information | | Starterror | 24 | When the API fails to start up | Information | | Storagenotfound | 25 | When the API is unable to find the mounted storage location | Information | ## Engine | Event | Id | Description | Log level | | ----------------------------- | ---- | --------------------------------------------------------------------------------- | --------------- | | Metricsoutput | 2 | When the Engine service logs metrics to stdout | Information | | Subscribedtoqueue | 3 | When the Engine service has subscribed to the request queue | Information | | Createdqueue | 4 | When the Engine service has created the request and logging queue | Information | | Sentmessage | 5 | When the Engine service has sent a message to the response or report queue | Information | | Completedmessage | 6 | When the Engine service has completed the request message from the API | Information | | Rejectedmessage | 7 | When the Engine service rejects the request message from the API | Rejectedmessage | | Getstream | 8 | When the Engine service gets stream of a file stored on the mounted disk | Information | | Writtenfile | 9 | When the Engine service has written the rebuilt file to the mounted storage | Information | | Servicestopping | 10 | When the Engine service stops running | Information | | Messagereceivedduringshutdown | 11 | When the Engine service receives a message during its shutdown process | Information | | Handlingmessage | 12 | When the Engine service begins to handle a request | Information | | Writeanalysisfailed | 14 | When the Engine service fails to write the analysis report to the mounted storage | Warning | | Writecleanfilefailed | 15 | When the Engine service fails to write the clean file to the mounted storage | Warning | | Retryingpolicy | 16 | When the Engine service retries an operation based on a defined retry policy | Warning | | Engineversionfailure | 17 | When the Engine service fails to get the version from the embedded Engine | Warning | | Rabbitmqhealthfailure | 18 | When the RabbitMQ health check fails | Warning | | Errorhandlingmessage | 20 | When the Engine service fails to handle a request message | Error | | Messageerror | 21 | When the Engine service has an issue processing a message from the request queue | Error | | Servicedisconnected | 22 | When the Engine service disconnects | Critical | | Storagenotfound | 23 | When the Engine service cannot find the mounted storage | Information | | Starterror | 24 | When the Engine service fails to start | Information | | Trackeractionfinished | 25 | When the Engine service completed tracking an event within the service | Warning | | Servicestarting | 26 | When the Engine service starts up | Information | | Storagehealthfailure | 27 | When the storage health check fails | Warning | | Writeexportfilefailed | 28 | When the Engine service fails to write the export file to the mounted storage | Warning | | Detectingpathfiletype | 2001 | When detectfiletypefromfile is called | Information | | Detectingmemoryfiletype | 2002 | When detectfiletypefrommemory is called | Information | | Nolicensesetatsessioncreation | 2003 | When no active license was found. In this case, no CDR occurs | Critical | | Enginelicenseerror | 2004 | An active license was found, but it is invalid or expired | Critical | ## Report extractor | Event | Id | Description | Log level | |-------------------------------------|----|--------------------------------------------------------------------------------------------|-----------| | Unabletoconnecttomessagebroker | 1 | When the service is unable to connect to the message broker at start up | Critical | | Unabletoconnecttomessagebrokerqueue | 2 | When the service is unable to connect to the response queue at start up | Critical | | Unexpectedmessagebrokerexception | 3 | When the API has an unexpected error connecting to the message broker or queue at start up | Critical | | Storagehealthfailure | 4 | When the storage health check fails | Error | | Readanalysisreport | 5 | When the report extractor has read the analysis report from the mounted drive | Information | | Failedtoreadanalysisreport | 6 | When the report extractor fails to read the analysis report from the mounted drive | Error | ## API access | Event | Id | Description | Log level | | -------------------- | --- | --------------------------------------------------- | ----------- | | Anonymousauth | 1 | When a request is allowed via anonymous auth | Information | | Missingauthheader | 2 | When a request is missing the required auth header | Information | | Validorganisation | 3 | When a request is made with the correct auth org id | Information | | Invalidauthheader | 4 | When a request is made with invalid auth header | Information | | Servicefailedtostart | 5 | When the service fails to start | Critical | | Upstreamresponse | 6 | When an upstream service returns a response | Information | | Upstreamrequest | 7 | When a request to an upstream service is made | Information | | Requesterror | 8 | When a request errors | Error | ## Portal access | Event | Id | Description | Log level | | ---------------- | --- | --------------------------------------------- | ----------- | | Upstreamresponse | 1 | When an upstream service returns a response | Information | | Upstreamrequest | 2 | When a request to an upstream service is made | Information | | Starterror | 3 | When the service fails to start | Critical | | Requesterror | 4 | When a request errors | Error | ## Async API | Event | Id | Description | Log level | | ----------------------------------- | --- | ----------------------------------------------------------------------------------------------- | ----------- | | Servicestarting | 1 | When the API starts up | Information | | Servicestopping | 2 | When the shutdown process is called for async API | Information | | Messagereceivedduringshutdown | 3 | When the API receives a message during its shutdown process | Warning | | Handlingmessage | 4 | When the API is handling a message from the queue | Information | | Handledmessage | 5 | When the API has successfully handled a message from the queue | Information | | Messageerror | 6 | When the API has an issue processing a message from the queue | Error | | Servicedisconnected | 7 | When the API disconnects from the message broker | Critical | | Subscribedtoqueue | 8 | When the API subscribed to receive message from the queue | Information | | Sentmessage | 9 | When the API has sent a message for processing | Information | | Filewritten | 10 | When the API successfully writes a file to the file storage | Information | | Fileread | 11 | When the API successfully reads the rebuilt file or analysis from storage | Information | | Incorrectstatusstored | 12 | When the status for a stored item is not the expected status | Warning | | Operationfailed | 13 | When an operation within the API fails | Critical | | Retryingpolicy | 14 | When an operation retries based on the defined retry policy | Warning | | Requestfailed | 16 | When the request to the API fails for a known reason | Error | | Unknownrequestfailure | 17 | When the request to the API fails for an unknown reason | Error | | Pipelinestarted | 18 | When an action within the API begins | Information | | Pipelinecompleted | 19 | When an action within in the API has completed | Information | | Starterror | 20 | When the API fails to start up | Information | | Storagenotfound | 21 | When the API is unable to find the mounted storage location | Information | | Hashingerror | 22 | When the API is unable to generate hashes for the file | Error | | Deletefailed | 23 | When the API is unable to delete the file from the mounted storage location | Error | | Unabletoconnecttomessagebroker | 24 | When the API is unable to connect to the message broker at start up | Critical | | Unabletoconnecttomessagebrokerqueue | 25 | When the API is unable to connect to the response queue at start up | Critical | | Unexpectedmessagebrokerexception | 26 | When the API has an unexpected error connecting to the message broker or queue at start up | Critical | | Databasehealthfailure | 27 | When the API's database health check fails to connect to the database | Warning | | Messagebrokerhealthfailure | 28 | When the API's message broker health check fails to connect to the message broker | Warning | | Databasefailure | 29 | When the API is unable to complete an operation on the database after maximum number of retries | Error | | Databaseitemretrieved | 30 | When the API has performed a retrieval on an item in the database | Information | | Databaseitemstored | 31 | When the API has stored an item in the database | Information | | Databaseitemupdated | 32 | When the API has updated an item in the database | Information | | Handlesinglefileresponse | 33 | When the API has received a response relating to a single file | Warning | | Handlearchivecontentresponse | 34 | When the API has received a response relating to an archive content item | Warning | ## Tally accumulator | Event | Id | Description | Log level | | ------------------------------------ | --- | ------------------------------------------------------------------------------------------------------------------- | ----------- | | Unabletoconnecttomessagebroker | 1 | When the API is unable to connect to the message broker at start up | Critical | | Unabletoconnecttomessagebrokerqueue | 2 | When the API is unable to connect to the response queue at start up | Critical | | Unexpectedmessagebrokerexception | 3 | When the API has an unexpected error connecting to the message broker or queue at start up | Critical | | Starterror | 4 | When the API fails to start up | Critical | | Processtransactioneventstart | 5 | When a supported transaction is received and processing has started | Information | | Processtransactioneventend | 6 | When a supported transaction is received and processing has finished successfully | Information | | Licensetransactionstoreupsertstart | 7 | When an upsert is about to happen against the license transaction store | Information | | Licensetransactionstoreupsertsuccess | 8 | When an upsert against the license transaction store has finished successfully | Information | | Tallystoreupsertstart | 9 | When an upsert is about to happen against the tally store | Information | | Tallystoreupsertcomplete | 10 | When an upsert against the tally store is successful | Information | | Tallystoregetusagestart | 11 | When license usage is about to be retrieved from the tally store | Information | | Tallystoregetusagesuccess | 12 | When license usage is retrieved from the tally store successfully | Information | | Unexpectednegativefilesize | 13 | When a license transaction event for a file has a negative file size | Warning | | Unexpectedrequestexception | 14 | When an unexpected exception is raised by the service that will result in a 500 status code for a given API request | Error | | Unsupportedentitlementlimitname | 15 | When the supplied filter for the usage endpoint is not a valid entitlement limit | Warning | | Databasehealthfailure | 16 | When the service health checks fail to contact the database instances | Warning | | Erroredfilereceived | 17 | When an error occurs upstream. this will result in the tally not being incremented | Warning | ## Metrics collation (deprecated) > As of version 2.6.2 of Halo, the metrics collation service is no longer required. | Event | Id | Description | Log level | | ----------------------------------- | --- | ---------------------------------------------------------------------------------------------- | ----------- | | Unabletoconnecttomessagebroker | 1 | When the service is unable to connect to the message broker at start up | Critical | | Unabletoconnecttomessagebrokerqueue | 2 | When the service is unable to connect to the response queue at start up | Critical | | Unexpectedmessagebrokerexception | 3 | When the service has an unexpected error connecting to the message broker or queue at start up | Critical | | Storagehealthfailure | 4 | When the storage health check fails; this is usually a problem with database connectivity | Error | | Writingeventtomongostorage | 5 | When an incoming event is about to be persisted in the database | Information | | Mongowriteerror | 7 | When an incoming event fails to be persisted in the database | Error | | Jsonexceptionerror | 8 | When the message listener fails to deserialize an event | Error | | Unknownfailedtoprocess | 10 | When an unhandled exception occurs while processing an incoming event | Critical | | Failedtoimporttracedata | 11 | When the trace provider fails to import trace data from an incoming event header | Warning | | Storagehealthcheckstartup | 12 | When the service performs a storage health check at startup | Information | ## Metrics projection | Event | Id | Description | Log level | | --------------------------------- | --- | ---------------------------------------------------------------------------------------------------------------- | ----------- | | Starterror | 1 | When the service fails to start up | Critical | | Storagehealthfailure | 2 | When the storage health check fails; this is usually a problem with database connectivity | Warning | | Changeprocessorstart | 5 | When the change processor is about to setup a watcher for database changes | Information | | Changeprocessorstarted | 6 | When the change processor has successfully setup a watcher for database changes | Information | | Changeprocessorstopping | 7 | When the change processor is about to stop watching for database changes | Information | | Changeprocessorstopped | 8 | When the change processor has successfully stopped watching for database changes | Information | | Changeprocessorstopfailure | 9 | When the change processor fails to stop watching for database changes, i.e. it was not watching for changes | Warning | | Changeprocessorbatchstart | 10 | When the change processor has received a batch of database changes and is about to start processing them | Information | | Changeprocessorbatchend | 11 | When the change processor has finished processing the current batch of database changes | Information | | Changeprocessorunhandledexception | 13 | When the change processor encounters a problem while processing a change; this may cause inconsistencies in data | Error | | Transactioneventprojected | 14 | When an event (change) has been successfully mapped to a projection | Debug | | Changehandlerstarted | 15 | When a change handler is about to start processing changes after being called upon by the change processor | Information | | Changehandlerfinished | 16 | When the change handler has finished processing changes | Information | | Changehandlerfailure | 17 | When the change handler has failed to process changes | Critical | | Changeprocessorstartfailure | 19 | When the change processor fails to start watching for database changes | Error | | Changeprocessorwatchstart | 20 | When the change processor has started actively watching for database changes (after setting up a watcher) | Information | | Daywisesummaryincremented | 21 | When a day-wise summary has been successfully incremented based on a completed file | Information | | Daywisesummarynotincremented | 22 | When a day-wise summary will not be incremented based on a (usually incomplete) file | Debug | | Retryingpolicy | 23 | When an operation retries based on the defined retry policy | Warning | | Materializedviewpersisted | 24 | When a materialized view (projection) has been persisted in the database after being projected upon | Information | ## License management | Event | Id | Description | Log level | | ------------------------- | --- | ---------------------------------------------------------------------------------- | ----------- | | Starterror | 1 | When the service fails to start up | Critical | | Licenseprocessingerror | 2 | When a request fails for an unforeseen reason | Error | | Deletelicensestart | 3 | When the service begins deleting the current active license | Information | | Deletelicenseend | 4 | When the service has finished deleting the current active license | Information | | Updatelicensestart | 5 | When the service begins updating the current active license | Information | | Updatelicenseend | 6 | When the service has finished updating the current active license | Information | | Getlicensestart | 7 | When the service begins retrieving the current active license | Information | | Getlicenseend | 8 | When the service has finished retrieving the current active license | Information | | Licensenotfound | 9 | When the current active license cannot be found during update or retrieval | Error | | Licenseexpired | 10 | When the uploaded license has expired, causing the update to fail | Error | | Fetchinglicenseusagestart | 11 | When the service begins retrieving the current active license and usage data | Information | | Fetchinglicenseusageend | 12 | When the service has finished retrieving the current active license and usage data | Information | ## ICAP server | Event | Id | Description | Log level | |----------------------------------------|------|----------------------------------------------------------------------------------------------------------------------------------|-------------| | Backgroundservicestarting | 1 | When the ICAP listener is starting | Information | | Backgroundservicestarted | 2 | When the ICAP listener has started | Information | | Backgroundservicestopping | 3 | When the service shuts down and the ICAP listener is stopping | Information | | Backgroundservicestopped | 4 | When the service shuts down and the ICAP listener has completed stopping | Information | | Starterror | 5 | When the service fails to start | Critical | | Halocallfailure | 8 | When a file cannot be built via the Halo API | Warning | | Profilestoremaintainerstarting | 9 | When the background thread starts listening for ICAP profile changes to persist locally | Information | | Profilestoremaintainerstarted | 10 | When the background thread listening for ICAP profile changes has started successfully | Information | | Profilestoremaintainerstopping | 11 | When the service shuts down and the profile background thread stops listening for profile updates | Information | | Profilestoremaintainerstopped | 12 | When the service shuts down and the profile background thread completes its shutdown process | Information | | Profilestoremaintainerheartbeaterror | 13 | When profiles are queried and cannot be persisted due to an error | Error | | Profilestoremaintainerheartbeatsuccess | 14 | When profiles are queried successfully | Information | | Profileusagefailure | 15 | When the profile specified in an ICAP request is not found | Warning | | Errorhandled | 17 | When the ICAP server finishes processing an unexpected exception by sending an error response to the ICAP client | Error | | Unlicensedrequest | 18 | When an ICAP request is received for an unlicensed ICAP server | Critical | | Retryingpolicy | 19 | When an action is being retried, for example a failed call into the Halo API | Warning | | Fileblocked | 20 | When an ICAP request is processed containing a file that is blocked | Information | | Fileprocessingstarted | 21 | When starting to process a file within an ICAP request | Information | | Profilefailedtofetch | 22 | When a profile cannot be retrieved from the policy API | Error | | Rebuiltfileexceedscachesize | 23 | When the ICAP request contains a file too large to be stored in the cache | Warning | | Fileretrievedfromcache | 24 | When an ICAP request is processed and an associated rebuilt file is found in the cache | Information | | Fileprocessingfinished | 25 | When finished processing a file within an ICAP request | Information | | Clearingcache | 26 | When running cache cleanup starts | Information | | Cachecleared | 27 | When running cache cleanup finishes | Information | | Errorclearingcache | 28 | When running cache cleanup errors | Error | | Filebypassed | 29 | When a file has been bypassed during modification by the ICAP policy in use | Information | | Fileprocessed | 30 | When a file has been processed during modification by the ICAP policy in use | Information | | Fileerrored | 31 | When an error occurs while attempting to process a file during modification | Information | | Tlsdisabled | 32 | When no valid certificates are provided to the ICAP server, the server stops listening on secure port 11344 and tls is disabled. | Warning | | Tlsenabled | 33 | The ICAP server has found a valid tls certificate and is listening on the secure port 11344. | Information | | Serverstarting | 1000 | When the ICAP server is starting | Information | | Serverstarted | 1001 | When the ICAP server has started | Information | | Clientsocketerror | 1002 | When an unexpected exception occurs while listening to the ICAP client | Error | | Unsupportedicaprequest | 1003 | When the ICAP server receives a request that is not supported, for example the method is not reqmod, options or respmod | Warning | | Clientconnectionhandleerror | 1004 | When an error occurs when processing data sent to the server | Error | | Sendfailure | 1005 | When attempting to send a response to the ICAP client fails | Warning | | Serverlistenercancelled | 1006 | When the ICAP listener thread is cancelled, for example on shutdown | Warning | | Errorduringsocketclose | 1007 | When closing a connection raises an error, for example when the client has already disconnected | Information | | Clientreceiveerror | 1008 | When listening to data on the socket sent by the ICAP client and error occurs | Warning | | Requestprocesserror | 1009 | When an unhandled exception occurs during the processing of an ICAP request | Error | All Glasswall Halo services have implemented health checks to communicate their state to Kubernetes. If a pod enters an unhealthy state, Kubernetes will disable it and create a new one in its place. ## Synchronous API and Asynchronous API Our API's are hosted in the Kestrel web server which leverages HTTP health checks. These return a 200 ok message when they are in a healthy state and a 503 when in an unhealthy state. The checks can be made with a call to: ``` /api/health ``` ## Glasswall Engine, Report aggregator and Cleanup These services are not hosted with a web server, instead they spin up their own TCP listeners that accept inbound connections. As long as a connection occurs, a liveness probe will see that as being in the 'healthy' state. If the service is in an unhealthy state, the listener must be stopped. Glasswall Halo is comprised of multiple services that communicate using asynchronous messages rather than HTTP. For these services, we have made considerations about how the platform handles failure in the event of an error when processing a message. ## Synchronous API and Asynchronous API The APIs subscribe to response queues from the Glasswall Engine, however in the event of failure an error response will be returned to the http client. ## Engine service The architecture is request/response using RabbitMQ reply-to queues. This means in the event of failure, when handling a message we still send a response message containing the reason for failure. ## Report aggregator The report aggregator behaves as a standard message consumer and extra functionality was introduced to enable message retrying. Two configuration options are present on the service. `Retrymessages` is a bool defaulting to true that enables the functionality to retry messages. If turned off all failed messages are immediately put in a dead letter queue. `Messagettl` is an integer that represents the number of seconds a message can be in the queue. This is to avoid messages backing up because of a failed message infinitely retrying. When the time to live has elapsed for a message, it is put in the dead letter queue. the code default is `30` seconds. The report aggregator's functionality is not critical to clients receiving their files and is used to produce aggregated report data. Due to this the dead lettering of a report aggregator message is not deemed a critical failure and dead letters are tolerated (only in extreme cases). Glasswall Halo implements **graceful shutdown** to allow services to complete in-flight requests while rejecting new requests during the shutdown process. This ensures that outstanding work is handled correctly and reduces the risk of data loss or partially processed requests. ## Synchronous API and Asynchronous API The Glasswall Halo APIs use the ASP.NET Core graceful shutdown mechanism. More information is available in the [Microsoft documentation](https://learn.microsoft.com/en-us/dotnet/api/microsoft.aspnetcore.server.kestrel.kestrelserveroptions.shutdowntimeout) for Kestrel shutdown timeout. By default, the shutdown timeout is set to **60 seconds**. This value defines how long the server will wait after shutdown begins before open connections are forcefully closed. During this period: - Kestrel waits for active request processing to complete. - No new connections or requests are accepted. It is possible for some background tasks to continue running after the shutdown timeout has elapsed. This behaviour is described in the [Microsoft documentation](https://learn.microsoft.com/en-us/aspnet/core/fundamentals/host/hosted-services?view=aspnetcore-7.0#stopasync) for hosted services stop handling. Even after cancellation is requested, tasks are not abandoned. The host waits for all tasks to complete before final termination. ## Report aggregator and Engine services The **Report Aggregator** and **Engine** services are implemented as hosted services, allowing custom logic to be executed during service startup and shutdown. Further details are available in the [Microsoft documentation for IHostedService](https://learn.microsoft.com/en-us/dotnet/api/microsoft.extensions.hosting.ihostedservice). When shutdown begins: - Both services allow in-flight requests to complete. - A default shutdown timeout of **5 seconds** is enforced. - If the timeout is exceeded, the service is forcefully terminated. This design ensures predictable shutdown behaviour while balancing reliability and shutdown speed. From time to time, you will need to upgrade Glasswall Halo to stay up to date with the latest fixes, features, and releases. ## Upgrading Glasswall Halo ### 1. Set the default Kubernetes namespace Ensure your current Kubernetes context is set to the `cdrplatform` namespace: ```bash kubectl config set-context --current --namespace=cdrplatform ``` ### 2. Identify the latest image and chart versions Use the Azure CLI to list available image tags in the Glasswall Artifact Registry: ```bash az acr repository show-tags --repository --name glasswallhub --username "${token_ID}" --password "${token}" --orderby time_desc ``` Replace `` with the appropriate image repository name. ### 3. Authenticate to the Helm registry Log in to the Glasswall Helm registry using your credentials: ```bash helm registry login glasswallhub.azurecr.io --username "${token_ID}" --password "${token}" ``` ### 4. Pull the latest Helm chart Download and extract the required Helm chart version: ```bash helm pull oci://glasswallhub.azurecr.io/helm/ --version "${version}" --untar --untardir . ``` Replace `` with the chart repository name and `${version}` with the desired chart version. ### 5. Upgrade the Helm release Run the Helm upgrade command using the required image tag and repository: ```bash helm upgrade --install --set image.tag= --set image.repository=glasswallhub.azurecr.io/ ``` Replace the placeholders with the appropriate chart name, image tag, and repository. **Note:** always review the release notes before upgrading to ensure compatibility and to understand any breaking changes. ## After upgrading: historical data and reporting Reporting data shown in the portal (processed-file history, risk reports, ICAP reports) is stored in **MongoDB**, while files in flight are held on the persistent volume. An upgrade migrates the platform services; it does not by itself move or reprocess historical reporting records. Set expectations accordingly after an upgrade: - **Historical records remain in MongoDB.** The portal reads reporting data through the `cdrplatform-metrics-projection` service. If historical files are not visible immediately after an upgrade, confirm that MongoDB and the metrics-projection service are running and have completed reconciliation before assuming data loss. - **Schema or projection changes** between distant versions can mean that records created on the old version are reprojected before they reappear in the portal. Allow time for this to complete. - **Large version jumps** (for example, upgrading across several minor versions at once) are higher risk. Where possible, upgrade incrementally and review the release notes for each intervening version for data-migration notes. If historical records are still not visible after the platform is healthy and reconciliation has completed, do not delete or recreate volumes—contact [Glasswall Support](https://www.glasswall.com/support) with your source and target versions so the migration can be investigated. ## Upgrading the Glasswall Halo OVA Follow the steps below to upgrade your Glasswall Halo OVA while preserving MongoDB data. --- ### 1. Back up MongoDB data First, create a MongoDB backup using `mongodump`. ```shell kubectl port-forward svc/mongo-svc 27017 ``` Open another terminal on the VM and run: ```shell username=$(kubectl get secret mongodb-cdrplatform-cdrp-user -o json | jq -r '.data.username' | base64 -d) password=$(kubectl get secret mongodb-cdrplatform-cdrp-user -o json | jq -r '.data.password' | base64 -d) mongodump \ --host localhost \ --port 27017 \ --db cdrplatform \ --out halo_backup \ -u "${username}" \ -p "${password}" zip -r halo_backup.zip halo_backup ``` --- ### 2. Copy the backup to a local machine ```shell scp glasswall@:~/halo_backup.zip ./halo_backup.zip ``` --- ### 3. Deploy a new VM Create a new virtual machine using the latest OVA or VHD by following the VMware or Hyper-V deployment guide. --- ### 4. Copy the backup to the new VM ```shell scp ./halo_backup.zip glasswall@:~/ ``` --- ### 5. Restore MongoDB data on the new VM ```shell kubectl port-forward svc/mongo-svc 27017 ``` Open another terminal on the VM and run: ```shell unzip halo_backup.zip username=$(kubectl get secret mongodb-cdrplatform-cdrp-user -o json | jq -r '.data.username' | base64 -d) password=$(kubectl get secret mongodb-cdrplatform-cdrp-user -o json | jq -r '.data.password' | base64 -d) mongorestore \ --host localhost \ --port 27017 \ --authenticationMechanism SCRAM-SHA-256 \ --authenticationDatabase cdrplatform \ -u "${username}" \ -p "${password}" \ halo_backup ``` --- ### 6. Clean up backup files ```shell rm -rf halo_backup halo_backup.zip ``` --- ## Structural repair in Glasswall CDR Structural repair is a core component of Glasswall's Content Disarm and Reconstruction (CDR) technology. It focuses on the safe reconstruction of digital files by validating and repairing their internal structures against manufacturer specifications and international standards. The structural repair process is composed of two key phases: **Inspect** and **Rebuild**. --- ### Structural analysis [Inspect] During the analysis phase, the Glasswall Embedded Engine analyzes the file’s internal composition, identifying and mapping all structural components. * The file is parsed into a tree-like structure, similar to the Document Object Model (DOM) used in HTML. * Any compressed elements are automatically decompressed to enable full inspection. * If the file contains embedded objects (e.g., a JPEG image within a Word document), each embedded item is treated as a distinct file. Separate tree structures are generated for each. * The output of this stage is a detailed structural analysis report, outlining the format, size, hierarchy, and compliance of the file’s components. --- ### Structural repair [Rebuild] In the repair phase, the Glasswall Embedded Engine processes the structural tree created during inspection to validate and reconstruct the file. * An iterator traverses the tree, validating each node against a "known good" specification, such as ISO 32000 for PDFs. * If a structure fails validation, the engine attempts to repair the structure to match the required specification. * If repair is not possible, the issue is logged, and the system may choose not to regenerate that part of the file. This phase ensures that only safe, standards-compliant data is retained in the reconstructed version of the file. ### Outcome By performing structural repair, Glasswall ensures that all output files are fully standards-compliant, free from hidden or malformed content, and safe for use in secure environments. Glasswall CDR can identify various types of risky content and remove them from your files. You're able to set preferences for the following risky content depending on your file type: | Risky content type | Risk level | Description | | --- | --- | --- | |**Acroform** | High risk | An AcroForm is a PDF that contains form fields ('Acrobat Form'). In addition to looking like a form, it may also contain active code (e.g. JavaScript) that could be malicious. AcroForms can also be used to hide objects inside other objects. | | **Connections** | High risk | The Excel connections feature controls connections to external data sources. This may present a risk if the external data source is compromised with malware or lead to SQL injection attacks, where the external data source is an SQL database. | | **Digital signatures** | Low risk | The source document may have been signed with a digital signature. While the signing may not represent a threat, if the ownership and trust of the certificate chain has been compromised, it could trick a user into viewing a document containing something malicious. | | **DDE** | High risk | Dynamic Data Exchange (DDE) allows data to be shared between applications in some old versions of Windows. DDE within a Microsoft document presents risk, as it may be used to execute malicious code on the recipient's computer. | | **Embedded Files** | High risk | Embedded objects are files, items, or entities that have been added into a different program or object (for instance, a gif in a Word document). Embedded objects within files may present risk if they provide a way for active code to be triggered or hide data within a document. | | **Embedded images** | Medium risk | Embedded images are pictures where data has been incorporated into the file. They may present risk if they provide a way for malicious content to be hidden inside the image. | | **External hyperlinks** | Medium risk | External hyperlinks redirect you to open something outside the file. A hyperlink may seem innocent, but it could route to a different destination than the link that's displayed. Caution is advised when clicking links in documents. | | **Foreign objects** | Medium risk | Foreign objects may include items like HTML code, images, or other media types. They allow the author to include elements from other file formats or programming languages within the file that could be malicious.​ | | **GeoTIFF** | Medium risk | GeoTIFF is an extension to the TIFF file format and can include metadata that describes the coordinate system, projection, and other spatial properties of an image that the owner may not intend to disclose. | | **Internal hyperlinks** | Medium risk | Internal hyperlinks route you to a different place within a file. An internal hyperlink may seem innocent, but it could route you to a different destination than the link that's displayed. Caution is advised when clicking links in documents. | | **Javascript** | High risk | JavaScript is a form of active code that may be benign in nature, but all too often is used by bad actors to mount an attack against the user or receiving system in a business document. | | **Macros** | High risk | Macros are sequences of events (including keystrokes or clicks) that can be automated. Macros may seem benign in nature, but all too often are used by bad actors to mount an attack against the user or receiving system in a business document. | | **Metadata** | Medium risk | Metadata is data about other data. Metadata may reveal information the owner may not intend to disclose, such as what computer was used to create the document or the original author's name. | | **PDF actions** | High risk | PDF actions are built-in functionality in PDF documents. PDFs could include buttons that execute JavaScript or links to untrustworthy external resources. This means they could be used to launch a phishing attack, download malware, or steal sensitive information. | | **Review comments** | Medium risk | Review comments are comments that have been added to Microsoft files. They may reveal information the owner may not intend to disclose, such as tracked changes or the original author's name. | | **Scripts** | High risk | A script is a form of active code that may be benign in nature, but all too often is used by bad actors to mount an attack against the user or receiving system in a business document. | | **Hidden data** | Medium risk | Hidden data is content that has been concealed from view, but that can be revealed again easily, such as hidden Excel columns, PowerPoint slides or Word text. Revealing hidden data may show information the owner didn't intend to disclose. | | **Tracked changes** | Medium risk | Tracked changes convey the history of edits made within a file. They may reveal information the owner didn't intend to disclose. | | **Slide notes** | Medium risk | Slide notes appear in a field where authors can add supplemental text to help in presentations. They may reveal information the owner may not intend to disclose. | | **Document variables** | Medium risk | Document variables are hidden values stored inside a Microsoft Word document that can be read or set by macros. They can be used to conceal sensitive information within a file or to hide parts of a malicious payload that a macro later assembles and executes. | | **Headers and footers** | Medium risk | Headers and footers appear at the top and bottom of document pages. They may reveal metadata the owner didn't intend to disclose or contain hidden content. | | **Non-standard sections** | High risk | Non-standard sections are data stored in a TIFF file outside the standard image structure, giving attackers space to hide malicious content. Removing them won't affect the image's appearance, but it could impact functionality in third-party programs. | | **Web video extension** | Medium risk | Web video extension embeds video from external sources. These elements can carry HTML and JavaScript that run when a user interacts with the document. | For more information please contact [Glasswall Support](https://www.glasswall.com/contact). Glasswall has partnered with ReversingLabs by integrating their cutting edge file reputation technology into Glasswall Halo. ReversingLabs runs the hash for each file through its file reputation database and classifies how malicious it's likely to be. If this feature is enabled for users, they will see a section displaying the *Original malware status* of files in the Halo portal's *Reporting* page. The different malware statuses are described here: | Original malware status | Description | | --- | --- | | Malicious | These original files were confirmed as malicious by ReversingLabs before Glasswall processed them. | | Suspicious | These original files were considered to be suspicious by ReversingLabs before Glasswall processed them. They may be considered malicious or likely good over time, when more information changes their threat profile. | | Likely good | These original files were found in the file reputation database and were considered to be benign by ReversingLabs. | | Not recognized | These original files weren't found in the file reputation database by ReversingLabs. This usually means that they're files that have only circulated internally, or they haven't circulated enough to appear in the file reputation database. | | Unavailable | The malware status of these original files is unavailable. This can happen for a few reasons, including exceeding the license limit of file reputation requests, API failure, or credential failure. | ### Severity of malicious files Original files that have been confirmed as malicious by ReversingLabs each receive a threat severity score of 1-5. * **High (5-4)** * Highest severity of malicious file threat levels designated for malware like different types of Trojans. * **Medium (3-2)** * Medium severity of malicious file threat levels, designated for malware like first stage exploits. * **Low (1)** * Lowest severity of malicious file threat levels, designated for malware like adware or potentially unwanted applications (PUA). --- [Learn more about ReversingLabs](https://www.reversinglabs.com/products/spectra-intelligence) Glasswall Halo processes files to ensure they are safe from threats. Once a file has been deconstructed and rebuilt, the application assigns a _Risk Level_ to the original file. These risk levels, explained below, provide users with a high-level summary of the threat level the original file contained. | | | | --- | --- | | | **Low** risk means the original file did not appear to contain any threats or items that might increase risk for your environment. | | | **Medium** risk means the original file contained items that may be used by an attacker to compromise your environment or expose information about the file itself... | | | **High** risk means the original file contained software or executable code that may be used by attackers to compromise your environment. | Glasswall Meteor provides Zero Trust file protection in an easy to install desktop application. Instead of looking for malicious content, our advanced CDR (Content Disarm and Reconstruction) technology treats all files as untrusted, validating, rebuilding and cleaning each file to a safe and compliant standard; automatically removing potential threats. You can also seamlessly connect with 200+ cloud and network providers and enhance Meteor’s remote storage connectivity options with our complimentary tool [Meteor Connect](/meteor/about-meteor-connect), which enables secure file transfer within your current cloud and network environments. [View supported file types](/docs/supported-file-types) .png) ## Enable employees to work safely with any file * Enable users to work risk-free with any file received outside your organization * Instantly disarm threats in files from unknown sources (e.g. a customer’s USB drive) * Users get immediate feedback on risks identified in a file and how they were removed * Retain both original and rebuilt files for audit purposes ## Use cases * File uploads or downloads from unknown source * Malware risk removal * Zero-Day threat defense ## How it works Glasswall Meteor is a user friendly application that installs directly on a user's device. No special training is required. Users simply drag and drop files into the desktop interface as needed, or set up automatic folder synchronization to ensure that only safe, clean, and fully functioning files enter the organization. ## Meteor Connect [Glasswall Meteor Connect](/meteor/about-meteor-connect) combines with **Glasswall Meteor** to ensure files that are stored in remote locations can benefit from Glasswall’s industry leading Zero Trust CDR technology. File-based threats are removed by Glasswall Meteor, with clean files being automatically transferred to a safe target location. Meteor Connect provides support for over 200 cloud and network providers — [contact us](https://www.glasswall.com/contact) to find out more. [](/meteor/about-meteor-connect) ### Glasswall Meteor *Essentials* * Glasswall Meteor *Essentials* analyzes and protects up to 150 files in a 24-hour period, during your software subscription period. * Meteor *Essentials* can process individual files up to 25 MB in size. * Meteor *Essentials* supports most popular file types such as *.pdf*, *.doc*, *.xls*, *.ppt*, *.png*, *.jpg*, etc. but to analyse and rebuild archive files such as *.zip* or *.7z*, Desktop *Pro* is required. ### Glasswall Meteor *Pro* * Glasswall Meteor *Pro* analyzes and protects up to 1,000 files in a 24-hour period, during your software subscription period. * If you require a software license to process more files each day, please contact the support team. Glasswall Meteor *Pro* can process file up to 1 GB in size. ## Glasswall Meteor version comparison The following table can be used to compare the features and capabilities offered by the 3 different versions of Glasswall Meteor. | Features | Essentials | Pro | Clean a file | | ------------------------------------ | ------------------------------------------------------ | ---------------------------------------------------------------- | ---------------------------------------------------------------- | | Maximum files/day | 150 | 1,000-10,000 | 10,000+ | | Supported file types | PDF, Word, Excel, PowerPoint, Images, Audio and Movies | PDF, Word, Excel, PowerPoint, Images, Audio, Movies and Archives | PDF, Word, Excel, PowerPoint, Images, Audio, Movies and Archives | | Adjustable content management policy | Yes | Yes | Yes | | True file type detection | Yes | Yes | Yes | | Audit trail folder | Yes | Yes | Yes | | Maximum file size | 25 MB | 1 GB | 100 MB | | Nested archive support | No | Yes | Yes | | Operating systems | Windows 10+ | Windows 10/11 & Windows Server 2019+ | Machine Image or Managed Kubernetes deployment | | Customer support | Standard | Standard | Standard | | Deployment type | Desktop | Desktop & Server | Server | [View supported file types](/docs/supported-file-types) To find out more, please [contact us](https://www.glasswall.com/contact/). Everything you need to install and begin using Glasswall Meteor. ## System requirements * **CPU**: 1 GHz or faster with 2 or more cores on a x84-64 (or x64) processor * **RAM**: 8 GB Recommended (4 GB minimum) * Windows 10 (64 bit) and Windows Server 2019 onwards * Earlier versions of Windows may operate, but are unsupported ## Microsoft Visual C++ Before you can begin installation of Glasswall Meteor, Microsoft Visual C++ runtime libraries need to be installed on your machine. When you execute the *Glasswall Meteor.exe* file, you will be prompted to install it first. The following steps will guide you through the quick and easy Glasswall Meteor installation process. ## Installation 1. From your file explorer, double-click the executable installation file provided to you by Glasswall.\ Read the license agreement and if in agreement, click **I Agree**. **Note:** You must accept the agreement in order to continue. 2. When the dialog box displays, select the type of installation required (all users or the current user) and click **Next >**. 3. Click **Install**. 4. If you are prompted to install Microsoft Visual C++, click **Yes**. 5. Ensure that *Run Glasswall Meteor* is selected and click **Finish**. Glasswall Meteor will launch automatically. To find out more about install paths and file storage, please refer to [Appendix A](/meteor/appendix-a-install-paths). #### Learn more * [License management](/meteor/glasswall-meteor-license-management) * [Supported file types](/docs/supported-file-types) * [Meteor Connect](/meteor/about-meteor-connect) ## Activate your license 1. From the Glasswall Meteor launch screen, click **+ Add license**. 2. From the *System settings* screen, click **Browse** to upload your license provided to you by Glasswall. 3. Once the license has been added, a pop up appears to inform you that the application will be relaunched. Click **OK**. ### Silent install It is also possible to silently install Glasswall Meteor via the following command: `"Glasswall Meteor Setup 3.2.4.exe" /LicenseFilePath="D:\path\to\license\gwkey.lic" /S` **Note:** If you choose to install Glasswall Meteor silently, only Admins will be able to change the license via the application. Glasswall Meteor processes files to ensure they are safe from threats. Once a file has been deconstructed and rebuilt, the application assigns a *Risk Level* to the original file. These risk levels explained below provide users with a high level summary of the level of threat the original file contained. | Risk Level | Description | |-----------|-------------| | | **High risk** — The original file contained software or executable code that may be used by attackers to compromise your environment. | | | **Medium risk** — The original file contained items that may be used by an attacker to compromise your environment or expose information about the file. Metadata such as review comments can hide information entering or leaving your environment. Hyperlinks may be benign but can lead users to insecure external locations. | | | **Low risk** — The original file did not appear to contain any threats or items that might increase risk for your environment. | ## How Glasswall made your file safe This section lists the risk features removed within the file structure, as well as risky content sanitized (removed) from your file according to the policies set by you or your company. There are various types of actions performed across all supported file types: * Unrecognized objects hidden within the file structure that are not defined in the ‘known good’ specification are removed. * Where possible, components of a file that deviate from the manufacturer’s specification are corrected back to the standards set in the specification. Otherwise, an issue is reported. For more information on what the different terms mean, please refer to our [definitions](/meteor/definitions). Glasswall Meteor can also be installed and uninstalled silently using Windows Command Line Interface (CLI). ## Installation Unattended installation is a procedure for installing software without user intervention. Example: *"Glasswall Meteor Setup.exe" /S /D="D:\My Files" --force-run* ``` /S :silent /D :install directory /allusers - Install for all users (requires command to be executed as Admin) /currentuser - Install for current user only (default) --force-run :to run directly after installation ``` ## Uninstallation For Windows, an uninstaller is provided in the preferred installation directory. From the example above, this would be found in D:\My Files. Please refer to Appendix A for default install paths. Example: *"D:\My Files\Uninstall Glasswall Meteor.exe” /S* ``` /S :silent ``` ## Predefined configuration System administrators may prefer to install the software with a configuration that limits users from making changes within the UI. A source JSON file can be referenced, and this will be copied to the ‘config’ folder under the ../AppData/Roaming/Glasswall Meteor path during the installation command. Example: "Glasswall Meteor Setup 3.0.0.exe" /ConfigPath="C:\config.json" A sample config file, that is inactive by default will be copied into the same directory by default and does not require a ‘ConfigPath’ location to be provided. This can serve as a template for an active config. Sample default ‘config.json’ file created in the config directory: ```json { "configActive": false, "readOnlyUI": true, "policyEditable": false, "defaultManualCleanFolder": "C:\\Users\\userabc\\Desktop\\clean", "auditFolder": "C:\\Users\\userabc\\Documents\\audit", "Syncs": { "syncId1": { "name": "examplesync1", "sourceFolder": "C:\\Users\\userabc\\Documents\\examplesource1", "destFolder": "C:\\Users\\userabc\\Documents\\exampledest1" }, "syncId2": { "name": "examplesync2", "sourceFolder": "C:\\Users\\userabc\\Documents\\examplesource2", "destFolder": "C:\\Users\\userabc\\Documents\\exampledest2" }, "syncId3": { "name": "examplesync3", "sourceFolder": "C:\\Users\\userabc\\Documents\\examplesource3", "destFolder": "C:\\Users\\userabc\\Documents\\exampledest3" }, "syncId4": { "name": "examplesync4", "sourceFolder": "C:\\Users\\userabc\\Documents\\examplesource4", "destFolder": "C:\\Users\\userabc\\Documents\\exampledest4" }, "syncId5": { "name": "examplesync5", "sourceFolder": "C:\\Users\\userabc\\Documents\\examplesource5", "destFolder": "C:\\Users\\userabc\\Documents\\exampledest5" } } } ``` Explanation of example: * The configuration is currently inactive, and therefore will be ignored on startup. * The user interface is read-only, meaning that file paths cannot be modified. * The content management policy is not editable. * The default manual clean folder is located at "C:\Users\userabc\Desktop\clean". * The audit folder is located at "C:\Users\userabc\Documents\audit". * There are five syncs defined, each with a unique sync ID. * Each sync has a name, a source folder, and a destination folder. * The source and destination folders are specified using file paths on the user's computer. The Archive Support functionality ensures users can safely access archive files. ## **Features** * Process archive files containing a nesting limit of up to 5 archive levels. * Archive files exceeding 5 nested levels will result in a failure message. * Archive files containing unlimited unarchived nested level folders and files are supported (within limitations of your file counter). * Once an archive file is processed, the risk level is displayed within Glasswall Meteor for each individual file, however your designated Clean Folder will only contain the clean archived file. * You will need to extract the individual cleaned files manually using a file extracting software. ### ***Note*** * Encrypted or Password protected archive files are **not** supported. * These won’t be processed, resulting in a failure message. * Archive files containing any individual file(s) above the standard supported size (1024 MB) are **not** supported. * These won’t be processed, resulting in a failure message. * The file counter will decrease based on the total number of files contained inside the processed archive file. Glasswall Meteor can also interact with files on removable media such as a USB drive or memory card. Although the application can ensure unknown files are inspected for malware, hidden software programs could also be present, and may get triggered when the media connects to a network machine. This is why it is important that an endpoint protection solution such as an antivirus program is always used in conjunction with the Glasswall Meteor, so that any unauthorized software execution is blocked. If endpoint protection needs to be disabled for any reason, please ensure that all removable media is disconnected from your machine. This option can be used to clean and rebuild files offline, directly from your machine without sending the file outside of your trust boundary. 1. From the Welcome screen or side navigation menu, select **Clean files now**. 2. In the window, you can input files in two ways: * **Drag & drop** – drop a file directly into the blue drop zone. * **File input** – click **select files** to select a file from your device. .png) The file is instantly cleaned, rebuilt, and saved in the specified folder. **Note:** You can change the location of your saved files by selecting **Change clean files location**. 3. After your file has been processed and rebuilt, the following items are available for your reference: * **Time started** - the time you initiated the file processing. * **Original file** - lists the original file name. * **Clean file** - a link that allows you to choose where the cleaned file should be saved. * **Original risk status** - the risk status of the original file. * **Current Protection status** - confirmation that your file was successfully rebuilt. * **Analysis** - a detailed analysis and breakdown of the changes made to your original file during the sanitization process. 4. Click **View details** to see a detailed report of the changes made to the original file. The following content is included in the **File analysis** report: - Processed File information - Details about what has been removed from your file - Structural repair that occurred - Redactions that have occurred - Original file information - Foresight threat prediction (see [Foresight documentation](/foresight/about-glasswall-foresight)) - Risk analysis of the original file - Structural analysis of the original file - Find and redact information - File Details - Detected file type and Original extension - Original and Processed file size - File hashes **Note:** if you have set custom policy settings allowing certain types of risky content, these will be flagged as allowed in the processed file information. With Glasswall Meteor's *Sync and clean* feature, you're able to setup folder synchronization's (a maximum of 20) which automatically perform CDR on any new files added to the folder, after which the sanitized files are added to a designated 'clean' folder. Before syncing a folder, you can run it through the [Clean files now](/meteor/glasswall-meteor-clean-files-now) feature to ensure all existing files have been cleaned first. ## Sync a new folder 1. From the Welcome screen or side navigation bar, select **Sync and clean**.\ 2. Click **Add a new folder**. You will arrive at the *Set up a new folder to Sync* page.\ 3. Enter a **Sync name**. This should help you easily identify the folder and its contents. 4. From the **Original folder** field, click **Browse** and select the desired folder. 5. From the **Clean folder** field, click **Browse** and select the desired folder. 6. Click **Save & start sync**. The folder and its status are now displayed under the *Your clean synced folders* page.\ 7. Click **View results** to see a complete list of the cleaned files within the synced folder, showing the total number of files cleaned, their status, and original file risk level. When using Glasswall Meteor, you’re able to set your own preferences or ‘policies’ for each of the supported file formats via the *Policy settings* section. Policy settings are used to specify whether risky content is sanitized (removed from the file), allowed (left in the file) or disallowed (the file is not rebuilt). * From the side navigation bar, select **Policy settings**. The numerous settings for each file format are displayed. * Modify as required and click **Save**. As a default, we recommend setting all content management settings to *Sanitize*. This will provide the best balance between security and usability. Allowing certain types of risky content presents a risk to you if an attacker has placed malware within a file. You can choose to disallow files with certain criteria, but this will block CDR from being performed on those files. For these files, the rebuild process status will be marked as *Can't be cleaned*. Glasswall Meteor’s *Find and redact* tool is a data loss prevention feature which allows users to set customized rules to identify and remove sensitive information in Office documents. **Note:** This feature is currently only available in ‘Preview’ mode. 1. From the Welcome screen or from the side navigation menu, select **Find and redact**. 2. Click **Add new rule**. You will arrive at the *Create your rule* screen. 3. Select the appropriate tab for the custom rule you would like to create. For every rule, the following actions can be assigned: * **Find** - identify files containing sensitive information. * **Redact** – obscure or remove sensitive information. * **Block** – stop a file from moving into your clean files. 4. Enter a name for your new rule in the **Name your rule** field. 5. Add your regular expression code in the corresponding field. **Note:** Currently the regular expression matching is case insensitive. 6. If you selected the *Redact* tab, enter the appropriate character in the **Replace your character** field. 7. Click **Save rule**. Once a rule is created, it will be applied to all files processed via [Clean files now](/meteor/glasswall-meteor-clean-files-now) or [Sync and clean](/meteor/glasswall-meteor-sync-and-clean). If you would like to view previously processed files, you can do so via the **Job history** section. * From the side navigation bar, select **Job history**. * A list of processed files is displayed for your reference. From the Help window, you can access our online and offline Glasswall Meteor documentation as well as contact our Customer Support team and inquire about upgrading. PDF processing is designed to preprocess PDF files to meet standards for further processing. It extracts and reconstructs visual content so your PDF can be processed by Meteor's Content Disarm and Reconstruction (CDR) protection. Only malformed or corrupt PDFs will be regenerated. ## Misbehaving PDF Settings Under the system settings you can change the behavior of PDF processing. ### Processing You can choose to turn processing on or off; it’s included and enabled by default in Meteor 3.6.0 and higher. The purpose of process is to fix malformed or corrupt PDFs so if you don't select it, these files will remain unable to be processed by Meteor. Regenerated files can sometimes show some visual differences to the original, such as using a different font, and can have a larger file size. ### Watermark PDF processing allows you to add a watermark to regenerated PDFs so the reader knows the document has been regenerated, this can be turned on or off independent of regeneration. The default watermark text is 'Processed for security: Visual elements may vary' but this can be configured by the user up to a 256-character limit. ## Accessing cloud and remote storage with Glasswall Meteor Connect Glasswall Meteor Connect combines with Glasswall Meteor to ensure files that are stored in remote locations can benefit from Glasswall’s industry leading Zero Trust Content Disarm and Reconstruction protection. File-based threats are removed by Glasswall Meteor, with clean files being automatically transferred to a safe target location.You can also seamlessly connect with 200+ cloud and network providers and enhance Glasswall Meteor’s remote storage connectivity options and enable secure file transfer within your current cloud and network environments. ## Features ### **Seamless connectivity across storage platforms** Glasswall Meteor Connect, effortlessly bridges the gap between your local files and remote servers. With its comprehensive support for all major server and cloud storage protocols, you're empowered to connect to virtually any server or cloud service, including Amazon S3, Google Cloud Storage, Microsoft OneDrive, and more. ### **Optimized file transfers and management** Experience hassle-free file transfers to accounts like Amazon S3 and Google Storage. Organize and browse through your files in a structured, hierarchical manner, and effortlessly manage access control lists and bucket configurations. ### **Secure and reliable access** Glasswall Meteor Connect prioritizes your data's security. Benefit from secure TLS connections for FTP transfers, advanced SSH configurations with public key authentication, and the ability to connect to highly secure enterprise filesharing solutions. ### **Efficient cloud storage solutions** Discover cost-effective cloud storage with services like Backblaze B2, offering storage solutions at a fraction of the cost of Amazon S3. Additionally, connect to smart cloud storage services tailored for modern teams, such as Files.com. ### **Streamlined setup with connection profiles** Simplify your setup process with Glasswall Meteor Connect's connection profiles. These plugins describe specific connection settings for various hosting providers, making it easier than ever to establish connections without the need to manually enter connection details. ### **Universal access and collaboration** Whether it's accessing Windows file shares via SMB, managing unstructured data with Windows Azure, or collaborating securely from anywhere with Box.com, Glasswall Meteor Connect ensures you're always connected and in control.Glasswall Meteor Connect is provided to Glasswall Meteor customers without charge. The software is powered by Mountain Duck, an Iterate GmbH solution under license to Glasswall Solutions Ltd. ## Why Glasswall Meteor Connect? * Zero cost for each Glasswall Meteor user subscription * Supports Windows 10 or later & Windows server 2016 or later * Wide range of notifications for file and folder actions * Throttle upload and download speed * Proxy aware software * Support for wide range of languages Everything you need to install and begin using Glasswall Meteor Connect. * **CPU**: 1 GHz or faster with 2 or more cores on a x84-64 (or x64) processor * **RAM**: 8 GB recommended (4 GB minimum) * Windows 10 (64 bit) and Windows server 2016 onwards * Earlier versions of Windows may operate, but are unsupported ## Install Glasswall Meteor Connect The following steps will guide you through the quick and easy Glasswall Meteor Connect installation process. ## Installation 1. From your file explorer, double-click the executable installation file provided to you by Glasswall. 2. Click **Install** to proceed. **Note:** you can change the default location by clicking **Options**. 3. When the dialog box displays, select the type of installation required (all users or the current user) and click **Next >**. 4. Save any unsaved work and then click **Restart**. This reboots your PC and the software installation will be completed. An icon representing Glasswall Meteor will appear in the Taskbar. You may need to click the *Show hidden icons* arrow to view this. Clicking the Glasswall Meteor Connect icon will reveal a menu of options. This confirms that the software is installed and ready to be configured. For extended installation guidance (based on Mountain Duck core behaviour), see [Mountain Duck installation documentation](https://docs.mountainduck.io/mountainduck/installation/). Glasswall Meteor Connect is built on the Mountain Duck platform. Many underlying behaviors and advanced capabilities are identical, so this page provides curated links to the authoritative Mountain Duck documentation for deeper guidance. - - - ## Connecting to storage For instructions on adding, configuring, and managing remote connections, see the [Mountain Duck connections guide](https://docs.mountainduck.io/mountainduck/#connections). - - - ## Working with files Learn how to open, edit, and save files using mounted storage in the [Working with files section](https://docs.mountainduck.io/mountainduck/#working-with-files). - - - ## File sync and caching For details on sync operations, caching, and offline behavior, refer to the\ [Sync and caching documentation](https://docs.mountainduck.io/mountainduck/#sync). - - - ## Storage protocols Meteor Connect supports the same storage protocol backends as Mountain Duck.\ Use the protocol-specific links below for configuration details, behaviors, and limitations. * [S3-Compatible Storage](https://docs.mountainduck.io/protocols/s3/) * [Azure Blob Storage](https://docs.mountainduck.io/protocols/azure/) * [WebDAV](https://docs.mountainduck.io/protocols/webdav/) * [FTP / FTPS](https://docs.mountainduck.io/protocols/ftp/) * [OpenStack Swift](https://docs.mountainduck.io/protocols/openstack/) * [Backblaze B2](https://docs.mountainduck.io/protocols/b2/) * [Google Cloud Storage](https://docs.mountainduck.io/protocols/googlecloudstorage/) **Note:** Your organization may restrict Meteor Connect to a subset of these protocols.) - - - ## Connect mode For organizations using **Connect Mode**, the following Mountain Duck resources provide the full behavior overview: * [Connect Mode overview](https://docs.mountainduck.io/mountainduck/connect/) - - - ## Encryption and security Information on encrypted vaults and related functionality is available in the [Encryption and vault documentation](https://docs.mountainduck.io/mountainduck/cryptomator/). - - - ## Troubleshooting Guidance for diagnostic steps, connectivity issues, and drive-mounting failures is available in the [Troubleshooting section](https://docs.mountainduck.io/mountainduck/#troubleshooting). - - - ## Glasswall Support For configuration or operational details unique to Glasswall deployments, contact your administrator or Glasswall Support . ## Fixes and improvements * Custom icons for protocols (nextcloud, owncloud) * Detect proxy configuration change for already open connections * Support authentication using ssh certificates (sftp) * Display notification when sync is paused in "smart synchronization" connect mode * Crash when disconnecting in "online" connect mode with no network connectivity (Windows) * Failure retrieving previous versions of file (owncloud) * Missing shared folders (OneDrive business) * Unable to dismiss share prompt window (macOS) * Show prompt to disconnect volume when offline in "online" connect mode * Slow saving files in "online" connect mode * Failure sharing file (owncloud) * Bugfix ntlm authentication failures (webdav, Sharepoint) Glasswall Meteor Connect is provided to Glasswall Meteor subscribers without charge. The software is powered by Mountain Duck, an Iterate GmbH solution under license to Glasswall Solutions Ltd. ## Fixes and improvements * Always pause synchronization when server is not reachable in "smart synchronization" connect mode * No repeated attempts connecting when server is not reachable in "online" connect mode * Missing digest header when committing large file upload (box) * Modification date not set in upload (dropbox) * Setting modification date truncates file (smb) * Select previous versions in info → versions (backblaze b2) * Failure renaming or moving files (google storage) * Delayed display of folder contents Glasswall Meteor Connect is provided to Glasswall Meteor subscribers without charge. The software is powered by Mountain Duck, an Iterate GmbH solution under license to Glasswall Solutions Ltd. ## Fixes and improvements * Always pause synchronization when server is not reachable in "smart synchronization" connect mode * No repeated attempts connecting when server is not reachable in "online" connect mode * Missing digest header when committing large file upload (box) * Modification date not set in upload (dropbox) * Setting modification date truncates file (smb) * Select previous versions in info → versions (backblaze b2) * Failure renaming or moving files (google storage) * Delayed display of folder contents Glasswall Meteor Connect is provided to Glasswall Meteor subscribers without charge. The software is powered by Mountain Duck, an Iterate GmbH solution under license to Glasswall Solutions Ltd. ## Fixes and improvements - Bugfix - Failure parsing expiration from cached temporary credentials in AWS CLI configuration from AWS SSO (S3) - Bugfix -Unable to authenticate with S3 (Credentials from AWS Command Line Interface) connection profile (S3) - Bugfix - Reduce load indexing files in background in "Smart Synchronization" connect mode (S3, Google Storage, OpenStack Swift, Backblaze B2) - Bugfix - Refreshed OAuth tokens not saved in credentials manager (Dropbox, Google Drive, Google Storage, Microsoft OneDrive, SharePoint) - Bugfix - No prompt to choose sharee when sharing file (ownCloud, Nextcloud) (Windows) Glasswall Meteor Connect is provided to Glasswall Meteor subscribers without charge. The software is powered by Mountain Duck, an Iterate GmbH solution under licence to Glasswall Solutions Ltd. Glasswall Meteor allows users to set up automatic folder synchronizations with various cloud drives. The steps outlined below assume that you intend to install the storage connection software from one of the four providers, illustrated in this section. Glasswall Meteor Connect provides users with the ability to create storage connections without the need to install additional 3rd party software from Box, Dropbox, Google Drive or OneDrive. If you would like to benefit from this option, please [read about Meteor Connect](/meteor/about-meteor-connect). ## Sync cloud folders Select your cloud drive to follow the step by step guide: * [Box](/meteor/sync-meteor-with-box) * [Dropbox](/meteor/sync-meteor-with-dropbox) * [Google Drive](/meteor/sync-meteor-with-google-drive) * [OneDrive](/meteor/sync-meteor-with-one-drive) ## Edit or delete folders You're also able to edit or delete any actively synced folders displayed in the *Sync and clean* section. On the far right of the folder listing, click the vertical ellipsis icon to view a dropdown list and make changes as required. Glasswall Meteor allows users to set up automatic folder synchronizations with various cloud drives. Follow these steps to sync Glasswall Meteor with folders in your *Box* drive: 1. Download and install the [Box Drive](https://www.box.com/en-gb/resources/downloads) desktop app. 2. Login using your account details. 3. From your File Explorer, navigate to your *Box* drive. 4. Identify the folder you would like to sync, and create a new folder for your clean files. .png) .png) 3. Open Glasswall Meteor and navigate to the *Sync and clean* page. 4. Click **Add a new folder**. 5. Enter a **Sync name**. 6. From the *Original folder* field, click **Browse** and select the folder you would like to sync in your *Box* drive. 7. From the *Clean folder* field, click **Browse** and select the folder you created earlier to store your clean files. **** 8. Click **Save & start sync**, and a confirmation popup will appear. .png) 9. Click **Ok**. The folder now appears in the list of synced folders. Once the folder has finished syncing, the *Status* will display as **Synced**. Glasswall Meteor allows users to set up automatic folder synchronizations with various cloud drives. Follow these steps to sync Glasswall Meteor with folders in your *Dropbox* drive: 1. Download and install the [Dropbox Desktop](https://www.dropbox.com/desktop) app. 2. Login using your account details. 3. From your File Explorer, navigate to your *Dropbox* drive. 4. Identify the folder you would like to sync, and create a new folder for your clean files. 5. Open Glasswall Meteor and navigate to the *Sync and clean* page. 6. Click **Add a new folder**. 7. Enter a **Sync name**. 8. From the *Original folder* field, click **Browse** and select the folder you would like to sync in your *Dropbox* drive. 9. From the *Clean folder* field, click **Browse** and select the folder you created earlier to store your clean files. 10. Click **Save & start sync**, and a confirmation popup will appear. .png) 11. Click **OK**. The folder now appears in the list of synced folders. Once the folder has finished syncing, the *Status* will display as **Synced**. Glasswall Meteor allows users to set up automatic folder synchronizations with various cloud drives. Follow these steps to sync Glasswall Meteor with folders in your *Google Drive*: 1. Download and install the [Google Drive Desktop](https://www.google.com/intl/en-GB/drive/download/) app. 2. Login using your account details. 3. From your File Explorer, navigate to *Google Drive > My Drive*. 4. Identify the folder you would like to sync, and create a new folder for your clean files. 5. Open Glasswall Meteor and navigate to the *Sync and clean* page. 6. Click **Add a new folder**. 7. Enter a **Sync name**. 8. From the *Original folder* field, click **Browse** and select the folder you would like to sync in your *Google Drive*. 9. From the *Clean folder* field, click **Browse** and select the folder you created earlier to store your clean files. 10. Click **Save & start sync**, and a confirmation popup will appear. .png) 11. Click **OK**. The folder now appears in the list of synced folders. Once the folder has finished syncing, the *Status* will display as **Synced**. Glasswall Meteor allows users to set up automatic folder synchronizations with various cloud drives.Follow these steps to sync Glasswall Meteor with folders in your OneDrive: 1. Download and install the [OneDrive](https://www.microsoft.com/en-gb/microsoft-365/onedrive/download) desktop app. **Note:** if you're using Windows 10, OneDrive is already installed on your PC. 2. Login using your account details, if applicable. 3. From your File Explorer, navigate to your OneDrive. 4. Identify the folder you would like to sync, and create a new folder for your clean files. 5. Launch Glasswall Meteor and navigate to the *Sync and clean* page. 6. Click **Add a new folder**. 7. Enter a **Sync name**. 8. From the *Original folder* field, click **Browse** and select the folder you would like to sync in your OneDrive. 9. From the *Clean folder* field, click **Browse** and select the folder you created earlier to store your clean files. 10. Click **Save & start sync**, and a confirmation popup will appear. .png) 11. Click **OK**. The folder now appears in the list of synced folders. Once the folder has finished syncing, the *Status* will display as **Synced**. ## Software path The Glasswall Meteor installation process places a shortcut to your desktop. If the ‘*all users*’ option was selected, the shortcut will point to the following software installation path: * C:\Program Files (x86)\Glasswall Meteor\Glasswall Meteor.exe If the single user option was selected, the shortcut will point to the following software installation path: * C:\Users\[username]\AppData\Local\Programs\glasswall-meteor\Glasswall Meteor.exe ## Working directory path Glasswall Meteor creates a working data folder at the following location on each individual machine: * C:\Users\[username]\AppData\Roaming\glasswall-meteor ## Clean file path You can change where your files are stored once they have been cleaned, however the default location is set as: * C:\Users\[username]\AppData\Roaming\glasswall-meteor\clean **Note**: It is important to keep clean files separate from original source files, which could contain malware. ## Audit folder path The Glasswall Meteor retains a report of all actions performed, so that users can analyze what risky items or broken document structures were found in the source file. The audit folder location can be found at: * C:\Users\[username]\AppData\Roaming\glasswall-meteor\audit_folder ## Folders Each processing job (which can include one or more files) will have a new folder created with an eight-character name. For each file processed as part of the job, a folder is created using a universally unique identifier (UUID) to record the artefacts that are created as part of the CDR process. Four sub-folders are created under each UUID parent folder. 1. **Report** – This contains the master analysis XML file, which is the most detailed report about what was found in the original file. 2. **Analysed** – This contains a digest of information about sanitisation and remediation items that were found in the original file. The details are contained in json files within the folder. Where sanitisation occurs, potentially risky content has been removed. Remediation relates to deviations from the industry specifications for the file format, and how they have been repaired. 3. **Original** – This contains the original file that was provided to the Glasswall Meteor. Caution should be exercised as this file has not been protected by CDR. The document is preserved to protect the audit trail. 4. **Clean** – This contains an additional copy of the clean file, which has undergone CDR and is therefore safe to interact with. Additional files (config.*) relate to the content management policy and the settings provided to the Glasswall engine during the CDR process for the file. ### Acroforms An 'Acrobat Form' in addition to looking like a form, may also contain active code (e.g., JavaScript) which could be malicious. They can also be used to hide objects inside other objects. ### Actions all An action within a PDF may be benign, but it’s designed to make the document dynamic. An attacker may use the action to trigger active code (e.g., JavaScript) or send data to a URL. The functionality can be misused to cause harm to the recipient. ### Allow The file is analysed and rebuilt with the defined risky content left intact (the object is not removed). Eg. you can configure Word so that macros are allowed (not removed) from Word files. ### Digital signatures The source document may have been signed with a digital signature. Whilst the signing may not represent a threat, if the ownership and trust of the certificate chain has been compromised, this could trick a user into viewing a document that could contain something malicious. The sanitise setting is a good option to select if there is any doubt about the provenance of the document. ### Disallow The file is analysed but not rebuilt if the defined risky content is found during analysis. Eg. you can specify that macros are disallowed from Word files. Word files are analysed and if macros are found the file is not rebuilt and is not made available to you. ### Dynamic data exchange Dynamic data exchange within Microsoft documents is known to present risk as the protocol may be used to execute malicious code on the recipient's computer. ### Embedded files Embedded objects within files may present risk if they provide a way for active code to be triggered, or to hide data within a document. ### Embedded images Embedded images within files may present risk if they provide a way for malicious content to be hidden inside the image. ### External hyperlinks / Internal hyperlinks External and internal hyperlinks may appear innocent. However, a link in a document may appear to have a different destination than the real link. Caution is advised when clicking on links in documents. ### Macros & Javascript Macros & JavaScript are forms of active code, which may be benign in nature, but all too often are used by bad actors to mount an attack against the user or receiving system when expressed in a business document. ### Review Comments & Metadata Metadata can reveal information which the owner may not intend to disclose to the recipient, such as review comments, or the original author's name. ### Sanitise The file is analysed and rebuilt with the defined risky content removed. Eg. you can specify that macros are sanitised (removed) from Word files. ## Refund policy for Glasswall Meteor Thank you for choosing Glasswall Meteor. We are committed to ensuring your satisfaction with our product. Please review our refund policy below. ### 1. Refund eligibility * **Prorated refund**: If you cancel your subscription, you may be eligible for a prorated refund for any remaining full months of service, excluding the current month. * **Refund conditions**: Refunds will be issued only if the software was used in a supported environment according to our documentation and under the following circumstances: * The software did not perform as described. * Essential features were unavailable, and our support team was unable to resolve the issue within a reasonable timeframe. ### 2. Non-refundable circumstances * Violation of Terms: Refunds will not be granted if the software's terms of use were violated or if the issue resulted from unsupported configurations or environments. ### 3. How to request a refund To request a refund, please follow these steps: * **Contact support**: Reach out to our support team at support@glasswall.com or through the AWS Marketplace refund request form. * **Provide details**: Include a detailed explanation of the issue and any relevant correspondence with our support team. * **Review process**: We will review your request within 7-14 business days and notify you of our decision. If approved, refunds will be processed within an additional 7 days. ### 4. Contact information For questions or concerns regarding our refund policy, please contact our support team at support@glasswall.com or visit our website at [www.glasswall.com/support](https://www.glasswall.com/support). import ThemedImage from '@theme/ThemedImage'; import useBaseUrl from '@docusaurus/useBaseUrl'; Zero Trust file protection delivered by the Glasswall Embedded Engine is different. Instead of looking for malicious content, our advanced CDR (Content Disarm and Reconstruction) process treats all files as untrusted, validating, rebuilding and cleaning each one against their manufacturers ‘known-good’ specification. The Glasswall Embedded Engine allows organizations to embed the power of Glasswall CDR, via a software development kit (SDK) which includes an API, with new and existing application deployments. Once integrated, any files the engine encounters are processed in accordance with user-configurable security preferences that allow security teams to shape their security policy to match their organization's risk appetite. Glasswall's Embedded Engine can: * validate, clean (sanitize and remediate), and reconstruct files. * produce XML reports detailing deep analysis of files. * manage files via a configurable content management policy. * carry out reliable file type identification. * perform search and redaction of text in MS Office file types. * create and manipulate a document object model (DOM) for supported file types. ## Features ### Analysis mode In Analysis Mode, the input file is decomposed and an XML report is generated for each file. The analysis report contains 'Content' Items (structures found in the file), 'Issue' Items (structures that do not match the specification), 'Sanitisation' Items (items that would be taken out as per the content management policy) and 'Remedy' Items (structures that would be automatically corrected back to specification). ### Protect mode In Protect Mode, the input file is decomposed as per Analysis Mode but the file is then regenerated using valid Content Items and remediated Content Items. As part of this process, the Content Management rules are applied so that certain 'Sanitisation' and 'Remedy' Items are removed (e.g., macros) from the regenerated file, so only known good items are present in the new file. ### Export In Export Mode, the input file is broken down to its structural components and an internal DOM (Document Object Model) tree is created. The DOM tree is validated and cleaned before serialising and writing out as one of two configurable intermediate file formats (XML or SISL). The exported content can be further processed and modified externally prior to importing. ### Import In Import Mode, the input is the exported SISL or XML files (with or without external modification). The XML or SISL files are deserialised, reconstructing the internal DOM tree, which is validated, cleaned and used to regenerate a clean and compliant file. Content Management rules can be configured and applied in both Export and Import modes. ### Automatic file corrections Automatic corrections back to the file specification are performed upon file regeneration. The purpose of this is to enable the Glasswall Embedded Engine to remove threats that are hidden within the file structure, as well as preventing the possibility of activating exploits via the misuse of structural components in the file. In Analysis Mode, where automatic corrections are applied to a file, these are reported as remedy items in the analysis report. These items are reported irrespective of whether a regenerated out file is required ### Configuration management Content management policies are a set of content management switches that can be applied to a particular file type. The content management switch is used to identify a file element type and associated action. The content management setting specifies the action to be carried out by Glasswall for a particular content management switch. Each content management switch can be set to one of three settings. ### Issue IDs Reported content items and associated actions carried out by the Glasswall engine are logged in an XML analysis report. Each unique content item entry has a unique issue ID associated to it. Issue IDs are generated when files are processed in Analysis Mode and can be found in the output analysis report. Glasswall provides API functions responsible for retrieving information related to issue IDs. ### Word Search & Redaction The Glasswall Embedded Engine can search and redact the text and metadata of a file for forbidden words in certain file types. [View supported file types](/docs/supported-file-types) ## System requirements The Glasswall Embedded Engine supports a wide range of architectures and operating systems, operating in two distinct modes: **Protect** and **Export or Import**. ### General performance notes Resource usage is driven more by the selected operational mode and file complexity than by the underlying system architecture. While performance for Export and Import is generally comparable. For more information see the [Engine Performance Data](/embedded-engine/embedded-engine-performance-data) ### Memory requirements | Mode | Minimum RAM | Recommended RAM | |----------------------|-------------|-----------------| | **Protect** | 4 GB | 8 GB | | **Export or Import** | 8 GB | 16 GB | Memory consumption depends on how complex a file is, based on the number and type of elements the engine evaluates. For example, a small but complex Excel workbook with many sheets may consume more memory than a large but simple image file. File complexity scores are included in the engine’s analysis report. In most cases, real files score no higher than **7**, with a maximum value of **10**. If available memory is too low, processing may fail. ### Processing time The Glasswall Embedded Engine is designed to process files using a single thread. To make use of multiple CPU cores, run multiple instances of the engine. Best performance is achieved on modern processors. Export mode usually requires significantly more time to process files than Protect mode. ## Supported operating systems ### Linux The Embedded Engine supports the following Linux distributions: | OS | x86-64 | ARM64 | | -------------- | :--------------------: | :--------------------: | | RHEL 9.x | ✔ | ✔ | | RHEL 8.x | ✔ | ✔ | | Rocky Linux 9.x | ✔ | ✔ | | Rocky Linux 8.x | ✔ | ✔ | | Oracle 9.x | ✔ | ✔ | | Oracle 8.x | ✔ | ✔ | | Ubuntu 23.x | ✔ | ✔ | | Ubuntu 22.x | ✔ | ✔ | | Ubuntu 20.x | ✔ | ✔ | ### Windows The Embedded Engine supports the following Windows operating systems: | OS | x86-64 | ARM64 | | ------------------- | :--------------------: | :--------------------: | | Windows 11 | ✔ | ✘ | | Windows 10 | ✔ | ✘ | | Windows Server 2022 | ✔ | ✘ | | Windows Server 2019 | ✔ | ✘ | | Windows Server 2018 | ✔ | ✘ | | Windows Server 2016 | ✔ | ✘ | ### StopOS The Embedded Engine supports the following StopOS versions: | OS | x86-64 | ARM64 | | ------------------- | :--------------------: | :--------------------: | | StopOS 8.11.0 | ✔ | ✘ | ## Overview Since Release 5 of the Glasswall Embedded Engine, a valid licence key is required to invoke embedded engine processing modes. Licence keys are generated on a per customer basis. Please contact support@glasswall.com should your require a licence key for evaluation or production use. The licence key file 'gwkey.lic' can be placed into the same working directory to where the engine binaries are located. If you wish to load the licence key from an alternative path, the following optional APIs may be used to assign a licence key to a session. ### Registering a licence key - [GW2RegisterLicenceFile](/embedded-engine/embedded-engine-gw2registerlicencefile) - [GW2RegisterLicenceMemory](/embedded-engine/embedded-engine-gw2registerlicencememory) ### Licence policies Embedded Engine licence keys include a number of 'entitlements'. The presence or absence of entitlements enables or restricts the use of Embedded Engine features. Entitlements are bundled into the following overarching policies: | Entitlement code | Feature description | Standard CDR | Transformation | |-----------------------|:--------------------|:-------------:|:--------------:| | Engine_Main |- | ✔ | ✔ | | Engine_Protect |[Protect](/embedded-engine/embedded-engine-protect-mode) | ✔ | ✔ | | Engine_Analysis |[Analysis](/embedded-engine/embedded-engine-reporting)| ✔ | ✔ | | Engine_ExportImport |[Export / Import](/embedded-engine/embedded-engine-content-export-and-import)| ✘ | ✔ | **Note:** Engine_Main is an Embedded Engine mandatory entitlement. ### Retrieving licence information - [GW2LicenceDetails](/embedded-engine/embedded-engine-gw2licencedetails) #### Example licence information ``` Days Remaining: 212 Entitlements: Engine_ExportImport Engine_Analysis Engine_Protect Engine_Main ``` Content Management policies are a set of content management switches that can be applied to a particular file type. The content management switch is used to identify a file element type and associated action. The content management **setting** specifies the action to be carried out by Glasswall for a particular content management switch. Each content management switch can be set to one of three settings: - **Allow** - The Glasswall Embedded Engine processes any associated file element types and they remain in the regenerated file. The associated structure is logged in the **Analysis** report as an **Allowed Item**. - **Disallow** - If any of the associated file element types are identified in the file, the Glasswall Embedded Engine identifies the file as being non-conforming and the file will not be regenerated. The associated structure is logged in the **Analysis** report as an **Issue Item**. - **Sanitise** - If any of the associated file element types are identified in the file, the Glasswall Embedded Engine removes them from the regenerated document. The associated structure is logged in the **Analysis** report as a **Sanitisation Item**. ### Content management reporting The following sections show how content that is under the control of a content management switch is presented in the XML Analysis report, depending on the content switch setting. #### **Allow** This is an excerpt from the XML report for a Word (.doc) Binary file, which contains metadata. The content management switch `metadata` has been set to *allow*. ```xml metadata allow ... Metadata detected in #05SummaryInformation 1 4096 ``` #### **Disallow** This is an excerpt from the XML report for a Word (.doc) Binary file which has metadata inside it. The content management switch `metadata` has been set to *disallow*. In Protect Mode, this would cause the file to be marked as non-conforming. ```xml metadata disallow ... Metadata detected in #05SummaryInformation 96 1 Medium ``` #### **Sanitise** This is an excerpt from the XML report for a Word (.doc) Binary file which has metadata inside it. The content management switch `metadata` has been set to *sanitise*. In Protect Mode, this would result in the metadata being removed from the regenerated file. ```xml metadata sanitise ... Metadata detected in #05SummaryInformation 1 4096 ``` ### Content management policies These are the available content management policies: | Content Management Switch | Description | | ------------------------- | ------------------------------------------------------------ | | pdfConfig | Content management switch for PDF file type | | wordConfig | Content management switch for Word file type | | pptConfig | Content management switch for PowerPoint file type | | xlsConfig | Content management switch for Excel file type | | tiffConfig | Content management switch for TIFF file type | | svgConfig | Content management switch for SVG file type | | webpConfig | Content management switch for WebP file type | | gifConfig | Content management switch for GIF file type | | jpegConfig | Content management switch for JPEG file type | | sysConfig | Content management switch to control different Engine settings | **Note:** The `xlsConfig`, `pptConfig` and `wordConfig` content management policies cover both Office Open XML and Office Binary file types. The available content management switches and applicable file types are shown in the table below: | Content Management Switch | Description | | ------------------------------- | ------------------------------------------------------------ | | acroform | Controls Interactive form (AcroForm) content | | javascript | Controls JavaScript code embedded in files | | external_hyperlinks | Controls hyperlinks to locations outside the file | | embedded_files | Controls Embedded file content | | metadata | Controls file metadata | | actions_all | Controls PDF Actions such as Rendition, Sound, Movie, Hide, SetOCGState, GoTo3DView | | internal_hyperlinks | Controls hyperlinks to locations within the file | | value_outside_reasonable_limits | Controls Glasswall defined restrictions such as values exceeding a reasonable range e.g. object sizes | | digital_signatures | Controls digital signature content for signed files or signed objects within files. NOTE: the 'allow' setting will preserve the signature, but it will no longer be a valid signature, as the document will have been edited by the CDR process. | | macros | Controls VBA Macros which use Visual Basic code to create custom user-generated functions | | review_comments | Controls document review comments within a file | | embedded_images | Controls embedded image content for the Glasswall supported image formats | | dynamic_data_exchange | Controls DDE commands and DDE content in documents | | tracked_changes | Controls tracked changes in documents | | hidden_data | Controls hidden data in documents | | in_text_comments | Controls in text comments in documents | | slide_notes | Controls slide notes in documents | | connections | Controls connections to external data sources and information for constructs such as OLAP formulas, QueryTables or PivotTables | | scripts | Controls XML Scripts that allow for the creation, storage and manipulation of variables and data during processing | | foreign_objects | Controls embedded objects in XML based formats such as SVG | | hyperlinks | Controls external and internal hyperlinks | | geotiff | Controls georeferencing information embedded within a TIFF file | | jfif | Controls JFIF marker segments within a JPEG image file | | undefined_type | Controls TIFF IFD segments of undefined type | | doc_variables | Controls document variables in DOCX | | web_video_extension | Controls use of Web Video Extensions in DOCX | | headers_footers | Controls Header and Footer content in DOCX and XLSX | | scenarios | Controls Scenarios content in XLSX | | footnotes_endnotes | Controls Footnotes and Endnotes content in DOCX | | field_codes | Controls field codes in DOCX | | embedded_fonts | Controls embedded fonts in DOCX and PPTX | | write_reservation | Controls write reservation passwords in DOCX, XLSX and PPTX | | printer_information | Controls printer settings in DOCX, XLSX and PPTX | The switches currently available for each format are depicted in the table below: | Switch | PDF | DOC | DOCX | PPT | PPTX | XLS | XLSX | GIF | JPEG | SVG | WEBP | TIFF | |----------------------------------|------------------|-----------------------|-----------------------|---------------------|---------------------|--------------------|--------------------|------------------|----------------------|--------|------|------| | acroform | *✓* | | | | | | | | | | | | | actions_all | *✓* | | | | | | | | | | | | | connections | | | | | | | *✓* | | | | | | | digital_signatures | *✓* | | *✓* | | *✓* | | *✓* | | | | | | | dynamic_data_exchange | | *✓* | | | | *✓* | | | | | | | | embedded_files | *✓* | *✓* | *✓* | *✓* | *✓* | *✓* | *✓* | | | | | | | embedded_images | *✓* | *✓* | *✓* | *✓* | *✓* | *✓* | *✓* | | | | | | | external_hyperlinks | *✓* | *✓* | *✓* | *✓* | *✓* | *✓* | *✓* | | | | | | | foreign_objects | | | | | | | | | | *✓* | | | | geotiff | | | | | | | | | | | | *✓* | | hidden_data | | | *✓* | | *✓* | | *✓* | | | | | | | hyperlinks | | | | | | | | | | *✓* | | | | internal_hyperlinks | *✓* | *✓* | *✓* | *✓* | *✓* | *✓* | *✓* | | | | | | | in_text_comments | | | *✓* | | | | | | | | | | | javascript | *✓* | | | | | | | | | | | | | jfif | | | | | | | | | *✓* | | | | | macros | | *✓* | *✓* | *✓* | *✓* | *✓* | *✓* | | | | | | | metadata | *✓* | *✓* | *✓* | *✓* | *✓* | *✓* | *✓* | *✓* | | | *✓* | | | retain_exported_streams | *✓* | | | | | | | | | | | | | review_comments | | *✓* | *✓* | *✓* | *✓* | *✓* | *✓* | | | | | | | slide_notes | | | | | *✓* | | | | | | | | | scripts | | | | | | | | | | *✓* | | | | tracked_changes | | | *✓* | | | | *✓* | | | | | | | value_outside_reasonable_limits | *✓* | | | | | | | | | | | | | undefined_type | | | | | | | | | | | | *✓* | | doc_variables | | | *✓* | | | | | | | | | | | web_video_extension | | | *✓* | | | | | | | | | | | headers_footers | | | *✓* | | | | *✓* | | | | | | | scenarios | | | | | | | *✓* | | | | | | | footnotes_endnotes | | | *✓* | | | | | | | | | | | field_codes | | | *✓* | | | | | | | | | | | embedded_fonts | | | *✓* | | *✓* | | | | | | | | | write_reservation | | | *✓* | | *✓* | | *✓* | | | | | | | printer_information | | | *✓* | | *✓* | | *✓* | | | | | | All content types not represented by a content management type for a specific file format will be **automatically remediated by the Glasswall engine if identified as malicious.** ## Embedded files The "Embedded Files" content management type applies to non-image file formats which are located within a distinct container file. For MS-Office formats, the policy for embedded files is applied differently depending on whether the file considered is supported and accessible to the engine: **Action applied to embedded file according to content management policy for Microsoft Office files:** | | Allow | Sanitise | Disallow | | ----------- | ------------------------------------------------------------ | ------------------------------------------------------------ | ------------------------------------------------------------ | | Supported | Treated as standalone file. If file is non-conforming, containing file is rejected and reason for non-conformance reported as an Issue Item. | Treated as standalone file. If file is non-conforming, containing file is rejected and reason for non-conformance reported as an Issue Item. | Containing file is rejected, with the embedded file described in an Issue Item. | | Unsupported | Regenerated without alteration and reported as an Allowed Item. | Removed from containing file, alongside all references to it, and reported as a Sanitisation Item. | Containing file is rejected, with the embedded file described in an Issue Item. | The table below outlines which embedded file formats are supported (✓) within each container file type, and which are not (✗). | Embedded File Format ↓ / Container Format →| DOCX/XLSX/PPTX | DOC/XLS/PPT | PDF | |--------------------------------------------|----------------|------------------|------------------| | Office 2007 | *✓* | ✗ | ✗ | | Office 2003 | *✓* | *✓* | ✗ | | Office 1997 | *✓* | *✓* | ✗ | | PDF | ✗ | ✗ | ✗ | | MP3 | ✗ | n/a † | *✓* ‡ | | MP4 | ✗ | n/a † | *✓* ‡ | | MPEG | ✗ | n/a † | *✓* ‡ | | WAV | ✗ | ✗ | *✓* ‡ | | Formats unsupported by Glasswall | ✗ | ✗ | ✗ | [†]: Disallowed by container format [‡]: Not removed by Embedded Files switch but may be removed by All Actions switch. Embedded file is regenerated without being processed. > ⚠️ **Note:** To preserve visual integrity between the original and sanitised versions of files, **associated visual elements** (such as thumbnails and `blip` references) of unsupported embedded files are **not removed** during sanitisation. This ensures that post-processed files remain visually consistent with their original versions. ### Embedding depth support The Embedded Engine supports up to **nine** levels of nested embedded content within OfficeXML files. If any embedded files are found beyond this depth, the container file will be rejected, and an Issue Item will be raised indicating that the maximum recursion limit has been exceeded. This limit applies only to the depth of nesting, and multiple embedded files at the same level do not count against it. ## Embedded images For image file formats, the "Embedded Images" content management switch should be used. This has the following behaviour depending on switch setting: **Action applied to embedded image according to content management policy:** | | Allow | Sanitise | Disallow | | ----------------------- | ------------------------------------------------------------ | ------------------------------------------------------------ | ------------------------------------------------------------ | | Supported | Treated as standalone file. If file is non-conforming, containing file is rejected and reason for non-conformance reported as an Issue Item. | Treated as standalone file. If file is non-conforming, containing file is rejected and reason for non-conformance reported as an Issue Item. | Containing file is rejected, with the embedded image described in an Issue Item | | Unsupported* | Regenerated without alteration and reported as an Allowed Item | Removed from containing file, alongside all references to it, and reported as a Sanitisation Item | Containing file is rejected, with the embedded image described in an Issue Item | [ * ] : *Unsupported embedded images may be instead handled by the "embedded_files" switch if the engine does not recognise the filetype* The table below shows which image formats we attempt to regenerate (`✓`) when "Embedded Images" is set to `sanitise` versus those which are removed (`✗`): | Embedded Image Format | DOCX/XLSX/PPTX | DOC/XLS/PPT | PDF | |-------------------------------------|------------------|------------------|------------------| | BMP, JPEG, GIF, PNG, EMF, SVG, TIFF | *✓* | *✓* | *✓* ⸸ | | WMF, EMF | *✓* | *✓* | *✓* ⸸ | | WebP | *✓* ⸸ | *✓* ⸸ | *✓* ⸸ | | Formats unsupported by Glasswall | ✗ | ✗ | *✓* ⸸ | [⸸]: Will be converted to a different format by container file Please note that when the "Embedded Images" is set to "Disallow", any images being encountered will result in the rejection of the containing file. ***This includes thumbnails of the containing or embedded documents*** and so may supersede the "Embedded File" content management switch. ## Macros The `macros` content switch for MS Office files applies to both Microsoft Visual Basic for Applications (VBA) and Excel 4.0 macros. ### Microsoft Visual Basic for Applications VBA macros are written in the Visual Basic programming language and can be included in any MS Office file format. The handling of VBA macros can be configured as follows: - ***Sanitise*** - VBA macros are removed from files. - ***Disallow*** - VBA macros are forbidden. If one is found, the containing file is rejected. - ***Allow*** - VBA macros are processed and regenerated as part of the containing file providing they conform to specification. #### Export mode behaviour In Export mode, VBA Project Binaries count toward the recursion limit. This means the maximum nesting depth is reduced to **eight** if the deepest embedded file contains a VBA macro. ### Excel 4.0 macros Excel 4.0 macros are a legacy feature included in XLSX and XLS files. XLSX files containing Excel 4.0 macros will be saved using the ".xlsm" file extension and will produce an error if this extension is modified. The handling of Excel 4.0 macros can be configured as follows: - ***Sanitise*** - In XLS files, the file will be blocked and `Excel 4.0 Macro found: Not supported` reported as an issue item. In XLSX/XLSM files, sheets containing macros will be removed from the document and reported as a sanitisation item. If this causes the file to be malformed (i.e. reducing the number of visible sheets to zero), the file will be rejected and an appropriate issue item reported. - ***Disallow*** - Excel 4.0 macros are forbidden. If one is found, the containing file is rejected. - ***Allow*** - In XLS files, the file will be blocked and `Excel 4.0 Macro found: Not supported` reported as an issue item. In XLSX/XLSM files, the file will be regenerated with macros intact. ## Metadata In OOXML, metadata refers to information that describes the content, structure, and properties of a document but is not part of the document's main content. Metadata in OOXML documents is primarily stored in XML files located within the docProps directory: 1. core.xml: Contains core properties based on the Dublin Core Metadata Element Set. 2. app.xml: Contains extended properties specific to Microsoft Office applications. 3. custom.xml: Contains custom properties. The handling of OOXML metadata can be configured as follows: - ***Sanitise*** - The file is regenerated with metadata removed (see below for all the properties currently sanitised) - ***Disallow*** - Metadata is forbidden. If any metadata (properties listed below) is found, the containing file is rejected. - ***Allow*** - The file is processed, and the metadata is regenerated. As part of the 'metadata' content management switch, we currently sanitise the following in: - core.xml: title, subject, creator, keywords, description, lastModifiedBy, revision, lastPrinted, created, modified, category, contentStatus, language, and version. - app.xml: manager, company, and hyperlinkBase - custom.xml: any custom properties added to the OOXML document. ## Digital signatures ### In PDF **Overview** PDF files may contain Digital Signatures and AcroForms, certain types of AcroForms can contain digital signatures. While digital signatures are used to verify the authenticity and integrity of a document, AcroForms provide the structural foundation for interactive form fields. When a digital signature is present in the PDF, then the AcroForm has the visible representation of the signature itself. When processing PDF files that include digital signatures, the Glasswall CDR engine applies a sanitisation process designed to preserve visual integrity while removing active and/or potentially risky content. **How the CDR Engine handles Digital Signatures** To ensure both document safety and consistency, the Glasswall CDR engine performs the following actions during sanitisation: * Removes the cryptographic signature data, including any embedded certificates, validation logic, or scripts. * Strips signature-related metadata and interactive behavior to eliminate execution pathways or potential exploits. * Preserves the visual appearance of the signature widget, such as the signature image, signer name, and date/time text. This is achieved by flattening it into the static content layer of the PDF. | **AcroForm** | **Digital Signature** | **Expected AcroForm behavior** | **Expected Digital Signature behavior** | **Behavior of AcroForm section containing Digital Signature** | **Is File Regenerated?** | | ------------ | --------------------- | -------------------------------- | ------------------------------------------ | ------------------------------------------------------------------------------------------------------- | ------------------------ | | Allow | Allow | Regenerated without sanitisation | Regenerated without sanitisation | Entire section (including interactive form and digital signature) is preserved as-is | Yes | | Sanitise | Allow | Sanitised (removed or flattened) | Regenerated without sanitisation | Visual digital signature is preserved; AcroForm field it resides in is sanitised or removed | Yes | | Allow | Sanitise | Regenerated without sanitisation | Sanitised (cryptographic elements removed) | Visual part of digital signature is preserved as part of the AcroForm; signature becomes non-functional | Yes | | Sanitise | Sanitise | Sanitised | Sanitised | Entire digital signature section, including AcroForm fields, is removed or flattened visually | Yes | | Disallow | * | Not applicable | Not applicable | File is **not regenerated** due to disallowed AcroForm presence | No | | * | Disallow | Not applicable | Not applicable | File is **not regenerated** due to disallowed Digital Signature presence | No | #### Auditability and chain of custody To support traceability and accountability in secure environments, the Glasswall CDR engine records the cryptographic hashes of both the input and output files. This enables a system integrator: * To verify file provenance through hash comparison. * To provide assurance that, where a digital signature is no longer valid, the chain of custody is maintained and can be proven. ### In OfficeXML Digitally signed Office documents store the signature inside an `_xmlsignatures` directory. This directory only contains the cryptographic data of the signature, while the visual representation is just a standard XML element inside the main document. The handling of OOXML digital signature can be configured as follows: - ***Sanitise*** - The file is regenerated with the `_xmlsignature` directory removed. The visual fields where the signature was embedded are preserved, but the document is now unsigned. - ***Disallow*** - Digital signatures are forbidden. Any signed file is rejected. - ***Allow*** - The file is processed, and the `_xmlsignature` directory is regenerated exactly as it was. The signature is no longer valid, as the file has been altered by the CDR process. Word/Excel/PowerPoint will show a warning, informing the user of an invalid signature. ## OfficeXML (DOCX, XLSX, PPTX) exclusive switches ### Hidden data Office file formats offer multiple different ways of legitimately "hiding" text or data, including whole Excel sheets, PowerPoint slides or lines of text in a Word document. The Glasswall engine deals with hidden data in the following ways, depending on the content management switch setting: - ***Sanitise*** - The file is regenerated with all hidden data "unhidden", so it is completely visible to the user. - ***Disallow*** - Hidden data is forbidden. If any hidden data is found, the containing file is rejected. - ***Allow*** - Any hidden data is regenerated and remains hidden. _Note: For the purposes of this content management setting, “Hidden Data” does not refer to the varied ways to obfuscate or bury data in Office 2007 files. Rather, it is specific to the methods of hiding data that are readily available in the Office 2007 GUI. Obfuscated or concealed data is managed by the policy setting corresponding to the method used, e.g., `metadata` will remove data concealed within free-text fields contained within the document's metadata._ ### Tracked changes The `tracked_changes` content management switch refers to content added by the "Track Changes" functionality in DOCX and XLSX files, also known as "revisions". These can contain historic data related to previous versions of the document, including names of contributors and records of content that has since been removed or obfuscated. The handling of tracked changes can be configured as follows: - ***Sanitise*** - All historic data is removed and "Track Changes" disabled. The regenerated document will be equivalent to the final state of the original document. - ***Disallow*** - Tracked changes are forbidden. If there is any evidence of previous revisions or tracked changes still present in the file, the file will be rejected. - ***Allow*** - The file is regenerated with all historic changes, revisions and tracked changes intact. ### Slide notes The `slide_notes` content management switch refers to content added by the "Notes" functionality in PPTX files, also known as "slide notes" (and/or "speaker notes"). The Glasswall engine deals with these slide notes in the following ways, depending on the configuration of the content management switch setting: - ***Sanitise*** - The file is regenerated with all slide notes removed. - ***Disallow*** - Slide notes are forbidden. If any slide notes are found, the containing file is rejected. - ***Allow*** - Any slide notes are regenerated and remain in the file. ### In-Text comments The `in_text_comments` switch refers to content added by the "In-Text Comments" functionality in DOCX files. The handling of the switch can be configured as follows: - ***Sanitise*** - In-Text Comment is removed alongside the corresponding document metadata found in `core.xml`. - ***Disallow*** - In-Text Comment is forbidden. Any DOCX containing an in-text comment will block the file from being regenerated. - ***Allow*** - The file is regenerated with the In-Text Comment present in the DOCX file. _Note: When `in_text_comments` sanitise is set to allow and `metadata` switch is set to sanitise then the regenerated file will have the in-text comment present without any data since the metadata switch sanitises the corresponding description from the `core.xml` file._ ### Doc-Variables The `doc_variables` switch refers to content added by the "Document Variables" functionality in DOCX files. The handling of the switch can be configured as follows: - ***Sanitise*** - The file is regenerated with all document variables removed. - ***Disallow*** - Document variables are forbidden. Any DOCX containing document variables will block the file from being regenerated. - ***Allow*** - The file is regenerated with the document variables in the DOCX file. _Note: `doc_variables` controls the stored variable definitions; the `DOCVARIABLE` field that displays a variable in the body is controlled separately by `field_codes`. The two act independently._ ### Web Video Extension The `web_video_extension` switch refers to content added using the "Web Videos Extension" functionality in DOCX files. The handling of the switch can be configured as follows: - ***Sanitise*** - The file is regenerated with all web video extensions removed. - ***Disallow*** - Use of web video extension is forbidden. Any DOCX containing a web video extensions will block the file from being regenerated. - ***Allow*** - The file is regenerated with the web video extension included in the DOCX file. ### Header and Footers The `headers_footers` switch refers to the content added to Header or Footer sections of a page in DOCX or sheet in XLSX, this content can be different on the first, even or odd page / sheet or the same throughout the document. The handling of the switch can be configured as follows: - ***Sanitise*** - The file will be regenerated with all header and footer content removed. - ***Disallow*** - Use of headers and footers is forbidden. Any DOCX or XLSX containing either headers or footers will block the file from being regenerated. - ***Allow*** - The file is regenerated with the headers and footers included in the DOCX or XLSX file. ### Scenarios The `scenarios` switch refers to content added by the "Scenarios" functionality in XLSX files. The handling of the switch can be configured as follows: - ***Sanitise*** - The file will be regenerated with all scenarios removed. - ***Disallow*** - Use of scenarios is forbidden. Any XLSX containing scenarios will block the file from being regenerated. - ***Allow*** - The file is regenerated with the scenarios included in the XLSX file. ### Footnotes and Endnotes The `footnotes_endnotes` switch refers to content added by the "Footnotes" or "Endnotes" functionality in DOCX files. The handling of the switch can be configured as follows: - ***Sanitise*** - The file will be regenerated with all footnotes and endnotes removed. - ***Disallow*** - Use of footnotes and endnotes is forbidden. Any DOCX containing footnotes or endnotes will block the file from being regenerated. - ***Allow*** - The file is regenerated with the footnotes and endnotes included in the DOCX file. ### Field codes The `field_codes` switch refers to content added by the "Smart Formatting" feature in DOCX files. The handling of the switch can be configured as follows: - ***Sanitise*** - The file is regenerated with field code definitions removed, and the field code's most recently generated value is retained. - ***Disallow*** - Field codes are forbidden. Any DOCX containing field codes will block the file from being regenerated. - ***Allow*** - The file is regenerated with field code definitions and results preserved. This switch currently only controls a subset of field codes. The list can be found below: - `=` - `ADVANCE` - `ASK` - `AUTOTEXT` - `AUTOTEXTLIST` - `COMMENTS` - `COMPARE` - `DATABASE` - `DOCPROPERTY` - `DOCVARIABLE` - `FILLIN` - `FORMCHECKBOX` - `FORMDROPDOWN` - `FORMTEXT` - `GLOSSARY` - `IF` - `IMPORT` - `INCLUDE` - `INCLUDEPICTURE` - `INCLUDETEXT` - `INFO` - `LINK` - `MACROBUTTON` - `PRINT` - `QUOTE` - `RD` - `SET` - `SHAPE` - `SYMBOL` _Note: `DOCVARIABLE` here refers to the in-body field that displays a variable's value, not the stored variable definitions—those are controlled separately by `doc_variables`._ ### Embedded Fonts The `embedded_fonts` switch refers to content added by the "Embedded Fonts" feature in DOCX and PPTX files. The handling of the switch can be configured as follows: - ***Sanitise*** - The file is regenerated with Embedded Fonts removed. - ***Disallow*** - Embedded Fonts are forbidden. Any DOCX or PPTX containing Embedded Fonts will block the file from being regenerated. - ***Allow*** - The file is regenerated and Embedded Fonts remain in the file. ### Write Reservation The `write_reservation` switch refers to content added by the "Write Reservation Passwords" feature in DOCX, XLSX and PPTX files. The handling of the switch can be configured as follows: - ***Sanitise*** - The file is regenerated with the write reservation password removed. The document opens normally without any password prompt and is fully editable. - ***Disallow*** - Write reservation passwords are forbidden. Any files containing a write reservation password will block the file from being regenerated. - ***Allow*** - The file is regenerated with the write reservation password preserved. The recipient sees the password prompt on open, identical to the original file. _Note: This switch only affects write reservation (the password required to modify the file)._ ### Printer Information The `printer_information` switch refers to printer settings in DOCX, XLSX and PPTX files. The handling of the switch can be configured as follows: - ***Sanitise*** - The file is regenerated with all printer settings removed. - ***Disallow*** - Printer settings are forbidden. Any files containing printer settings will block the file from being regenerated. - ***Allow*** - The file is regenerated with printer settings preserved. Any printer settings streams in the file will be checked for correct structure. ### System configuration "sysConfig" switches The `sysConfig` switches control the overall behaviour of the Embedded Engine in various ways that don't fall under Content Management. | Switch name | Switch setting | Default | Description | | ---------------------------- | ------------------------ | ------------- | ------------------------------------------------------------ | | enable_hash_sha256 | true/false | true | Calculates SHA256 hashes of input and output files, adding them into analysis reports. Enabling this will increase processing time. | | enable_text_support | false/true | false | **(beta feature)** Word Search only. Enabling this will allow UTF-8 or ASCII encoded text files to be processed when at least one "Require" action is specified. When disabled, the file will be rejected as an unsupported filetype. | | export_embedded_images | true/false | true | Export embedded images to SISL or XML when set to 'true', or save as raw images when set to 'false'. | | interchange_best_compression | false/true | false | Compress the Export archive package at the maximum compression level. Enabling this will increase processing time. | | interchange_pretty | false/true | false | Format the intermediate SISL or XML data structure to be more human-readable. Enabling this will slightly increase intermediate file size. | | interchange_type | sisl/xml | sisl | The intermediate file format for the exported document object model. | | linux_memory_limit | 0 to 256 (integer only) | 0 | Enforces a limit on the memory usage of the process, while processing a session, in GiB. This limit triggers process termination. '0' disables this limit. This setting has no effect on non-Linux platforms. See below for details. | | session_timeout | 0 to 60 (integer only) | 60 | Enforces a time limit on running a session, in minutes. This limit triggers process termination. '0' disables this timeout. See below for details. | | enable_export_xml_headers | true/false | true | An XML header is added to the beginning of the xml file(s) generated during export. Requires the interchange_type to be xml. | #### linux_memory_limit - technical details This option controls a limit on the memory used by the process. This limit is enforced for the duration of a call to [GW2RunSession](/embedded-engine/embedded-engine-gw2runsession), and does not apply outside of this API call. The value is specified in GiB, integers only. The value '0' for this option means 'no limit'. This is the default. It applies only to Linux-based platforms. On other platforms, this option has no effect. When the limit is reached, the process will print an error message to stderr and terminate. The exit code seen after termination is platform dependent, but will be consistent with application termination by SIGABRT. The limit is imposed on the peak Resident Set Size (RSS), which is the amount of physical memory the process consumes. #### session_timeout - technical details This options controls a limit on the amount of time a call to [GW2RunSession](/embedded-engine/embedded-engine-gw2runsession) can last. It does not apply outside of this API call. The value is specified in minutes, integers only. The value '0' for this option means 'no timeout', meaning GW2RunSession will continue for as long as required. The default value is '60', giving a failsafe timeout of one hour. When the timeout is reached before GW2RunSession completes, the process will print an error message to stderr and terminate. The exit code seen after termination is platform dependent, but will be consistent with application termination by SIGABRT Subsequent calls to GW2RunSession begin their own timer. #### Content management "sysConfig" switches ``` sisl false false true true 0 60 false ``` The Glasswall distribution consists of all libraries, wrappers, tools, config and test files required to get started within a single archive. You will need to extract the archive and include the relevant artifacts in your project. The directory structure used in the examples below is correct for version 16.4.0 onwards, for previous versions of the engine you will need to adjust the file system paths shown. The Glasswall libraries are split up according to the supported Operating System and architecture: - `/libs/debian/x86-64` - Linux x86-64 OS libraries - `/libs/debian/arm64` - Linux arm64 OS libraries - `/libs/windows/x86-64` - Windows x86-64 OS libraries Language specific wrappers can be found in `/libs/OS/ARCHITECTURE/sdk_editor_wrappers` and `/libs/OS/ARCHITECTURE/sdk_word_search_wrappers`. These allow users to use a wider range of programming languages for their own applications that use the Glasswall engine. Wrappers are available for: - Java (8, 11 and 18) - .NET (8 or higher) - Python (3.6 or higher) ## General requirements The general requirements are: - The Glasswall libraries: `glasswall_core2.dll` for Windows OS and `libglasswall_core2.so` for Linux. These libraries contain the Glasswall engine. - The Glasswall camera libraries: `*_camera.dll` for Windows OS and `*_camera.so` for Linux and Alpine OS. These are libraries used by the Glasswall engine to process specific filetypes. - Content management policies that modify the processing behaviour. Examples can be found in `/configs`. - A way of executing the Glasswall engine. This can be an application written using one of the provided language wrappers, or an application that directly interacts with the Glasswall library. - Content management policy settings that can be applied to alter the CDR processing behaviour. Examples can be found in the `configs` directory. ## Installation ### Environment variables The following environment variable will need to be set up in order for Editor to find the Glasswall camera libraries: `QT_PLUGIN_PATH=...`: Where `...` is the full path to the directory containing the Editor and Editor camera libraries. If you are having issues with no documents being processed even though they are supported then double check that this environment variable is setup correctly. ##### Environment variables in Linux A temporary environment variable can be created by running `export QT_PLUGIN_PATH=...`. This will create the environment variable, but only for the current shell and its child processes. Once the shell is closed then this variable will no longer be available. A permanent environment variable can be created by adding `export QT_PLUGIN_PATH=...` to the `~/.bashrc` file. This enables the environment variable to be set up every time a shell instance is created. After adding this line you can call `source ~/.bashrc` to enable this variable to be present for the current shell. ##### Environment variables in Docker A permanent environment variable can be created by adding `ENV QT_PLUGIN_PATH=...` to your Docker file. ##### Environment variables in Windows A temporary environment variable can be created by running `set QT_PLUGIN_PATH=...`. This will create the environment variable, but only for the current shell and its child processes. Once the shell is closed then this variable will no longer be available. A permanent environment variable can be created by running `setx QT_PLUGIN_PATH=...`. This will create the environment variable when a new shell instance is created. After adding this line you can run `set QT_PLUGIN_PATH=...` to enable this variable to be present for the current shell. WARNING: The above commands will truncate your environment variable if it exceeds 1024 characters. In this case you will need to use another method, such as using the graphical interface, to create the environment variable. ### Linux On Linux there are a few different ways of installing the Glasswall libraries. Here we demonstrate a temporary and some permanent solutions. For each example the release package has been extracted to `/home/Glasswall`. You may need to adapt the filepaths to match your environment. #### Example 1 - temporary solution A temporary setup can be created by extracting the archive to a directory and then setting up the environment variable `LD_LIBRARY_PATH=...` to point to the directories where the Glasswall libraries are located. - Run; - `export QT_PLUGIN_PATH=/home/Glasswall/libs/debian/x86-64/sdk_editor`. - This will set up the `QT_PLUGIN_PATH` to point to the directory containing the Glasswall camera libraries. - Run; - `export LD_LIBRARY_PATH=/home/Glasswall/libs/debian/x86-64/sdk_editor:/home/Glasswall/libs/debian/x86-64/sdk_word_search`. - This will set up the `LD_LIBRARY_PATH` to point to the directories that contain the different libraries. If you are using Java you will also need to add paths to the Java wrapper directories to `LD_LIBRARY_PATH`. #### Example 2 - installation in /usr/lib A permanent setup can be created by copying all the libraries to `/usr/lib/` or another directory that is searched by the linker, and then running `ldconfig` to update the list of installed libraries. - Run with admin privileges; - `find /home/Glasswall/libs/debian/x86-64 -type f -name "*.so*" -exec install -m 755 "{}" /usr/lib \;` - This will copy all the Glasswall libraries into `/usr/lib` directory and set the permissions to `755`. If you are using Java you will also need to do this for the Java wrapper libraries. - Run with admin privileges; - `ldconfig` - This will update the list of installed libraries so that the Glasswall libraries can be found. - Run; - `echo 'QT_PLUGIN_PATH=/usr/lib/' > ~/.bashrc` - This will permanently add the `QT_PLUGIN_PATH` environment variable to point to the location of the Glasswall libraries. - Run; - `source ~/.bashrc` - This will update the current shell to include the `QT_PLUGIN_PATH` environment variable. #### Example 3 - installation in /opt/glasswall Another permanent setup can be created by copying the content of the extracted archive to a place such as `/opt/glasswall`, and creating a config file in `/etc/ld.so.d/` that tells the system where to find the Glasswall libraries. - Run with admin privileges; - `mkdir -p /opt/glasswall` - This will create the directory where the content of the release package will be stored. - Run with admin privileges; - `cp -r /home/Glasswall/* /opt/glasswall/` - This will copy the extracted contents of the release package into the `/opt/glasswall/` directory. - Run with admin privileges; - `find /opt/glasswall/ -name "*.so*" -exec chmod 755 "{}" \;` - This will find all the Glasswall libraries and set their permissions to 755 - Run with admin privileges; - `echo -e "/opt/glasswall/libs/debian/x86-64/sdk_editor\n/opt/glasswall/libs/debian/x86-64/sdk_word_search" >> /etc/ld.so.conf.d/glasswall.conf` - This will create the configuration file `/etc/ld.so.conf.d/glasswall.conf` containing the paths to the locations of the Glasswall libraries. If you are using Java you will also need to add paths to the Java wrapper libraries to glasswall.conf. - Run with admin privileges; - `ldconfig` - This will update the list of installed libraries so that the Glasswall libraries can be found. - Run; - `echo 'QT_PLUGIN_PATH=/opt/glasswall/libs/debian/x86-64/sdk_editor' > ~/.bashrc` - This will permanently add the `QT_PLUGIN_PATH` environment variable to point to the location of the Glasswall libraries. - Run; - `source ~/.bashrc` - This will update the current shell instance to include the `QT_PLUGIN_PATH` environment variable. ### Windows The set up for Windows is very similar to the set up for Linux where you need to find a place for the Glasswall libraries to be stored, and then set up the environment variables so that the Glasswall libraries can be found. In our example we are storing the contents of the release archive in `C:\Program Files\Glasswall`. - Extract the contents of the Release archive into `C:\Program Files\Glasswall` - Add the following paths to the `PATH` environment variable: - `C:\Program Files\Glasswall\libs\windows\x86-64\sdk_editor` - `C:\Program Files\Glasswall\libs\windows\x86-64\sdk_word_search` - If you are using Java, add the appropriate paths containing the Java wrapper libraries to the `PATH` environment variable. - Add the Editor cameras directory to the `QT_PLUGIN_PATH` environment variable. For example: - `C:\Program Files\Glasswall\libs\windows\x86-64\sdk_editor` #### OpenSSL and Visual C++ Redistributable Windows users will also need to install OpenSSL and the Visual C++ Redistributable packages. A Chocolatey installation of OpenSSL is recommended which will also install the Visual C++ Redistributable. - The instructions to install Chocolatey can be found [here](https://chocolatey.org/install). - Once Chocolatey is installed, open a command prompt or Windows Power Shell with administrative permissions. - Type the command `choco install openssl` and follow the instructions to complete the installation. - Once complete, restart any applications or console sessions to apply the updated environment variables. import Tabs from '@theme/Tabs' import TabItem from '@theme/TabItem' The following sections describe the key concepts for successful Glasswall Embedded Engine operation. ## Sessions The Application Programming Interface (API) is session based. Session is a type that represents a file and the mechanisms used to process that file. - You create a Session object by calling `GW2OpenSession` which returns a session handle. - As more than one session may be active, the `Session ID` is used to access data and set variables for a particular session. - You pass the session handle to other API functions to register the inputs and outputs and the forms of the inputs and outputs (memory or a file). - You then process the file by calling the function `GW2RunSession` and close the session by calling `GW2CloseSession`. When opened, each session is assigned its own series of memory buffers. These buffers remain accessible for as long as the session remains open. Once a session is closed, the data is no longer available and the allocated memory is released. **A session must always be closed**. This includes scenarios where a processing issue occurs and the expected flow of operation is interrupted. In this instance, numerous defensive coding patterns can be adopted that are designed for managing resources e.g. `try-with-resources` in Java, or `context-handlers` in Python. > **Note** that the more sessions that are concurrently open, the larger the overall memory requirements will be. Processing files using multiple threads is not supported. **Spawning multiple processes** is advised in order to facilitate parallel processing. **Use Cases:** Making a file safe and/or generating a report describing a files content **Use Cases:** Normalising the contents of a file to xml to enable further external processing **Use Cases:** Reconstituting exported xml content back to the original file format Further information on the Embedded Engine API's are documented across the relevant [API function pages](/embedded-engine/embedded-engine-api-functions). ## Policy files A policy file is used to determine how Glasswall should process supplied files. A sample policy file is provided in the Embedded Engine release package. Whilst a policy file is not required for Glasswall to run, in most cases one is used to customise processing to match requirements. If a policy file is to be used, it must be registered to each session. If no policy file is specified, content management settings are set to `sanitise` files by default. Default settings would also be applied for sysConfig options. See [Content Management](/embedded-engine/embedded-engine-content-management) and [System Configuration](/embedded-engine/embedded-engine-system-configuration) for further information. ## Return values from Glasswall Most functions within the Glasswall API return an integer indicating success or failure. A `0` or a `1` will indicate success, whilst a negative number (such as `-1`) would indicate a failure. Certain API calls do not follow this pattern; for example, the `GW2OpenSession` function will return either a `-1` error or the ID of the newly-created session (positive integer). A table of return values can be found on the [API Overview](/embedded-engine/embedded-engine-api-overview) page. ## Data management Glasswall processes and saves data using files and/or buffers. The data requirements for these processes are described in the sections below. ### Files File-based Glasswall API calls require a file path as a parameter. This file name must be encoded as a C-type string; an array of characters terminated with a NULL character. UTF-8 encoding should be used. ### Memory Memory-based Glasswall API calls require the use of one or more memory buffers. These buffers are defined by a pointer to the first element and the length of the buffer in bytes. This approach allows files containing NULL characters to be processed correctly. #### Import/Input functions These functions require a pointer to the first datum in the buffer and the length of the buffer in bytes. #### Export/Output functions These functions specify the buffer that Glasswall should use when returning processed data. Glasswall requires pointers to both the buffer length and the pointer to the first element, as described below: - A pointer-to-pointer to the buffer’s first element - A pointer to the length of the buffer (size_t *) This extra layer of abstraction is required as the size of this buffer is not known prior to running the session. Dereferencing these pointers allows access to the data returned from Glasswall. ## Prerequisites ### General - [Python >= 3.8](https://www.python.org/downloads/) - See [Installation](/embedded-engine/embedded-engine-installation) for general OS specific requirements and environment set up. :::note From version 5.12.0 the Glasswall Python Wrapper requires Python 3.8 or higher. If you are running Python 3.6 or 3.7, install version 5.11.0 instead (`pip install glasswall==5.11.0`). ::: ### Glasswall Python Wrapper installation #### Online installation ``` pip install --upgrade glasswall ``` #### Offline installation Run the following commands within the directory containing the offline installation files. ``` pip install --upgrade --no-index --find-links=. glasswall ``` >**Note:** The wheels available for offline installation include all necessary dependencies for their respective packages and have been tested on amazonlinux.2023, rockylinux.8.9, and ubuntu.22.04 environments. ## Loading a Glasswall library ### Editor Libraries are loaded on initialization and have one required argument: `library_path` which can be the path to a file or a directory. If a directory is specified it is recursively searched and the library with the latest change time will be loaded. ```py import glasswall # Load the Glasswall Editor library editor = glasswall.Editor(r"C:\gwpw\libraries\10.0") ``` ``` >>> 2025-03-15 12:27:42.337 glasswall INFO __init__ Loaded Glasswall Editor version 2.1464.1 from C:\gwpw\libraries\10.0\glasswall_core2.dll ``` ### WordSearch WordSearch has a dependency on the Editor libraries. When loading WordSearch, ensure that the WordSearch and Editor libraries are located within the same working directory. ```py import glasswall # Load the Glasswall WordSearch library word_search = glasswall.WordSearch(r"C:\gwpw\libraries\10.0") ``` ``` >>> 2025-06-03 11:19:09.223 glasswall.config.logging INFO __init__ Loaded Glasswall WordSearch version 1.249.0 from from C:\gwpw\libraries\10.0\glasswall.word.search.dll ``` ## Providing a licence By default, the `Editor` class expects a valid licence file to be located in the same directory as the `library_path`. You can also specify a different path to a `gwkey.lic` licence file using the `licence` argument. ```py import glasswall # Load the Glasswall Editor library with a specified licence file editor = glasswall.Editor(r"C:\gwpw\libraries\10.0", licence=r"C:\gwpw\licence\gwkey.lic") ``` Alternatively, you can pass the licence data in-memory as a bytes, bytearray, or io.BytesIO object. ```py import glasswall # Alternatively, load the licence from in-memory bytes or bytearray with open(r"C:\gwpw\licence\gwkey.lic", "rb") as f: licence_data = f.read() editor = glasswall.Editor( r"C:\gwpw\libraries\10.0", licence=licence_data # In-memory licence data ) ``` In Protect Mode, content management policies allow control of various file content types such as executable code, interactive form content and a number of actions (e.g., external links or the execution of JavaScript). These file elements are known to be common attack vectors when they are encountered within a file. The content management policy will define how the Glasswall Embedded Engine should process these structures. In [Analysis Mode](#analysis), these are reported as `SanitisationItems`. Content management policy differs across supported file types. Automatic corrections back to the file specification are also performed upon file regeneration. The purpose of this is to enable the Glasswall Embedded Engine to remove threats that are hidden within the file structure, as well preventing the possibility of activating exploits via the misuse of structural components in the file. In [Analysis Mode](#analysis), these are reported as `RemedyItems`. Files can be protected individually from a file path or in memory using the [protect_file](./8-Autogenerated%20Docs/libraries/editor/editor/editor.md#protect_file) or [protect_directory](./8-Autogenerated%20Docs/libraries/editor/editor/editor.md#protect_directory) methods. ## Examples - [Protect](#protect) - [Protect from file path to file path](#protect-from-file-path-to-file-path) - [Protect from file path to memory](#protect-from-file-path-to-memory) - [Protect from memory](#protect-from-memory) - [Protect files in a directory](#protect-files-in-a-directory) - [Protect files in a directory that may contain unsupported file types](#protect-files-in-a-directory-that-may-contain-unsupported-file-types) - [Protect files in a directory using a custom content management policy](#protect-files-in-a-directory-using-a-custom-content-management-policy) - [Protect files in a directory conditionally based on file format](#protect-files-in-a-directory-conditionally-based-on-file-format) - [Analysis](#analysis) - [Analyse from file path to file path](#analyse-from-file-path-to-file-path) - [Analyse from file path to memory](#analyse-from-file-path-to-memory) - [Analyse from memory](#analyse-from-memory) - [Analyse files in a directory](#analyse-files-in-a-directory) - [Analyse files in a directory that may contain unsupported file types](#analyse-files-in-a-directory-that-may-contain-unsupported-file-types) - [Analyse files in a directory using a custom content management policy](#analyse-files-in-a-directory-using-a-custom-content-management-policy) - [Analyse files in a directory conditionally based on file format](#analyse-files-in-a-directory-conditionally-based-on-file-format) - [Protect and Analyse](#protect-and-analyse) ## Protect ### Protect from file path to file path ```py import glasswall # Load the Glasswall Editor library editor = glasswall.Editor(r"C:\gwpw\libraries\10.0") # Use the default policy to sanitise a file, writing the sanitised file to a new path editor.protect_file( input_file=r"C:\gwpw\input\TestFile_11.doc", output_file=r"C:\gwpw\output\editor\protect_f2f\TestFile_11.doc", ) ``` ### Protect from file path to memory `protect_file` returns the protected file's bytes. The below example demonstrates assigning the variable `file_bytes`. We can see that after sanitisation the first 8 bytes of `file_bytes` matches the [file signature](https://en.wikipedia.org/wiki/List_of_file_signatures) for the Microsoft Compound File Binary (CFB) format, `D0 CF 11 E0 A1 B1 1A E1`. ```py import glasswall # Load the Glasswall Editor library editor = glasswall.Editor(r"C:\gwpw\libraries\10.0") # Use the default policy to sanitise a file in memory, returning the file bytes in memory file_bytes = editor.protect_file( input_file=r"C:\gwpw\input\TestFile_11.doc" ) assert file_bytes[:8] == b'\xd0\xcf\x11\xe0\xa1\xb1\x1a\xe1' ``` ### Protect from memory ```py import glasswall # Load the Glasswall Editor library editor = glasswall.Editor(r"C:\gwpw\libraries\10.0") # Read file from disk to memory with open(r"C:\gwpw\input\TestFile_11.doc", "rb") as f: input_bytes = f.read() # Use the default policy to sanitise a file file_bytes = editor.protect_file( input_file=input_bytes, ) assert file_bytes[:8] == b'\xd0\xcf\x11\xe0\xa1\xb1\x1a\xe1' ``` ### Protect files in a directory ```py import glasswall # Load the Glasswall Editor library editor = glasswall.Editor(r"C:\gwpw\libraries\10.0") # Use the default policy to protect a directory of files, writing the sanitised files to a new directory. editor.protect_directory( input_directory=r"C:\gwpw\input", output_directory=r"C:\gwpw\output\editor\protect_directory" ) ``` ### Protect files in a directory that may contain unsupported file types The default behaviour of the Glasswall Python wrapper is to raise the relevant exception (see: [glasswall.libraries.editor.errors](./8-Autogenerated%20Docs/libraries/editor/errors/errors.md)) if processing fails. Passing `raise_unsupported=False` will prevent an exception being raised and can be useful when working with a directory containing a mixture of both supported and unsupported file types when it is desirable to process as many of the files as possible instead of terminating on the first failure. ```py import glasswall # Load the Glasswall Editor library editor = glasswall.Editor(r"C:\gwpw\libraries\10.0") # Use the default policy to protect a directory of files, writing the sanitised files to a new directory. editor.protect_directory( input_directory=r"C:\gwpw\input_with_unsupported_file_types", output_directory=r"C:\gwpw\output\editor\protect_directory_unsupported", raise_unsupported=False ) ``` ### Protect files in a directory using a custom content management policy Using `glasswall.content_management.policies.Editor`: ```py import glasswall # Load the Glasswall Editor library editor = glasswall.Editor(r"C:\gwpw\libraries\10.0") # Use a custom Editor policy to sanitise all files in the input directory # and write them to the input_sanitised directory. If macros are present # in ppt or word files, the file will be marked as non-conforming and blocked. # If internal or external hyperlinks are present in word files they will not # be sanitised, and will remain in the regenerated document. editor.protect_directory( input_directory=r"C:\gwpw\input", output_directory=r"C:\gwpw\output\editor\protect_directory_custom", content_management_policy=glasswall.content_management.policies.Editor( default="sanitise", config={ "pptConfig": { "macros": "disallow", }, "wordConfig": { "internal_hyperlinks": "allow", "external_hyperlinks": "allow", "macros": "disallow", } } ) ) ``` ### Protect files in a directory conditionally based on file format The example below demonstrates processing only `.doc` and `.docx` files from a nested directory containing multiple file formats. ```py import os import glasswall # Load the Glasswall Editor library editor = glasswall.Editor(r"C:\gwpw\libraries\10.0") input_directory = r"C:\gwpw\input" output_directory = r"C:\gwpw\output\editor\protect_directory_file_format" # Iterate relative file paths from input_directory for relative_file in glasswall.utils.list_file_paths(input_directory, absolute=False): # Construct absolute paths input_file = os.path.join(input_directory, relative_file) output_file = os.path.join(output_directory, relative_file) # Get the file type of the file file_type = editor.determine_file_type( input_file=input_file, as_string=True, raise_unsupported=False ) # Protect only doc and docx files if file_type in ["doc", "docx"]: editor.protect_file(input_file, output_file) ``` --- ## Analysis An Embedded Engine report provides a detailed, file-type agnostic description of data and is logged in an XML format. The structure of this report follows an Analysis Report XSD, which is designed to simplify parsing and processing, ensuring easier integration and analysis of the data. See [Engine Reporting](/embedded-engine/embedded-engine-reporting). Files can be analysed individually from a file path or in memory using the [analyse_file](./8-Autogenerated%20Docs/libraries/editor/editor/editor.md#analyse_file) method, or all files from a directory can be analysed using the [analyse_directory](./8-Autogenerated%20Docs/libraries/editor/editor/editor.md#analyse_directory) method. ### Analyse from file path to file path ```py import glasswall # Load the Glasswall Editor library editor = glasswall.Editor(r"C:\gwpw\libraries\10.0") # Use the default policy to analyse a file, writing the analysis report to a new path editor.analyse_file( input_file=r"C:\gwpw\input\TestFile_11.doc", output_file=r"C:\gwpw\output\editor\analyse_f2f\TestFile_11.doc.xml", ) ``` ### Analyse from file path to memory `analyse_file` returns the analysis report xml file's bytes. The below example demonstrates assigning the variable `analysis_report` and checking the contents of the beginning of an Editor analysis report. ```py import glasswall # Load the Glasswall Editor library editor = glasswall.Editor(r"C:\gwpw\libraries\10.0") # Use the default policy to analyse a file analysis_report = editor.analyse_file( input_file=r"C:\gwpw\input\TestFile_11.doc", ) assert analysis_report[:500] == b'\n\n\t\n\t\t\n\t\t\t35840\n\t\t\tdoc\n\t\t\tNot Applicable\n\t\t\t9FDE85B8800C1019D2865FA298A7F75873E09870B71F9825827E354B865686A6\n\t\t\t\n\n\t\n\t\t\n\t\t\t35840\n\t\t\tdoc\n\t\t\tNot Applicable\n\t\t\t9FDE85B8800C1019D2865FA298A7F75873E09870B71F9825827E354B865686A6\n\t\t\t **Note**: `Editor` objects cannot be printed directly to inspect the policy XML. Use `create_config` for that purpose. ##### Engine defaults Calling `create_config()` with no arguments returns the engine's default policy: ```py import glasswall editor = glasswall.Editor(r"C:\gwpw\libraries\10.0") print(editor.create_config()) ``` ```xml sanitise sanitise sanitise sanitise sanitise sanitise sanitise sanitise sanitise sanitise sanitise sanitise sanitise sanitise sanitise sanitise sanitise sanitise sanitise sanitise sanitise sanitise sanitise sanitise sanitise sanitise sanitise sanitise true true true false false sisl sanitise sanitise sanitise sanitise sanitise sanitise sanitise sanitise sanitise sanitise sanitise sanitise sanitise sanitise sanitise sanitise sanitise sanitise sanitise sanitise sanitise sanitise sanitise sanitise sanitise sanitise sanitise sanitise sanitise sanitise sanitise sanitise sanitise sanitise sanitise sanitise sanitise sanitise ``` > **Note**: The switches shown above reflect the engine defaults at the time of writing. As the engine is updated, new switches may appear automatically in the output. ##### Sanitise all content switches Passing `default="sanitise"` explicitly sets every content switch to `sanitise`, overriding the engine defaults for any switch that may not already be `sanitise`: ```py import glasswall editor = glasswall.Editor(r"C:\gwpw\libraries\10.0") print(editor.create_config(default="sanitise")) ``` ##### Custom policy: allow all with targeted overrides The `config` parameter overrides individual switches on top of the `default`. In the example below, all content is allowed except macros in Word documents and embedded content in Excel spreadsheets: ```py import glasswall editor = glasswall.Editor(r"C:\gwpw\libraries\10.0") print(editor.create_config( default="allow", config={ "wordConfig": { "macros": "sanitise", }, "xlsConfig": { "embedded_files": "sanitise", "embedded_images": "sanitise", }, } )) ``` ```xml allow allow allow allow allow allow allow allow allow allow allow allow allow allow allow allow allow allow allow allow allow allow allow allow allow allow allow allow true true true false false sisl allow allow allow allow allow allow allow allow allow allow allow allow allow allow allow allow allow sanitise allow allow allow allow allow allow allow allow sanitise sanitise allow allow allow allow allow allow allow allow allow allow ``` ##### Using a previewed policy for processing The object returned by `create_config` can be passed directly to any processing function. This is useful when you want to confirm the policy XML before applying it: ```py import glasswall editor = glasswall.Editor(r"C:\gwpw\libraries\10.0") policy = editor.create_config( default="allow", config={ "wordConfig": {"macros": "sanitise"}, } ) # Inspect the policy before applying it print(policy) # Apply the policy editor.protect_directory( input_directory=r"C:\gwpw\input", output_directory=r"C:\input_sanitised", content_management_policy=policy, ) ``` --- ### WordSearch policies `WordSearch` policies configure text redaction. They can be printed directly or passed to processing functions without needing a loaded library. Elements within a content management policy may have attributes. Attributes can be set by prefixing a key with the `@` character. #### Setting a WordSearch policy ```py import glasswall # Redact instances of the string "lorem" by replacing each character # with an asterisk, and redact instances of "ipsum" by replacing each # character with the letter "X". print(glasswall.content_management.policies.WordSearch( default="allow", config={ "textSearchConfig": { "textList": [ {"name": "textItem", "switches": [ {"name": "text", "value": "lorem"}, {"name": "textSetting", "@replacementChar": "*", "value": "redact"}, ]}, {"name": "textItem", "switches": [ {"name": "text", "value": "ipsum"}, {"name": "textSetting", "@replacementChar": "X", "value": "redact"}, ]}, ] } } )) ``` ```xml xml lorem redact ipsum redact ``` ## Logging Logs are saved to the temp directory and are also output to the console with a default logging level of INFO. You can view the file path of the temp directory or the log file: ```py import glasswall print(glasswall._TEMPDIR) print(glasswall.config.logging.log_file_path) ``` ``` >>> C:\Users\\AppData\Local\Temp\glasswall >>> C:\Users\\AppData\Local\Temp\glasswall\logs\2021-03-15 122826.txt ``` The logging level can be modified. For a list of logging levels see https://docs.python.org/3/library/logging.html#logging-levels ```py import logging import glasswall # Modify logging level for logs to the console glasswall.config.logging.console.setLevel(logging.DEBUG) # Modify logging level for logs to file glasswall.config.logging.log.setLevel(logging.DEBUG) ``` ## Multiprocessing timeouts and memory limits The `GlasswallProcessManager` class is designed to manage multiprocessing with a designated timeout and memory limit for each file being processed by the Glasswall engine. The `GlasswallProcessManager` consumes `Task` objects which must be created and added to the queue. A `Task` object consists of a function that will be called, and arguments and keyword arguments that will be passed to that function. The `GlasswallProcessManager` produces either a list of `TaskResult` objects once processing has completed, or yields individual `TaskResult` objects as they are completed. A `TaskResult` object contains attributes related to the processing of the file. ### TaskResult attributes ```python task: Task success: bool # True if function did not raise an exception result: Any # function return value exception: Union[Exception, None], # the exception raised by the function exit_code: Union[int, None], # multiprocessing.Process.exitcode, 0 = success timeout_seconds: Optional[float] # time limit for each process memory_limit_in_gib: Optional[float] # memory limit for each process, 1 gibibyte = 1024 ** 3 bytes start_time: float # uses time.time(), current time in seconds since the Epoch end_time: float # uses time.time(), current time in seconds since the Epoch elapsed_time: float # end_time - start_time timed_out: bool # terminated for exceeding the time limit: 'timeout_seconds' max_memory_used_in_gib: float # the highest recorded memory usage of the process out_of_memory: bool # terminated for exceeding the memory limit: 'memory_limit_in_gib' ``` ### Producing a list of `TaskResult` objects once processing has completed In this example tasks are queued and processed in parallel up to the maximum number of workers, which by default is equal to the number of logical CPUs in the system. After all tasks are queued, processing begins automatically when exiting the `GlasswallProcessManager` context. Once all tasks are completed, the `process_manager.task_results` list attribute is populated with `TaskResult` objects that show the processing results. Once all tasks are completed, this example iterates `process_manager.task_results` in a for loop and prints each `TaskResult` object. ```py import os import time import glasswall from glasswall.multiprocessing import GlasswallProcessManager, Task INPUT_DIRECTORY = r"C:\gwpw\input" OUTPUT_DIRECTORY = r"C:\gwpw\output\editor\multiprocessing" LIBRARY_DIRECTORY = r"C:\gwpw\libraries\10.0" glasswall.config.logging.console.setLevel("CRITICAL") EDITOR = glasswall.Editor(LIBRARY_DIRECTORY) gw_policy = glasswall.content_management.policies.Editor(default="sanitise") def worker_function(*args, **kwargs): EDITOR.export_file(*args, **kwargs) def main(): start_time = time.time() input_files = glasswall.utils.list_file_paths(INPUT_DIRECTORY) with GlasswallProcessManager(max_workers=None, worker_timeout_seconds=5, memory_limit_in_gib=4) as process_manager: for input_file in input_files: relative_path = os.path.relpath(input_file, INPUT_DIRECTORY) output_file = os.path.join(OUTPUT_DIRECTORY, relative_path) + ".zip" task = Task( func=worker_function, args=tuple(), kwargs=dict( input_file=input_file, output_file=output_file, content_management_policy=gw_policy, ), ) process_manager.queue_task(task) for task_result in process_manager.task_results: print(task_result) print(f"Elapsed: {time.time() - start_time} seconds") if __name__ == "__main__": main() ``` ``` TaskResult(task=Task(func=worker_function, args=(), kwargs=(input_file='C:\\gwpw\\input\\TestFile_11.doc', outp..., success=True, result=None, exception=None, exit_code=0, timeout_seconds=5, memory_limit_in_gib=4, start_time=1710507465.3883162, end_time=1710507466.5565898, elapsed_time=1.168273687362671, timed_out=False, max_memory_used_in_gib=0.06385421752929688, out_of_memory=False) TaskResult(task=Task(func=worker_function, args=(), kwargs=(input_file='C:\\gwpw\\input\\TestFile_9.doc', outpu..., success=True, result=None, exception=None, exit_code=0, timeout_seconds=5, memory_limit_in_gib=4, start_time=1710507466.299694, end_time=1710507467.366209, elapsed_time=1.0665149688720703, timed_out=False, max_memory_used_in_gib=0.06365966796875, out_of_memory=False) TaskResult(task=Task(func=worker_function, args=(), kwargs=(input_file='C:\\gwpw\\input\\PDFWithGifAndJpeg.pdf'..., success=True, result=None, exception=None, exit_code=0, timeout_seconds=5, memory_limit_in_gib=4, start_time=1710507465.3763025, end_time=1710507467.7441902, elapsed_time=2.3678877353668213, timed_out=False, max_memory_used_in_gib=0.1662139892578125, out_of_memory=False) Elapsed: 6.226853370666504 seconds ``` ### Yielding individual `TaskResult` objects as they are completed This example uses the external library [tqdm](https://pypi.org/project/tqdm/) to visualise progress during processing. Tasks are queued and processed in parallel up to the maximum number of workers, which by default is equal to the number of logical CPUs in the system. After all tasks are queued, processing begins within the `GlasswallProcessManager` context by invoking the `process_manager.as_completed()` generator method. Once any task is completed, its corresponding `TaskResult` object is yielded. This allows results to be accessed as they become available, rather than waiting for the completion of all tasks. The `process_manager.task_results` list attribute will not be populated. As each task is completed, this example prints the yielded `TaskResult` object. ```py import os import time from tqdm import tqdm import glasswall from glasswall.multiprocessing import GlasswallProcessManager, Task INPUT_DIRECTORY = r"C:\gwpw\input" OUTPUT_DIRECTORY = r"C:\gwpw\output\editor\multiprocessing" LIBRARY_DIRECTORY = r"C:\gwpw\libraries\10.0" glasswall.config.logging.console.setLevel("CRITICAL") EDITOR = glasswall.Editor(LIBRARY_DIRECTORY) gw_policy = glasswall.content_management.policies.Editor(default="sanitise") def worker_function(*args, **kwargs): EDITOR.export_file(*args, **kwargs) def main(): start_time = time.time() input_files = glasswall.utils.list_file_paths(INPUT_DIRECTORY) with GlasswallProcessManager(max_workers=None, worker_timeout_seconds=5, memory_limit_in_gib=4) as process_manager: for input_file in tqdm(input_files, desc="Queueing files", miniters=len(input_files) // 10): relative_path = os.path.relpath(input_file, INPUT_DIRECTORY) output_file = os.path.join(OUTPUT_DIRECTORY, relative_path) + ".zip" task = Task( func=worker_function, args=tuple(), kwargs=dict( input_file=input_file, output_file=output_file, content_management_policy=gw_policy, ), ) process_manager.queue_task(task) for task_result in tqdm(process_manager.as_completed(), total=len(input_files), desc="Processing tasks", miniters=len(input_files) // 100): print(task_result) print(f"Elapsed: {time.time() - start_time} seconds") if __name__ == "__main__": main() ``` ``` Queueing files: 100%|███████████████████████████████████████████████████████████| 3/3 [00:00 Note that while the `GlasswallProcessManager` can handle large returns of data from the worker_function, holding this data in memory can quickly fill up the available RAM. When possible, it is advised not to return from the worker_function, and instead to rely on file to file processing. > If processing files to disk is undesirable or returning the file bytes from the worker function is required, we recommend the following steps: > - Limit `max_workers` to allow for at least 4 GiB of memory available for each process. > - Use the `as_completed` generator. > - Ensure file bytes are not retained after being yielded from `as_completed` so that the Python garbage collector will free up memory after the file bytes are no longer referenced. ### Yielding file bytes in file to memory mode and limiting max_workers This example uses the external library [tqdm](https://pypi.org/project/tqdm/) to visualise progress during processing. The worker_function has been modified to return the result of `EDITOR.export_file`, which will be either the export zip file's bytes, or None. The max_workers is limited based on the logical CPUs and RAM available. Tasks are queued and processed in parallel up to the specified number of workers. After all tasks are queued, processing begins within the `GlasswallProcessManager` context by invoking the `process_manager.as_completed()` generator method. Once any task is completed, its corresponding `TaskResult` object is yielded. This allows results to be accessed as they become available, rather than waiting for the completion of all tasks. The `process_manager.task_results` list attribute will not be populated. As each task is completed, this example prints the yielded `TaskResult` object, and if the `task_result.result` attribute is populated, it also prints information on the file size of the export zip file. ```py import os import time from tqdm import tqdm import glasswall from glasswall.multiprocessing import GlasswallProcessManager, Task from glasswall.multiprocessing.memory_usage import get_available_memory_gib INPUT_DIRECTORY = r"C:\gwpw\input" OUTPUT_DIRECTORY = r"C:\gwpw\output\editor\multiprocessing" LIBRARY_DIRECTORY = r"C:\gwpw\libraries\10.0" glasswall.config.logging.console.setLevel("CRITICAL") EDITOR = glasswall.Editor(LIBRARY_DIRECTORY) gw_policy = glasswall.content_management.policies.Editor(default="sanitise") def worker_function(*args, **kwargs): return EDITOR.export_file(*args, **kwargs) def main(): start_time = time.time() available_memory_gib = get_available_memory_gib() print(f"Available memory: {available_memory_gib} GiB") # Set max_workers to lowest between cpu_count or available memory // 4 (4gib per process) cpu_count = os.cpu_count() or 1 max_workers = int(min(cpu_count, available_memory_gib // 4)) print(f"Max workers: {max_workers}") input_files = glasswall.utils.list_file_paths(INPUT_DIRECTORY) with GlasswallProcessManager(max_workers=max_workers, worker_timeout_seconds=5, memory_limit_in_gib=4) as process_manager: for input_file in tqdm(input_files, desc="Queueing files", miniters=len(input_files) // 10): # No output_file specified, export_file will run in file to memory mode task = Task( func=worker_function, args=tuple(), kwargs=dict( input_file=input_file, content_management_policy=gw_policy, ), ) process_manager.queue_task(task) for task_result in tqdm(process_manager.as_completed(), total=len(input_files), desc="Processing tasks", miniters=len(input_files) // 100): print(task_result) # Do something with export zip file bytes in memory if task_result.result: print(f"Export zip file size is: {len(task_result.result)} bytes for input_file: '{task_result.task.kwargs['input_file']}'") # task_result no longer referenced and is garbage collected here, freeing up memory print(f"Elapsed: {time.time() - start_time} seconds") if __name__ == "__main__": main() ``` ``` Available memory: 13.020416259765625 GiB Max workers: 3 Queueing files: 100%|█████████████████████████████████████████████████████████████████████| 3/3 [00:00 str: """ Retrieve the Glasswall Session Process error message. Args: session (int): The session integer. Returns: error_message (str): The Glasswall Session Process error message. """ ``` **Returns** An error message as a string. **Synopsis** ```jsx /** * This function retrieves the error message reported by Glasswall. If more than one error * is reported, the last one will be returned. * */ GW2FileErrorMsg( session, errorMsgBuffer, errorMsgBufferLength) ``` **Returns** Returns an integer `GW2_RetStatus` enum value. Negative numbers indicate a failure. **Example** ```jsx function getFileErrorMsg(session_id, gw) { /* GW2FileErrorMsg API signature int GW2FileErrorMsg( Session session, char **errorMsgBuffer, size_t *errorMsgBufferLength ); */ // allocate pointer space to store the pointer to the message buffer let CString_ptr = ref.refType(ref.types.CString); let errorMsgBuffer = ref.alloc(CString_ptr); // allocate space to store the buffer length; use .deref() to extract it let errorMsgBufferLength = ref.alloc('size_t'); let rv = gw.GW2FileErrorMsg(session_id, errorMsgBuffer, errorMsgBufferLength); buf_len = errorMsgBufferLength.deref(); if (buf_len == 0) arr_buf = ""; else arr_buf = buffer_to_array(errorMsgBuffer, errorMsgBufferLength); let message = `\n GW2FileErrorMsg:return=${rv}`; message += "\n errorMsgBuffer = \"" + arr_buf.toString() + '"'; message += `\n errorMsgBufferLength = ${buf_len}`; return message; } ``` import Tabs from '@theme/Tabs' import TabItem from '@theme/TabItem' The **GW2FileSessionStatus** function provides a string that describes, at a high level, the processing carried out on the last file processed in the supplied session. ```cpp #include "glasswall.core2.api.h" int GW2FileSessionStatus( Session session, int *glasswallSessionStatus, char **statusMsgBuffer, size_t *statusBufferLength); ``` **Parameters**   **session** The ID of the session as returned by `GW2OpenSession`   **glasswallSessionStatus** An integer output parameter describing the return status of the session as it relates to the last file that was processed. See the [Return Types](/embedded-engine/embedded-engine-api-overview#return-types) table for an explanation of `glasswallSessionStatus`.   **statusMsgBuffer** A string pointer output parameter which is populated with a high level description of the processing carried out on the last file processed by the session. Eg `"Sanitisation Applied, Remedies Applied"`. The memory used by this pointer does **not** need to be freed by the user.   **statusBufferLength** An output parameter which is populated with the size in bytes of the memory pointed to by `statusMsgBuffer`. **Returns**   Returns an integer indicating whether the function call was successful. Negative numbers indicate a failure. See the [Return Types](/embedded-engine/embedded-engine-api-overview#return-types) table for an explanation of the return codes. If `session` is invalid `glasswallSessionStatus`, `statusMsgBuffer` and `statusBufferLength` will be undefined. **Synopsis** ```csharp /// Current open Glasswall session /// Session status value /// String that describes, at a high level, the processing carried out on the last document /// Size of the outputBuffer public int FileSessionStatus( int session, out IntPtr gwSessionStatus, out IntPtr outputBuffer, ref UIntPtr bufferLength) ``` **Returns** If **session** is invalid, it will return -1. If session is valid, the function returns 0 and **gwSessionStatus**, **outputBuffer** and **bufferLength** will be populated. **Synopsis** ```java import com.glasswall.core2javabridge.*; public FileSessionStatus GW2FileSessionStatusResult(int session) throws GlasswallException ``` **Note** This functionality previously required three separate function calls to retrieve the ID data. This has now been streamlined to return the settings as a `FileSessionStatus` object. The three original functions have been deprecated. **Returns** The **GW2FileSessionStatusResult** function returns a `FileSessionStatus` object containing the `session` summary status, and the `session` summary description. The `FileSessionStatus` object contains two variables that describe the status of `session` at the time the function is called. These are: | Variable | Description | |-------------------------------|------------------------------------------------| |```int summaryStatus``` | Status indicating the overall work carried out | |```String summaryDescription```| Description of the overall work carried out | A **GlasswallException** exception will be thrown if ```session``` is invalid, or if the session status information could not be retrieved. **Synopsis - Deprecated Functions** ```java import com.glasswall.core2javabridge.*; (Deprecated) public int GW2FileSessionStatus(int session) throws GlasswallException public int GetFileSessionStatusInt(int session) throws GlasswallException public byte[] GetStatusBuffer(int session) throws GlasswallException ``` **Description** The **GW2FileSessionStatus** function returns a status indicating whether the session status information could be retrieved for the session specified by `session`. Retrieve the session summary status through use of the **GetFileSessionStatusInt** function. Retrieve the session summary status data through use of the **GetStatusBuffer** function. **Returns - Deprecated Functions** The **GW2FileSessionStatus** function returns a **GW2_RetStatus** enumeration converted to `int`. The value will be negative if an error occurred. `0` indicates success. Refer to the API Overview/Return types for details. The **GetStatusBuffer** function returns a byte array buffer containing a summary description about the given session. This will be null if **GW2FileSessionStatus** has not been called. The **GetFileSessionStatusInt** function returns a status indicating the overall work carried out. A **GlasswallException** exception will be thrown if `session` is invalid, or if `session` status information could not be retrieved. **Synopsis** Retrieves the Glasswall session status message. Gives a high level indication of the processing that was carried out. ```py def file_session_status_message(self, session: int, raise_unsupported: bool = True) -> str: """ Retrieves the Glasswall session status message. Gives a high level indication of the processing that was carried out. Args: session (int): The session integer. raise_unsupported (bool, optional): Default True. Raise exceptions when Glasswall encounters an error. Fail silently if False. Returns: result.message (str):The file session status message. """ ``` **Returns** A file session status message as a string. **Synopsis** ```jsx /** * This function retrieves the Glasswall Session Status. This status gives a high level indication of the processing * that was carried out on the last document processed by the library */ GW2FileSessionStatus(session, glasswallSessionStatus, statusMsgBuffer, statusBufferLength) ``` **Returns** If **session** is invalid, it will return -1 and **glasswallSessionStatus**, **statusMsgBuffer** and **statusBufferLength** will be undefined. If session is valid, the function returns 0 and **glasswallSessionStatus**, **statusMsgBuffer** and **statusBufferLength** will be populated. **Example** ```jsx function getFileSessionStatus(session_id, gw) { /* Glasswall API signature: int GW2FileSessionStatus( Session session, int *glasswallSessionStatus, char **statusMsgBuffer, size_t *statusBufferLength ); */ // allocate space to store the session status; use .deref() to extract it let glasswallSessionStatus = ref.alloc('int'); let CString_ptr = ref.refType(ref.types.CString); let statusMsgBuffer = ref.alloc(CString_ptr); // allocate space to store the buffer length; use .deref() to extract it let statusBufferLength = ref.alloc('size_t'); let rv = gw.GW2FileSessionStatus(session_id, glasswallSessionStatus, statusMsgBuffer, statusBufferLength); arr_buf = buffer_to_array(statusMsgBuffer, statusBufferLength) // console.log(arr_buf.toString()); let message = `\n GW2FileSessionStatus:return=${rv}`; message += "\n glasswallSessionStatus = " + glasswallSessionStatus.deref(); message += "\n statusMsgBuffer = \"" + arr_buf.toString() + '"'; message += "\n statusBufferLength = " + statusBufferLength.deref(); return message; } ``` import Tabs from '@theme/Tabs' import TabItem from '@theme/TabItem' The **GW2GetPolicySettings** function provides a string corresponding to the contents of the policy settings file associated with the supplied **session**. ```cpp #include "glasswall.core2.api.h" int GW2GetPolicySettings ( Session session, char **policiesBuffer, size_t *policiesLength, Policy_format format); ``` **Parameters**   **session** The ID of the session as returned by `GW2OpenSession`   **policiesBuffer** A string pointer output parameter which is populated with the policy settings information. The memory used by this pointer does **not** need to be freed by the user. In the event of a failure this parameter may be set to nullptr.   **policiesLength** An output parameter which is populated with the size in bytes of the memory pointed to by `policiesBuffer`   **format** The format of the data. This must be `PF_XML`. **Returns**   Returns an integer indicating whether the function call was successful. Negative numbers indicate a failure. See the [Return Types](/embedded-engine/embedded-engine-api-overview#return-types) table for an explanation of the return codes. **Synopsis** ```csharp /// /// Retrieves policy settings for the session /// /// Current open Glasswall session /// A pointer to the object containing a pointer pointing to the policy data /// A pointer to a object containing the size in bytes public int GetPolicySettings( int session, out IntPtr policiesBufferPtr, ref UIntPtr policiesLengthPtr, int format) ``` **Returns** Returns an integer `GW2_RetStatus` enum value. Negative numbers indicate a failure. **Synopsis** ```java import com.glasswall.core2javabridge.*; public String GW2GetPolicySettingsString(int session, int format) throws GlasswallException ``` **Note** The **GW2GetPolicySettingsString** function outputs the currently registered policy settings for the session specified by `session`. Refer to API Overview/Return types for valid enumerators for ```format```. This functionality previously required two separate function calls to retrieve the policy setting data. This has now been streamlined to return the settings as a `String`. The two original functions have been deprecated. **Returns** The **GW2GetPolicySettingsString** function returns a `String` containing the policy settings. A **GlasswallException** exception will be thrown if `session` is invalid, or if the policy settings could not be retrieved. **Synopsis - Deprecated Functions** ```java import com.glasswall.core2javabridge.*; (Deprecated) public int GW2GetPolicySettings(int session, int format) throws GlasswallException public byte[] GetPolicyBuffer(int session) throws GlasswallException ``` **Description - Deprecated Functions** The **GW2GetPolicySettings** function outputs the currently registered policy settings for the session specified by `session` to the internal policy buffer. Retrieve this data through use of **GetPolicyBuffer** function. Refer to API Overview/Return types for valid enumerators for ```format```. **Returns - Deprecated Functions** The **GW2GetPolicySettings** function returns a **GW2_RetStatus** enumeration converted to `int`. The value will be negative if an error occurred. `0` indicates success. Refer to the API Overview/Return types for details. The **GetPolicyBuffer** returns a `byte[]` containing the policy settings. This will be null if **GW2GetPolicySettings** has not been called. A **GlasswallException** exception will be thrown if `session` is invalid, or if the policy settings could not be retrieved. **Synopsis** Returns the content management configuration for a given session. ```py def get_content_management_policy(self, session: int): """ Returns the content management configuration for a given session. Args: session (int): The session integer. Returns: xml_string (str): The XML string of the current content management configuration. """ ``` **Returns** xml_string (str): The XML string of the current content management configuration **Synopsis** ```jsx /** * This function returns the policy settings used for the specified session * * @param {number} session The ID of the session. * @param {string} policiesBuffer The pointer to the policy buffer. * @param {number} policiesLength The size of the data in the policy buffer * @param {number} format The format of the policy. */ GW2GetPolicySettings( session, policiesBuffer, policiesLength, format) ``` **Returns** Returns an integer `GW2_RetStatus` enum value. Negative numbers indicate a failure. **Example** ```jsx const ref = require('ref-napi'); ... function buffer_to_string(buffer, buffer_size) { if (!buffer.isNull() && ref.deref(buffer_size) > 0) { return Buffer.from(ref.reinterpret(ref.deref(buffer), ref.deref(buffer_size), 0)).toString(); } else { return ""; } } ... let policy_file_buffer = ref.alloc(ref.refType(ref.types.CString)); let policy_buffer_size = ref.alloc(ref.types.size_t, 0); let return_status = gw.GW2GetPolicySettings(session_id, policy_file_buffer, policy_buffer_size, 0); let xml_string = buffer_to_string(policy_file_buffer, policy_buffer_size); ... ``` import Tabs from '@theme/Tabs' import TabItem from '@theme/TabItem' The **GW2LicenceDetails** function returns the details of a licence loaded in for a **session**, including the number of days remaining and what entitlements are enabled. ```cpp #include "glasswall.core2.api.h" const char * GW2LicenceDetails(Session session); ``` **Parameters**   **session** The ID of the session as returned by `GW2OpenSession` **Returns**   A pointer to a constant string describing the details of the licence for the session, or the string `"Unknown Session ID"` if `session` is invalid or the licence details could not be retrieved. The memory of the returned string does not need to be freed by the user, however further calls to `GW2LicenceDetails` will invalidate the returned string. **Synopsis** The **GW2LicenceDetails** method returns a pointer to a constant string describing the details of the licence. ```csharp /// Current open Glasswall session public string LicenceDetails(int session) ``` **Returns** Returns a string describing the details of the licence loaded for the session. **Synopsis** ```java import com.glasswall.core2javabridge.*; public String GW2LicenceDetails(int session) throws GlasswallException ``` **Returns** The **GW2LicenceDetails** function returns a `String` containing the licence details for the session specified by `session`. A **GlasswallException** exception will be thrown if `session` is invalid, or the licence details could not be retrieved. **Synopsis** Returns a string containing details of the licence. ```py def licence_details(self): """ Returns a string containing details of the licence. Returns: result (str): A string containing details of the licence. """ ``` **Returns** A string containing details of the licence, for example: ``` Unlimited Licence Entitlements: Engine_Main Engine_Protect Engine_Analysis Engine_ExportImport ``` **Synopsis** The **GW2LicenceDetails** function returns a pointer to a constant string describing the details of the licence. ```javascript * @param {number} session The ID of the session. * @returns {string} The details of the licence currently associated with this session. GW2LicenceDetails(session) ``` import Tabs from '@theme/Tabs' import TabItem from '@theme/TabItem' The **GW2CloseSession** function closes the session designated by **session**. All resources allocated by the session will be destroyed. Only data stored to persistent storage can be processed by the client after a successful call to **GW2CloseSession**. **Synopsis** ```cpp #include "glasswall.core2.api.h" int GW2CloseSession(Session session); ``` **Returns** The **GW2CloseSession** function returns a **GW2_RetStatus** enumeration converted to `int`. The value will be negative if an error occurred. `0` indicates success. Refer to the API Overview/Return types for details. **Synopsis** ```csharp public int CloseSession(int session) ``` **Returns** Returns an integer `GW2_RetStatus` enum value. Negative numbers indicate a failure. **Synopsis** ```java import com.glasswall.core2javabridge.*; public int GW2CloseSession(int session) ``` **Returns** The **GW2CloseSession** function returns a **GW2_RetStatus** enumeration converted to **int**. The value will be negative if an error occurred. `0` indicates success. Refer to the API Overview/Return types for details. **Synopsis** ```py def close_session(self, session: int) -> int: """ Close the Glasswall session. All resources allocated by the session will be destroyed. Args: session (int): The session to close. Returns: status (int): The status code of the function call. """ ``` **Returns** Returns an integer `GW2_RetStatus` enum value. Negative numbers indicate a failure. **Synopsis** ```jsx GW2CloseSession(session) ``` **Returns** Returns an integer `GW2_RetStatus` enum value. Negative numbers indicate a failure. import Tabs from '@theme/Tabs' import TabItem from '@theme/TabItem' The **GW2DetermineFileTypeFromFile** function determines the file type for the given file. If it cannot discern the file type, it returns the value of the enumeration constant **ft_unknown** from the enumerated type **ft_t** ; otherwise, it returns a value from the enumerated type **ft_t** indicating the file type. ```cpp #include "glasswall.core2.api.h" ft_t GW2DetermineFileTypeFromFile(const char *path); ``` **Parameters**   **path** The absolute file path to the file you wish to examine. **Returns**   Enumerated type `ft_t` indicating the determined file type. If the file type cannot be determined then `ft_unknown` will be returned. **Example** ```cpp #include "glasswall.core2.api.h" #include “filetype.h” ft_t file_type = GW2DetermineFileTypeFromFile("c:\\dir1\\dir2\\filename"); char *type = NULL; switch (file_type) /* categorise them */ { case ft_doc: case ft_docx: case ft_ppt: case ft_pptx: case ft_xls: case ft_xlsx: type = "Microsoft Office"; break; case ft_png: case ft_jpg: case ft_gif: case ft_tiff: type = "Image file"; break; case ft_pdf: type = "PDF"; break; default: type = "Unrecognised file type"; break; } ``` **Synopsis** ```csharp /// /// Returns a value indicating the the file type from an input file by Glasswall. /// /// The file path to the input file /// A value indicating the determined file type public int DetermineFileTypeFromFile(string inputFilePathName) ``` **Returns** A value indicating the determined file type - A list of the enumeration constants is in the C source header file **filetype.h** are consistent with the returned value for this function. **Synopsis** ```java import com.glasswall.core2javabridge.*; public int GW2DetermineFileTypeFromFile(String filePath) throws GlasswallException, NullPointerException (Deprecated) public int GW2DetermineFileTypeFromFile(byte[] BYTE) throws GlasswallException, NullPointerException ``` **Note** The **GW2DetermineFileTypeFromFile** function parameters have been updated to use `String` in place of `byte[]`. The original function has been deprecated. **Returns** The **GW2DetermineFileTypeFromFile** function returns an enumerator of type `Ft_t` converted to an `int`. A list of the enumeration constants may be found in the C source header file **filetype.h**. A **NullPointerException** exception will be thrown if `filePath` is null or empty. A **GlasswallException** exception will be thrown if `filePath` cannot be retrieved. **Synopsis** Determine the file type of a given input file, either as an integer identifier or a string. ```py def determine_file_type(self, input_file: Union[str, bytes, bytearray, io.BytesIO], as_string: bool = False, raise_unsupported: bool = True) -> Union[int, str]: """ Determine the file type of a given input file, either as an integer identifier or a string. Args: input_file (Union[str, bytes, bytearray, io.BytesIO]): The input file to analyse. It can be provided as a file path (str), bytes, bytearray, or a BytesIO object. as_string (bool, optional): Return file type as string, eg: "bmp" instead of: 29. Defaults to False. raise_unsupported (bool, optional): Default True. Raise exceptions when Glasswall encounters an error. Fail silently if False. Returns: file_type (Union[int, str]): The file type. """ ``` **Returns** A value or extension string indicating the determined file type - A list of the enumeration constants is in the C source header file **filetype.h** are consistent with the returned value for this function. **Synopsis** ```jsx /** * This function determines the file type for a given file provided it is supported by Glasswall. Notice this is not related to a session. * @param {string} path The path to the provided file. * @returns {number} The determined file type, as enumerated in the C source header file filetype.h. */ GW2DetermineFileTypeFromFile(path) ``` **Returns** A value indicating the determined file type - A list of the enumeration constants is in the C source header file **filetype.h** are consistent with the returned value for this function. import Tabs from '@theme/Tabs' import TabItem from '@theme/TabItem' The **GW2DetermineFileTypeFromMemory** function determines the file type of the file whose content is in the buffer held in memory. **Synopsis** ```cpp #include "glasswall.core2.api.h" ft_t GW2DetermineFileTypeFromFile(const char *inputFileBuffer, size_t inputLength); ``` **Returns** If **GW2DetermineFileTypeFromMemory** cannot discern the file type, it returns the value of the enumeration constant **ft\_unknown** from the enumerated type **ft\_t** ; otherwise, it returns a value from the enumerated type **ft\_t** indicating the file type. A list of the enumeration constants is in the C source header file **filetype.h**. **Synopsis** ```csharp /// /// Returns a value indicating the the file type from an input file by Glasswall. /// /// The input buffer containing the file to be determined /// A value indicating the determined file types public int DetermineFileTypeFromMemory(byte[] inputFileBuffer) ``` **Returns** A value indicating the determined file type - A list of the enumeration constants is in the C source header file **filetype.h** are consistent with the returned value for this function. **Synopsis** ```java import com.glasswall.core2javabridge.*; public int GW2DetermineFileTypeFromMemory(byte[] fileBuffer) throws GlasswallException, NullPointerException public int GW2DetermineFileTypeFromMemory(byte[] fileBuffer, int length) throws GlasswallException, NullPointerException ``` **Note** The `length` of the `fileBuffer` may optionally be specified. **Returns** The **GW2DetermineFileTypeFromMemory** function returns an enumerator of type `ft_t` converted to an `int`. A list of the enumeration constants may be found in the C source header file **filetype.h**. A **NullPointerException** exception will be thrown if `fileBuffer` is null or empty. A **GlasswallException** exception will be thrown if `fileBuffer` cannot be retrieved. **Synopsis** Determine the file type of a given input file, either as an integer identifier or a string. ```py def determine_file_type(self, input_file: Union[str, bytes, bytearray, io.BytesIO], as_string: bool = False, raise_unsupported: bool = True) -> Union[int, str]: """ Determine the file type of a given input file, either as an integer identifier or a string. Args: input_file (Union[str, bytes, bytearray, io.BytesIO]): The input file to analyse. It can be provided as a file path (str), bytes, bytearray, or a BytesIO object. as_string (bool, optional): Return file type as string, eg: "bmp" instead of: 29. Defaults to False. raise_unsupported (bool, optional): Default True. Raise exceptions when Glasswall encounters an error. Fail silently if False. Returns: file_type (Union[int, str]): The file type. """ ``` **Returns** A value or extension string indicating the determined file type - A list of the enumeration constants is in the C source header file **filetype.h** are consistent with the returned value for this function. **Synopsis** ```jsx /** * This function determines the file type for a given file provided it is supported by Glasswall. Notice this is not related to a session. * @param {string} path The path to the provided file. * @returns {number} The determined file type, as enumerated in the C source header file filetype.h. */ GW2DetermineFileTypeFromMemory(inputFileBuffer, inputLength) ``` **Returns** A value indicating the determined file type - A list of the enumeration constants is in the C source header file **filetype.h** are consistent with the returned value for this function. import Tabs from '@theme/Tabs' import TabItem from '@theme/TabItem' The **GW2DetermineFileTypeAndReport** function determines the file type for a given file and produces a report. This is not related to a session. ```cpp #include "glasswall.core2.api.h" ft_t GW2DetermineFileTypeAndReport(const char *inputFileBuffer, size_t inputLength, const char *report) ``` **Parameters**   **inputFileBuffer** The memory location containing the file to be processed by this function.   **inputLength** The length of `inputFileBuffer` in bytes   **report** The path to a plain text file where the report will be written. If the file doesn't exist it will be created. **Returns**   Enumerated type `ft_t` indicating the determined file type. If the file type cannot be determined then `ft_unknown` will be returned. **Synopsis** **Returns** **Synopsis** ```java public int GW2DetermineFileTypeAndReport(byte[] fileBuffer, String reportFilePath) throws GlasswallException, NullPointerException public int GW2DetermineFileTypeAndReport(byte[] fileBuffer, int length, String reportFilePath) throws GlasswallException, NullPointerException ``` **Description** The **GW2DetermineFileTypeAndReport** function determines the file type of the file held in the byte array `fileBuffer` and writes out a summary XML report to the path specified in `reportFilePath`. The `length` of the `fileBuffer` may optionally be specified. **Returns** The **GW2DetermineFileTypeAndReport** function returns an integer indicating the determined file type. A list of the enumeration constants is in the C source header file **filetype.h**. A return value of 0 indicates the file type can't be determined. If successful, an XML report will be written to the path specified in `reportFilePath`. A **NullPointerException** exception will be thrown if `fileBuffer` or `reportFilePath` buffers are null or empty. A **GlasswallException** exception will be thrown if `fileBuffer` cannot be retrieved, or if the `reportFilePath` could not be converted to UTF-8. **Synopsis** **Returns** **Synopsis** **Returns** import Tabs from '@theme/Tabs' import TabItem from '@theme/TabItem' File type IDs are mapped to file types supported by the Glasswall Engine. This API provides the formal name of the file type in string format given a file type ID number. ```cpp #include "glasswall.core2.api.h" int GW2GetFileType( Session session, size_t fileId, size_t *bufferLength, char **outputBuffer); ``` **Parameters**   **session** The ID of the session as returned by `GW2OpenSession`   **fileId** An integer representing the file type ID   **bufferLength** An output parameter which is populated with size in bytes of the memory pointed to by `outputBuffer`.   **outputBuffer** A string pointer output parameter which is populated with the formal name of the file type associated with the supplied file ID, eg. `"BMP Image"`. The memory used by this pointer does **not** need to be freed by the user. **Returns**   Returns an integer indicating whether the function call was successful. Negative numbers indicate a failure. See the [Return Types](/embedded-engine/embedded-engine-api-overview#return-types) table for an explanation of the return codes. If successful the `outputBuffer` is populated with the formal name associated with the file type. **Example** ```cpp #include "glasswall.core2.api.h" #include "filetype.h" // ft_t enum which includes ft_bmp char *outbuf = NULL; size_t buflen = 0; if (GW2OpenSession() **Synopsis** ```csharp public int GW2GetFileType( int session, uint fileID, ref UIntPtr bufferLength, out IntPtr outputBuffer) ``` **Returns** Returns an integer `GW2_RetStatus` enum value. Negative numbers indicate a failure. If successful, the output buffer is populated with the formal name of the file type. **Example** ```csharp using glasswall_core2; ... Glasswall glasswall = new Glasswall(); // Instance of the Glasswall wrapper int session = glasswall.OpenSession(); int returnStatus = glasswall.GW2GetFileType(session, 22, ref bufferLength, out buffer); if (bufferLength >= 0) { byte[] msgArray = glasswall.CreateArrayFromBuffer(buffer, bufferLength); // Formal name for file type ID 22 now stored in a byte array } if (glasswall.CloseSession(session)) { // Error Handling } ``` **Synopsis** ```java import com.glasswall.core2javabridge.*; public String GW2GetFileTypeString(int session, int fileId) throws GlasswallException ``` **Note** This functionality previously required two separate function calls to retrieve the file type data. This has now been streamlined to return the settings as a `String`. The two original functions have been deprecated. **Returns** The **GW2GetFileTypeString** function returns a `String` representation of a filetype ID. A **GlasswallException** exception will be thrown if `session` is invalid, or if the file type could not be retrieved. **Synopsis - Deprecated Functions** ```java import com.glasswall.core2javabridge.*; (Deprecated) public int GW2GetFileType(int session, int fileId) throws GlasswallException public byte[] GetFileTypeBuffer(int session) throws GlasswallException ``` **Description - Deprecated Functions** The **GW2GetFileType** function outputs a representation of `fileId` for the session specified by `session`, to the internal FileType Buffer. Retrieve this data through use of the **GetFileTypeBuffer** function. **Returns - Deprecated Functions** The **GW2GetFileType** function returns a **GW2_RetStatus** enumeration converted to `int`. The value will be negative if an error occurred. `0` indicates success. Refer to the API Overview/Return types for details. The **GetFileTypeBuffer** function returns a UTF-8 encoded `byte[]` containing a representation of a filetype ID. This will be null if **GW2GetFileType** has not been called. A **GlasswallException** exception will be thrown if `session` is invalid, or if the file type could not be retrieved. **Synopsis** Retrieve information about a file type based on its identifier. ```py def get_file_type_info(self, file_type: Union[str, int]): """ Retrieve information about a file type based on its identifier. Args: file_type (Union[str, int]): The file type identifier. This can be either a string representing a file extension (e.g. 'bmp') or an integer corresponding to a file type (e.g. 29). Returns: - file_type_info (Union[int, str]): Depending on the input 'file_type': - If `file_type` is a string (e.g. 'bmp'): - If the file type is recognised, returns an integer corresponding to that file type. - If the file type is not recognised, returns 0. - If `file_type` is an integer (e.g. 29): - If the integer corresponds to a recognised file type, returns a more detailed string description of the file type (e.g. 'BMP Image'). - If the integer does not match any recognised file type, returns an empty string. """ ``` **Returns** Returns an integer or string depending on the input 'file_type'. This value is either an integer of a recognised file type, a detailed string description of a recognised file type or if the input 'file_type' was not recognised, 0 or an empty string. **Synopsis** This function places a pointer to a description of a specified fileID in a specified location. ```jsx /** * * @param {number} session The ID of the session. * @param {number} fileID The ID of the file type. * @param {number} bufferLength The length of the buffer. * @param {string} outputBuffer The location of the output buffer. */ GW2GetFileType( session, fileID, bufferLength, outputBuffer) ``` **Returns** Returns an integer `GW2_RetStatus` enum value. Negative numbers indicate a failure. If successful, the output buffer is populated with the formal name of the file type. **Example** ```jsx const ref = require('ref-napi'); ... function buffer_to_string(buffer, buffer_size) { if (!buffer.isNull() && ref.deref(buffer_size) > 0) { return Buffer.from(ref.reinterpret(ref.deref(buffer), ref.deref(buffer_size), 0)).toString(); } else { return ""; } } ... output_file_buffer = ref.alloc(ref.refType(ref.types.CString)); output_buffer_size = ref.alloc(ref.types.size_t, 0); return_status = gw.GW2GetFileType(session_id, 22, output_buffer_size, output_file_buffer); let error_description = buffer_to_string(output_file_buffer, output_buffer_size); ... ``` import Tabs from '@theme/Tabs' import TabItem from '@theme/TabItem' Every file type has an associated ID in the Glasswall Engine. This API provides the Glasswall file type ID in string format for the associated file extension supplied. ```cpp #include "glasswall.core2.api.h" int GW2GetFileTypeID( Session session, const char *fileType, size_t *bufferLength, char **outputBuffer); ``` **Parameters**   **session** The ID of the session as returned by `GW2OpenSession`   **fileType** A string containing the file extension. Eg `"bmp"`.   **bufferLength** An output parameter which is populated with size in bytes of the memory pointed to by `outputBuffer`.   **outputBuffer** A string pointer output parameter which is populated with the file type ID associated with the supplied extension. The memory used by this pointer does **not** need to be freed by the user. **Returns**   Returns an integer indicating whether the function call was successful. Negative numbers indicate a failure. See the [Return Types](/embedded-engine/embedded-engine-api-overview#return-types) table for an explanation of the return codes. If successful, the `outputBuffer` is populated with the file type ID. **Example** ```cpp #include "glasswall.core2.api.h" char *outbuf = NULL; size_t buflen = 0; if (GW2OpenSession() **Synopsis** ```csharp public int GetFileTypeID( int session, string fileType, ref UIntPtr bufferLength, out IntPtr outputBuffer) ``` **Returns** Returns an integer `GW2_RetStatus` enum value. Negative numbers indicate a failure. If successful, the output buffer is populated with the file type ID. **Example** ```csharp using glasswall_core2; ... Glasswall glasswall = new Glasswall(); // Instance of the Glasswall wrapper int session = glasswall.OpenSession(); int returnStatus = glasswall.GetFileTypeID(session, "pdf", ref bufferLength, out buffer); if (bufferLength >= 0) { byte[] msgArray = glasswall.CreateArrayFromBuffer(buffer, bufferLength); // File type ID for PDF is now stored in a byte array } if (glasswall.CloseSession(session)) { // Error Handling } ``` **Synopsis** **Not supported in Java** **Synopsis** Retrieve information about a file type based on its identifier. ```py def get_file_type_info(self, file_type: Union[str, int]): """ Retrieve information about a file type based on its identifier. Args: file_type (Union[str, int]): The file type identifier. This can be either a string representing a file extension (e.g. 'bmp') or an integer corresponding to a file type (e.g. 29). Returns: - file_type_info (Union[int, str]): Depending on the input 'file_type': - If `file_type` is a string (e.g. 'bmp'): - If the file type is recognised, returns an integer corresponding to that file type. - If the file type is not recognised, returns 0. - If `file_type` is an integer (e.g. 29): - If the integer corresponds to a recognised file type, returns a more detailed string description of the file type (e.g. 'BMP Image'). - If the integer does not match any recognised file type, returns an empty string. """ ``` **Returns** Returns an integer or string depending on the input 'file_type'. This value is either an integer of a recognised file type, a detailed string description of a recognised file type or if the input 'file_type' was not recognised, 0 or an empty string. **Synopsis** This function places a pointer to a description of a specified fileType in a specified location. ```jsx /** * * @param {number} session The ID of the session. * @param {string} fileType The extension of the file type. * @param {number} bufferLength The length of the buffer. * @param {string} outputBuffer The location of the output buffer. */ GW2GetFileTypeID( session, fileType, bufferLength, outputBuffer) ``` **Returns** Returns an integer `GW2_RetStatus` enum value. Negative numbers indicate a failure. If successful, the output buffer is populated with the file type ID. **Example** ```jsx const ref = require('ref-napi'); ... function buffer_to_string(buffer, buffer_size) { if (!buffer.isNull() && ref.deref(buffer_size) > 0) { return Buffer.from(ref.reinterpret(ref.deref(buffer), ref.deref(buffer_size), 0)).toString(); } else { return ""; } } ... output_file_buffer = ref.alloc(ref.refType(ref.types.CString)); output_buffer_size = ref.alloc(ref.types.size_t, 0); return_status = gw.GW2GetFileTypeID(session_id, "pdf", output_buffer_size, output_file_buffer); let error_description = buffer_to_string(output_file_buffer, output_buffer_size); ... ``` import Tabs from '@theme/Tabs' import TabItem from '@theme/TabItem' Every issue identified and reported by the Glasswall engine has a unique issue ID associated. This API provides a description for a given Issue ID number. **Synopsis** For session **session** , **GW2GetIdInfo** places in the object pointed to by **outputBuffer** , a pointer to a description of Glasswall Issue ID **issueId**. The length of the description, in bytes, is placed in the **size_t** object pointed to by **bufferLength**. ```cpp #include "glasswall.core2.api.h" int GW2GetIdInfo( Session session, size_t issueId, size_t *bufferLength, char **outputBuffer); ``` **Returns** Returns an integer `GW2_RetStatus` enum value. Negative numbers indicate a failure. If successful, the output buffer is populated with the Issue Description. **Example** ```cpp #include "glasswall.core2.api.h" char *outbuf = NULL; size_t buflen = 0; if (GW2OpenSession() **Synopsis** ```csharp public int GetIdInfo( int session, uint IssueID, ref UIntPtr bufferLength, out IntPtr outputBuffer) ``` **Returns** Returns an integer `GW2_RetStatus` enum value. Negative numbers indicate a failure. If successful, the output buffer is populated with the Issue Description. **Example** ```csharp using glasswall_core2; ... Glasswall glasswall = new Glasswall(); // Instance of the Glasswall wrapper int session = glasswall.OpenSession(); int returnStatus = glasswall.GetIdInfo(session, 96, ref bufferLength, out buffer); if (bufferLength >= 0) { byte[] msgArray = glasswall.CreateArrayFromBuffer(buffer, bufferLength); // Error description for issue ID 96 now stored in a byte array } if (glasswall.CloseSession(session)) { // Error Handling } ``` **Synopsis** ```java import com.glasswall.core2javabridge.*; public String GW2GetIdInfoString(int session, int issueId) throws GlasswallException ``` **Description** This functionality previously required two separate function calls to retrieve the ID data. This has now been streamlined to return the settings as a `String`. The two original functions have been deprecated. **Returns** The **GW2GetIdInfoString** function returns a `String` containing a description of the given `issueId` A **GlasswallException** exception will be thrown if `session` is invalid, or if the `issueId` description could not be retrieved. **Synopsis - Deprecated Functions** ```java import com.glasswall.core2javabridge.*; (Deprecated) public int GW2GetIdInfo(int session, int issueId) throws GlasswallException public byte[] GetIDBuffer(int session) throws GlasswallException ``` **Description - Deprecated Functions** The **GW2GetIdInfo** function outputs the description of a given `issueId` for a given `session` to the internal ID Buffer. Retrieve this data through use of **GetIDBuffer** function. **Returns - Deprecated Functions** The **GW2GetIdInfo** function returns a **GW2_RetStatus** enumeration converted to `int`. The value will be negative if an error occurred. `0` indicates success. Refer to the API Overview/Return types for details. **GetIDBuffer** returns a `byte[]` containing the ID description. This will be null if **GetIDBuffer** has not been called. A **GlasswallException** exception will be thrown if ```session``` is invalid, or if the issue description could not be retrieved. **Synopsis** Retrieves the group description for the given Issue ID. ```py def get_id_info(self, issue_id: int, raise_unsupported: bool = True): """ Retrieves the group description for the given Issue ID. e.g. issue_id 96 returns "Document Processing Instances" Args: issue_id (int): The issue id. raise_unsupported (bool, optional): Default True. Raise exceptions when Glasswall encounters an error. Fail silently if False. Returns: id_info (str): The group description for the given Issue ID. """ ``` **Returns** A string, the group description for the given Issue ID. **Synopsis** This function places a pointer to a description of a specified IssueID in a specified location. ```jsx /** * * @param {number} session The ID of the session. * @param {number} issueId The ID of the issue. * @param {number} bufferLength The length of the buffer. * @param {string} outputBuffer The location of the output buffer. */ GW2GetIdInfo( session, issueId, bufferLength, outputBuffer) ``` **Returns** Returns an integer `GW2_RetStatus` enum value. Negative numbers indicate a failure. If successful, the output buffer is populated with the Issue Description. **Example** ```jsx const ref = require('ref-napi'); ... function buffer_to_string(buffer, buffer_size) { if (!buffer.isNull() && ref.deref(buffer_size) > 0) { return Buffer.from(ref.reinterpret(ref.deref(buffer), ref.deref(buffer_size), 0)).toString(); } else { return ""; } } ... output_file_buffer = ref.alloc(ref.refType(ref.types.CString)); output_buffer_size = ref.alloc(ref.types.size_t, 0); return_status = gw.GW2GetIdInfo(session_id, 96, output_buffer_size, output_file_buffer); let error_description = buffer_to_string(output_file_buffer, output_buffer_size); ... ``` import Tabs from '@theme/Tabs' import TabItem from '@theme/TabItem' Every issue identified and reported by the Glasswall engine has a unique issue ID associated. This API provides a summary of all possible issue ID numbers and corresponding high level descriptions. For a **session** the **GW2GetAllIdInfo** function places XML data populated with Glasswall Issue ID descriptions and value ranges in an output buffer. **Synopsis** For session **session** the **GW2GetAllIdInfo** function places in the object pointed to by **outputBuffer** a pointer to XML data populated with Glasswall Issue ID descriptions and value ranges. The length, in bytes, of the populated output buffer is placed in the **size_t** object pointed to by **bufferLength**. ```cpp #include "glasswall.core2.api.h" int GW2GetAllIdInfo( Session session, size_t *bufferLength, char **outputBuffer); ``` **Returns** Returns an integer `GW2_RetStatus` enum value. Negative numbers indicate a failure. If successful, output buffer contains the XML file. **Example** ```cpp #include "glasswall.core2.api.h" ... char *outbuf = NULL; size_t buflen = 0; if (GW2OpenSession()) /* error opening session*/ else { int status = GW2GetAllIdInfo(session, &buflen, &outbuf); /* Buffer contains the XML file. * Either process the data pointed to, or copy the data and process it * after the session is closed */ if (GW2CloseSession()) /* error closing session */ } ``` **Synopsis** ```csharp public int GetAllIdInfo( int session, ref UIntPtr bufferLength, out IntPtr outputBuffer) ``` **Returns** Returns an integer `GW2_RetStatus` enum value. Negative numbers indicate a failure. If successful, output buffer contains the XML file. **Example** ```csharp ... UIntPtr bufferLength = UIntPtr.Zero; IntPtr buffer = new IntPtr(); int session = glasswall.OpenSession(); int returnStatus = glasswall.GetAllIdInfo(session, ref bufferLength, out buffer); using (StreamWriter writer = new StreamWriter(fileStream)) { writer.WriteLine($"GW2GetAllIdInfo completed with status code {returnStatus}"); if ((int)bufferLength != 0) { WriteBytesToFile(Path.Combine(di.FullName, $"22 - GW2GetAllIdInfo.xml"), glasswall.CreateArrayFromBuffer(buffer, bufferLength) ); } } ``` ```java import com.glasswall.core2javabridge.*; public String GW2GetAllIdInfoString(int session) throws GlasswallException ``` **Note** This functionality previously required two separate function calls to retrieve the ID data. This has now been streamlined to return the settings as a `String`. The two original functions have been deprecated. **Returns** The **GW2GetAllIdInfoString** function returns a `String` containing a description of all Glasswall IDs. A **GlasswallException** exception will be thrown if `session` is invalid, or if all ID information could not be retrieved. **Synopsis - Deprecated Functions** ```java import com.glasswall.core2javabridge.*; (Deprecated) public int GW2GetAllIdInfo(int session) throws GlasswallException public byte[] GetAllIDBuffer(int session) throws GlasswallException ``` **Description - Deprecated Functions** The **GW2GetAllIdInfo** function outputs the description of all Glasswall IDs for the session specified by `session` to internal All ID Buffer. Retrieve this data through use of the **GetAllIDBuffer** function. **Returns - Deprecated Functions** The **GW2GetAllIdInfo** function returns a **GW2_RetStatus** enumeration converted to `int`. The value will be negative if an error occurred. `0` indicates success. Refer to the API Overview/Return types for details. The **GetAllIDBuffer** returns a byte array containing the Glasswall ID descriptions. This will be null if **GW2GetAllIdInfo** has not been called. A **GlasswallException** exception will be thrown if `session` is invalid, or if all ID information could not be retrieved. **Synopsis** Retrieves the XML containing all the Issue ID ranges with their group descriptions. ```py def get_all_id_info(self, output_file: Optional[str] = None, raise_unsupported: bool = True) -> str: """ Retrieves the XML containing all the Issue ID ranges with their group descriptions Args: output_file (Optional[str], optional): The output file path where the analysis file will be written. raise_unsupported (bool, optional): Default True. Raise exceptions when Glasswall encounters an error. Fail silently if False. Returns: all_id_info (str): A string XML analysis report containing all id info. """ ``` **Returns** A string XML analysis report containing all id info. **Synopsis** ```jsx /** * This function places a pointer in a specified location to XML data populated with * Glasswall Issue ID descriptions and value ranges, for a specified session. * * @param {number} session The ID of the session. * @param {number} bufferLength The length of the buffer. * @param {string} outputBuffer The location of the output buffer. */ GW2GetAllIdInfo( session, bufferLength, outputBuffer) ``` **Returns** Returns an integer `GW2_RetStatus` enum value. Negative numbers indicate a failure. **Example** ```jsx const ref = require('ref-napi'); ... function buffer_to_string(buffer, buffer_size) { if (!buffer.isNull() && ref.deref(buffer_size) > 0) { return Buffer.from(ref.reinterpret(ref.deref(buffer), ref.deref(buffer_size), 0)).toString(); } else { return ""; } } ... let output_file_buffer = ref.alloc(ref.refType(ref.types.CString)); let output_buffer_size = ref.alloc(ref.types.size_t, 0); let return_status = gw.GW2GetAllIdInfo(session_id, output_buffer_size, output_file_buffer); let xml_string = buffer_to_string(output_file_buffer, output_buffer_size); ... ``` import Tabs from '@theme/Tabs' import TabItem from '@theme/TabItem' The **GW2LibVersion** function returns a string describing the current version of the Glasswall library in use. **Synopsis** The **GW2LibVersion** function returns a pointer to a constant string describing the current version of the library. ```cpp #include "glasswall.core2.api.h" const char * GW2LibVersion(void); ``` **Returns** Returns a pointer to a constant string describing the current library version **Synopsis** Retrieves the Glasswall library version and returns it as a string ```csharp /// /// Retrieves the Glasswall library version /// /// The Glasswall library version public string LibVersion() ``` **Returns** The Glasswall library version **Synopsis** ```java import com.glasswall.core2javabridge.*; public String GW2LibVersion() ``` **Returns** The **GW2LibVersion** function returns a `String` containing the Glasswall Engine library version. This will be null if the library version could not be retrieved. **Synopsis** Returns the Glasswall library version ```py def version(self): """ Returns the Glasswall library version. Returns: version (str): The Glasswall library version. """ ``` **Returns** version (str): The Glasswall library version. **Synopsis** This function retrieves the version string of the Glasswall Library ```jsx /** * This function retrieves the version string of the Glasswall Library * @returns {string} The Glasswall library version. */ GW2LibVersion() ``` **Returns** The Glasswall library version import Tabs from '@theme/Tabs' import TabItem from '@theme/TabItem' The **GW2RetStatusErrorMsg** function returns a string describing the error or status message associated with a status returned by other API functions. **Synopsis** The **GW2RetStatusErrorMsg** function returns a pointer to a constant string of the error message associated with the given **GW2_RetStatus** passed in **retStatus**. ```cpp #include "glasswall.core2.api.h" const char* GW2RetStatusErrorMsg(GW2_RetStatus retStatus); ``` **Returns** Returns the error message associated with a given GW2_ReturnStatus. **Synopsis** The **RetStatusErrorMsg** function returns the error message associated with a given GW2_ReturnStatus passed in **status**. ```csharp public string RetStatusErrorMsg(int status) ``` **Returns** Returns the error message associated with a given GW2_ReturnStatus. **Synopsis** This function returns the description of the given Glasswall status passed in **status** ```java public String GW2RetStatusErrorMsg(int status) throws GlasswallException ``` **Returns** String containing the description of the given status. The Glasswall Embedded Engine provides deep-file inspection, remediation, sanitisation, and reporting. The engine deconstructs a file to its structural components and builds an internal tree-like representation of the file. It walks each node of the tree, inspecting, repairing, and sanitising content items before reconstructing a new file. The Glasswall Embedded Engine also provides the ability to export and import the engine's internal representation of a file structure in an intermediate format such as XML. This allows internal components of a file to be made available to external programs for additional processing, before recomposing the file to include those externally modified components. The Glasswall Word Search engine is built on top of the export and import capability, performing text searching in the content and metadata of a file. Search strings, content management, and redaction rules are configured via an XML file. A user-configurable character substitution map defined in JSON form is used to provide support for text obfuscation. The engine also comes with built-in regular expression support. ## Word Search configuration The Word Search configuration specifies the text to search for, or the regular expression to be applied and how it should be treated when found within the document. The Word Search configuration is an extension to the Glasswall content management. ### Example policy files & schemas Example Word Search policy files and homoglyph dictionary can be found in the `/configs/sdk_word_search` folder of the release package. The Word Search XSD can be found in the `/schemas/sdk_word_search` folder of the release package. ### Example configuration policy #### Text settings The following sections showcases different `textSetting` that can be defined in a configuration policy. For more information on the different settings, refer to the [Word Search & Redaction page](/embedded-engine/embedded-engine-word-search-and-redaction). ##### Allow ```xml ((25[0-5]|(2[0-4]|1\d|[1-9]|)\d)\.?\b){4} allow Glasswall allow ``` ##### Disallow ```xml ((25[0-5]|(2[0-4]|1\d|[1-9]|)\d)\.?\b){4} disallow Glasswall disallow ``` ##### Redact ```xml ((25[0-5]|(2[0-4]|1\d|[1-9]|)\d)\.?\b){4} redact Glasswall redact ``` ##### Require ```xml ((25[0-5]|(2[0-4]|1\d|[1-9]|)\d)\.?\b){4} require ``` #### System configuration As with the core Glasswall engine, additional switches can be found under the `sysConfig` section. These control the behaviour of the Word Search engine while processing the input files. ```xml xml false ``` #### Known limitations - Processing both Office and text files at the same time is not possible - When processing text files, at least one require policy needs to be defined - `interchange_type` must always be specified as `xml` under `sysConfig` - Configuration policy that has a combination of the following `textSettings` with the same text/regex defined will always process the file: - require and redact - require and disallow - redact and allow - allow and disallow ### Example JSON homoglyph config The JSON file allows the user to create a mapping between characters and corresponding homoglyphs. This allows the engine to consider homoglyphs when generating search expressions, enabling support for homographs (look-alike words) and obfuscated text. Default Homoglyphs Config ```json { "!": "ǃⵑ", "$": "＄", "%": "％", "&": "ꝸ＆", "'": "`´ʹʻʼʽʾˈˊˋ˴ʹ΄՚՝י׳ߴߵᑊᛌ᾽᾿`´῾‘’‛′‵ꞌ＇｀𖽑𖽒", "(": "❨❲〔﴾（［", ")": "❩❳〕﴿）］", "*": "٭⁎∗＊𐌟", "+": "᛭＋𐊛", ",": "¸؍٫‚ꓹ，", "-": "˗۔‐‑‒–⁃−➖Ⲻ﹘", ".": "٠۰܁܂․ꓸ꘎．𐩐𝅭", "/": "᜵⁁⁄∕╱⟋⧸Ⳇ⼃〳ノ㇓丿／𝈺", "0": "OoΟοσОоՕօסه٥ھہە۵߀०০੦૦ଠ୦௦ం౦ಂ೦ംഠ൦ං๐໐ဝ၀ჿዐᴏᴑℴⲞⲟⵔ〇ꓳꬽﮦﮧﮨﮩﮪﮫﮬﮭﻩﻪﻫﻬ０Ｏｏ𐊒𐊫𐐄𐐬𐓂𐓪𐔖𑓐𑢵𑣈𑣗𑣠𝐎𝐨𝑂𝑜𝑶𝒐𝒪𝓞𝓸𝔒𝔬𝕆𝕠𝕺𝖔𝖮𝗈𝗢𝗼𝘖𝘰𝙊𝙤𝙾𝚘𝚶𝛐𝛔𝛰𝜊𝜎𝜪𝝄𝝈𝝤𝝾𝞂𝞞𝞸𝞼𝟎𝟘𝟢𝟬𝟶𞸤𞹤𞺄", "1": "Il|ƖǀΙІӀ׀וןا١۱ߊᛁℐℑℓⅠⅼ∣⏽Ⲓⵏꓲﺍﺎ１Ｉｌ￨𐊊𐌉𐌠𖼨𝐈𝐥𝐼𝑙𝑰𝒍𝓁𝓘𝓵𝔩𝕀𝕝𝕴𝖑𝖨𝗅𝗜𝗹𝘐𝘭𝙄𝙡𝙸𝚕𝚰𝛪𝜤𝝞𝞘𝟏𝟙𝟣𝟭𝟷𞣇𞸀𞺀", "2": "ƧϨᒿꙄꛯꝚ２𝟐𝟚𝟤𝟮𝟸", "3": "ƷȜЗӠⳌꝪꞫ３𑣊𖼻𝈆𝟑𝟛𝟥𝟯𝟹", "4": "Ꮞ４𑢯𝟒𝟜𝟦𝟰𝟺", "5": "Ƽ５𑢻𝟓𝟝𝟧𝟱𝟻", "6": "бᏮⳒ６𑣕𝟔𝟞𝟨𝟲𝟼", "7": "７𐓒𑣆𝈒𝟕𝟟𝟩𝟳𝟽", "8": "Ȣȣ৪੪ଃ８𐌚𝟖𝟠𝟪𝟴𝟾𞣋", "9": "৭੧୨൭ⳊꝮ９𑢬𑣌𑣖𝟗𝟡𝟫𝟵𝟿", "A": "4ΑАᎪᗅᴀꓮꭺＡ𐊠𖽀𝐀𝐴𝑨𝒜𝓐𝔄𝔸𝕬𝖠𝗔𝘈𝘼𝙰𝚨𝛢𝜜𝝖𝞐", "B": "ʙΒВвᏴᏼᗷᛒℬꓐꞴＢ𐊂𐊡𐌁𝐁𝐵𝑩𝓑𝔅𝔹𝕭𝖡𝗕𝘉𝘽𝙱𝚩𝛣𝜝𝝗𝞑", "C": "ϹСᏟℂℭⅭⲤꓚＣ𐊢𐌂𐐕𐔜𑣩𑣲𝐂𝐶𝑪𝒞𝓒𝕮𝖢𝗖𝘊𝘾𝙲🝌", "D": "ᎠᗞᗪᴅⅅⅮꓓꭰＤ𝐃𝐷𝑫𝒟𝓓𝔇𝔻𝕯𝖣𝗗𝘋𝘿𝙳", "E": "ΕЕᎬᴇℰ⋿ⴹꓰꭼＥ𐊆𑢦𑢮𝐄𝐸𝑬𝓔𝔈𝔼𝕰𝖤𝗘𝘌𝙀𝙴𝚬𝛦𝜠𝝚𝞔", "F": "ϜᖴℱꓝꞘＦ𐊇𐊥𐔥𑢢𑣂𝈓𝐅𝐹𝑭𝓕𝔉𝔽𝕱𝖥𝗙𝘍𝙁𝙵𝟊", "G": "ɢԌԍᏀᏳᏻꓖꮐＧ𝐆𝐺𝑮𝒢𝓖𝔊𝔾𝕲𝖦𝗚𝘎𝙂𝙶", "H": "ʜΗНнᎻᕼℋℌℍⲎꓧꮋＨ𐋏𝐇𝐻𝑯𝓗𝕳𝖧𝗛𝘏𝙃𝙷𝚮𝛨𝜢𝝜𝞖", "I": "", "J": "ͿЈᎫᒍᴊꓙꞲꭻＪ𝐉𝐽𝑱𝒥𝓙𝔍𝕁𝕵𝖩𝗝𝘑𝙅𝙹", "K": "ΚКᏦᛕKⲔꓗＫ𐔘𝐊𝐾𝑲𝒦𝓚𝔎𝕂𝕶𝖪𝗞𝘒𝙆𝙺𝚱𝛫𝜥𝝟𝞙", "L": "ʟᏞᒪℒⅬⳐⳑꓡꮮＬ𐐛𐑃𐔦𑢣𑢲𖼖𝈪𝐋𝐿𝑳𝓛𝔏𝕃𝕷𝖫𝗟𝘓𝙇𝙻", "M": "ΜϺМᎷᗰᛖℳⅯⲘꓟＭ𐊰𐌑𝐌𝑀𝑴𝓜𝔐𝕄𝕸𝖬𝗠𝘔𝙈𝙼𝚳𝛭𝜧𝝡𝞛", "N": "ɴΝℕⲚꓠＮ𐔓𝐍𝑁𝑵𝒩𝓝𝔑𝕹𝖭𝗡𝘕𝙉𝙽𝚴𝛮𝜨𝝢𝞜", "O": "0", "P": "ΡРᏢᑭᴘᴩℙⲢꓑꮲＰ𐊕𝐏𝑃𝑷𝒫𝓟𝔓𝕻𝖯𝗣𝘗𝙋𝙿𝚸𝛲𝜬𝝦𝞠", "Q": "ℚⵕＱ𝐐𝑄𝑸𝒬𝓠𝔔𝕼𝖰𝗤𝘘𝙌𝚀", "R": "ƦʀᎡᏒᖇᚱℛℜℝꓣꭱꮢＲ𐒴𖼵𝈖𝐑𝑅𝑹𝓡𝕽𝖱𝗥𝘙𝙍𝚁", "S": "$ЅՏᏕᏚꓢＳ𐊖𐐠𖼺𝐒𝑆𝑺𝒮𝓢𝔖𝕊𝕾𝖲𝗦𝘚𝙎𝚂", "T": "ŤΤτТтᎢᴛ⊤⟙ⲦꓔꭲＴ𐊗𐊱𐌕𑢼𖼊𝐓𝑇𝑻𝒯𝓣𝔗𝕋𝕿𝖳𝗧𝘛𝙏𝚃𝚻𝛕𝛵𝜏𝜯𝝉𝝩𝞃𝞣𝞽🝨", "U": "Սሀᑌ∪⋃ꓴＵ𐓎𑢸𖽂𝐔𝑈𝑼𝒰𝓤𝔘𝕌𝖀𝖴𝗨𝘜𝙐𝚄", "V": "Ѵ٧۷ᏙᐯⅤⴸꓦꛟＶ𐔝𑢠𖼈𝈍𝐕𝑉𝑽𝒱𝓥𝔙𝕍𝖁𝖵𝗩𝘝𝙑𝚅", "W": "ԜᎳᏔꓪＷ𑣦𑣯𝐖𝑊𝑾𝒲𝓦𝔚𝕎𝖂𝖶𝗪𝘞𝙒𝚆", "X": "ΧХ᙭ᚷⅩ╳ⲬⵝꓫꞳＸ𐊐𐊴𐌗𐌢𐔧𑣬𝐗𝑋𝑿𝒳𝓧𝔛𝕏𝖃𝖷𝗫𝘟𝙓𝚇𝚾𝛸𝜲𝝬𝞦", "Y": "ΥϒУҮᎩᎽⲨꓬＹ𐊲𑢤𖽃𝐘𝑌𝒀𝒴𝓨𝔜𝕐𝖄𝖸𝗬𝘠𝙔𝚈𝚼𝛶𝜰𝝪𝞤", "Z": "ΖᏃℤℨꓜＺ𐋵𑢩𑣥𝐙𝑍𝒁𝒵𝓩𝖅𝖹𝗭𝘡𝙕𝚉𝚭𝛧𝜡𝝛𝞕", "a": "@ɑαа⍺ａ𝐚𝑎𝒂𝒶𝓪𝔞𝕒𝖆𝖺𝗮𝘢𝙖𝚊𝛂𝛼𝜶𝝰𝞪", "b": "ƄЬᏏᖯｂ𝐛𝑏𝒃𝒷𝓫𝔟𝕓𝖇𝖻𝗯𝘣𝙗𝚋", "c": "ϲсᴄⅽⲥꮯｃ𐐽𝐜𝑐𝒄𝒸𝓬𝔠𝕔𝖈𝖼𝗰𝘤𝙘𝚌", "d": "ԁᏧᑯⅆⅾꓒｄ𝐝𝑑𝒅𝒹𝓭𝔡𝕕𝖉𝖽𝗱𝘥𝙙𝚍", "e": "еҽ℮ℯⅇꬲｅ𝐞𝑒𝒆𝓮𝔢𝕖𝖊𝖾𝗲𝘦𝙚𝚎", "f": "ſϝքẝꞙꬵｆ𝐟𝑓𝒇𝒻𝓯𝔣𝕗𝖋𝖿𝗳𝘧𝙛𝚏𝟋", "g": "ƍɡցᶃℊｇ𝐠𝑔𝒈𝓰𝔤𝕘𝖌𝗀𝗴𝘨𝙜𝚐", "h": "һհᏂℎｈ𝐡𝒉𝒽𝓱𝔥𝕙𝖍𝗁𝗵𝘩𝙝𝚑", "i": "ıɩɪ˛ͺιіӏᎥιℹⅈⅰ⍳ꙇꭵｉ𑣃𝐢𝑖𝒊𝒾𝓲𝔦𝕚𝖎𝗂𝗶𝘪𝙞𝚒𝚤𝛊𝜄𝜾𝝸𝞲", "j": "ϳјⅉｊ𝐣𝑗𝒋𝒿𝓳𝔧𝕛𝖏𝗃𝗷𝘫𝙟𝚓", "k": "ｋ𝐤𝑘𝒌𝓀𝓴𝔨𝕜𝖐𝗄𝗸𝘬𝙠𝚔", "l": "1", "m": "ｍ", "n": "ոռｎ𝐧𝑛𝒏𝓃𝓷𝔫𝕟𝖓𝗇𝗻𝘯𝙣𝚗", "o": "", "p": "ρϱр⍴ⲣｐ𝐩𝑝𝒑𝓅𝓹𝔭𝕡𝖕𝗉𝗽𝘱𝙥𝚙𝛒𝛠𝜌𝜚𝝆𝝔𝞀𝞎𝞺𝟈", "q": "ԛգզｑ𝐪𝑞𝒒𝓆𝓺𝔮𝕢𝖖𝗊𝗾𝘲𝙦𝚚", "r": "гᴦⲅꭇꭈꮁｒ𝐫𝑟𝒓𝓇𝓻𝔯𝕣𝖗𝗋𝗿𝘳𝙧𝚛", "s": "$ƽѕꜱꮪｓ𐑈𑣁𝐬𝑠𝒔𝓈𝓼𝔰𝕤𝖘𝗌𝘀𝘴𝙨𝚜", "t": "ｔ𝐭𝑡𝒕𝓉𝓽𝔱𝕥𝖙𝗍𝘁𝘵𝙩𝚝", "u": "ʋυսᴜꞟꭎꭒｕ𐓶𑣘𝐮𝑢𝒖𝓊𝓾𝔲𝕦𝖚𝗎𝘂𝘶𝙪𝚞𝛖𝜐𝝊𝞄𝞾", "v": "νѵטᴠⅴ∨⋁ꮩｖ𑜆𑣀𝐯𝑣𝒗𝓋𝓿𝔳𝕧𝖛𝗏𝘃𝘷𝙫𝚟𝛎𝜈𝝂𝝼𝞶", "w": "ɯѡԝաᴡꮃｗ𑜊𑜎𑜏𝐰𝑤𝒘𝓌𝔀𝔴𝕨𝖜𝗐𝘄𝘸𝙬𝚠", "x": "×хᕁᕽ᙮ⅹ⤫⤬⨯ｘ𝐱𝑥𝒙𝓍𝔁𝔵𝕩𝖝𝗑𝘅𝘹𝙭𝚡", "y": "ɣʏγуүყᶌỿℽꭚｙ𑣜𝐲𝑦𝒚𝓎𝔂𝔶𝕪𝖞𝗒𝘆𝘺𝙮𝚢𝛄𝛾𝜸𝝲𝞬", "z": "ᴢꮓｚ𑣄𝐳𝑧𝒛𝓏𝔃𝔷𝕫𝖟𝗓𝘇𝘻𝙯𝚣", "£": "₤", "©": "Ⓒ", "®": "Ⓡ" } ``` ### Example analysis report Here is an example analysis report that is generated when the search string is set to 'Glasswall', irrespective of the `textSetting` used. This includes an `ItemMatchCount` for every pattern that is matched in a given file. ```xml Glasswall 1 463 0 0 ``` ## API functions ### Status The APIs GwWordSearch and GwWordSearchDone returns a Status that indicates the result of the API call. API GwWordSearchTranslateStatus returns a description for a passed in Status. | Enumerator | Value | Description | | :------------------------ | :---: | ------------------------------------------------------------ | | ws_disallowedItemFound | -1024 | Item disallowed by policy found in file. | | ws_requiredItemNotFound | -1025 | Item required by policy not found in file. | | ws_illegalActionRedact | -1026 | Redact action specified but filetype doesn't support redaction. | | ws_illegalActionRequire | -1027 | Require action specified but filetype doesn't support require. | | ws_illegalActionNoRequire | -1028 | Require action not specified but filetype needs one. | | ws_filetypeUnsupported | -1029 | Filetype not supported by Word Search. | | eFail | 0 | General or otherwise unspecified error. | | eSuccess | 1 | The operation was successful. | #### C++ Each of the APIs return a Status, which is defined as follows: ```cpp enum Status { ws_disallowedItemFound = -1024, ws_requiredItemNotFound = -1025, ws_illegalActionRedact = -1026, ws_illegalActionRequire = -1027, ws_illegalActionNoRequire = -1028, ws_filetypeUnsupported = -1029, eFail = 0, eSuccess = 1, }; ``` #### C# To integrate Glasswall Word Search in C# the Glasswall **Word Search C# wrapper is required**. Each of the APIs returns a `WordSearchStatus` type, which is defined as follows: ```csharp /// /// Indicates whether the Word Search process was successful (WordSearchStatus.Success) /// or not (WordSearchStatus.Fail). Zero or negative values indicate a failure. /// public enum WordSearchStatus { DisallowedItemFound = -1024, RequiredItemNotFound = -1025, IllegalActionRedact = -1026, IllegalActionRequire = -1027, IllegalActionNoRequire = -1028, FiletypeUnsupported = -1029, Fail = 0, Success } ``` #### Java To integrate Glasswall Word Search in java the Glasswall **Word Search Java wrapper is required**. Each of the APIs returns a GlasswallWordSearchResult` type, which is defined as follows: ```java package com.glasswallsolutions; /** * Class used to hold the results from a Word Search process. */ public class GlasswallWordSearchResult { /** * The XML analysis report */ public String report; /** * The processed document */ public byte[] outputDocument; /** * boolean indicating whether the process was successful (true) or not (false) */ public boolean success; public GlasswallWordSearchResult() { report = null; outputDocument = null; success = false; } } ``` #### Python To integrate Glasswall Word Search in Python the **Glasswall Python wrapper is required**. Each of the APIs returns a generic `GwReturnObj` object, which will contain the attributes: "status" (int), "output_file" (bytes), "output_report" (bytes). The int statuses are defined as follows: ```py # glasswall\libraries\word_search\successes.py class Success(WordSearchSuccess): """ WordSearch success code 1. """ pass success_codes = { 1: Success, } ``` ```py # glasswall\libraries\word_search\errors.py class UnknownErrorCode(WordSearchError): """ Unknown error code. """ pass class Fail(WordSearchError): """ WordSearch error code 0. """ pass class DisallowedItemFound(WordSearchError): """ WordSearch error code -1024. Item disallowed by policy found in file. """ pass class RequiredItemNotFound(WordSearchError): """ WordSearch error code -1025. Item required by policy not found in file. """ pass class IllegalActionRedact(WordSearchError): """ WordSearch error code -1026. Redact action specified but filetype doesn't support redaction. """ pass class IllegalActionRequire(WordSearchError): """ WordSearch error code -1027. Require action specified but filetype doesn't support redaction. """ pass class IllegalActionNoRequire(WordSearchError): """ WordSearch error code -1028. Require action not specified but filetype needs one. """ pass class FiletypeUnsupported(WordSearchError): """ WordSearch error code -1029. Filetype supported by Editor but not by Word Search. """ pass error_codes = { 0: Fail, -1024: DisallowedItemFound, -1025: RequiredItemNotFound, -1026: IllegalActionRedact, -1027: IllegalActionRequire, -1028: IllegalActionNoRequire, -1029: FiletypeUnsupported, } ``` #### JavaScript To integrate Glasswall Word Search in JavaScript the Glasswall **Word Search JavaScript wrapper is required**. Each of the APIs returns a `WordSearchStatus` type, which is defined as follows: ```jsx /** * Used to indicate whether the Word Search process was successful or not */ export const enum WordSearchStatus { ws_disallowedItemFound = -1024, ws_requiredItemNotFound = -1025, ws_illegalActionRedact = -1026, ws_illegalActionRequire = -1027, ws_illegalActionNoRequire = -1028, ws_filetypeUnsupported = -1029, eFail = 0, eSuccess = 1, } ``` ### GwWordSearch This is used to call the Word Search engine, process the specified input file and produce an output file along with a Word Search analysis report. #### C++ ```cpp Status GwWordSearch( void* input_buffer, size_t input_buffer_len, void** output_buffer, size_t* output_buffer_len, void** output_report_buffer, size_t* output_report_buffer_len, const char* homoglpyhs, const char* xml_config_string ) ``` Name | Type | Direction | Description ---- | ---- | --------- | ----------- input_buffer | `void *` | In | A pointer to the buffer containing the input file to be processed input_buffer_len | `size_t` | In | The size of the input file buffer output_buffer | `void **` | Out | A pointer to a pointer to a buffer that will be populated with the processed file buffer. This buffer is allocated by the Word Search engine output_buffer_len | `size_t *` | Out | A pointer to the size of the output file buffer. This will be set by the Word Search engine output_report_buffer | `void **` | Out | A pointer to a pointer to a buffer that will be populated with the Word Search analysis report buffer. This buffer is allocated by the Word Search engine output_report_buffer_len | `size_t *` | Out | A pointer to the size of the Word Search analysis report. This will be set by the Word Search engine homoglyphs | `const char *` | In | A pointer to the buffer containing the homoglyphs file. This buffer needs to be null terminated xml_config_string | `const char *` | In | A pointer to the buffer containing the content management XML file. This buffer needs to be null terminated #### C# To integrate Glasswall Word Search in C# the Glasswall **Word Search C# wrapper is required**. ```csharp public WordSearchStatus GwWordSearch( byte[] inputBuffer, out byte[] outputFileBuffer, out String outputAnalysisReport, string homoglyphs, string xmlConfigString ) ``` Name | Type | Direction | Description ---- | ---- | --------- | ----------- inputBuffer | `byte[]` | In | The buffer containing the document to be processed outputFileBuffer | `out byte[]` | Out | The resulting buffer that will contain the processed document outputAnalysisReport | `out string` | Out | The output analysis report from the Word Search process homoglyphs | `string` | In | A JSON document containing the homoglyph mappings xmlConfigString | `string` | In | The XML content management policy #### Java To integrate Glasswall Word Search in Java the Glasswall **Word Search Java wrapper is required**. ```java public native GlasswallWordSearchResult wordSearch( byte[] inputDocument, String homoglyphs, String xmlConfig ) ``` Name | Type | Direction | Description ---- | ---- | --------- | ----------- inputDocument | `byte[]` | In | The buffer containing the document to be processed homoglyphs | `string` | In | A JSON document containing the homoglyph mappings xmlConfig | `string` | In | The XML content management policy **Note**: Unlike some other supported languages, all output is returned in the `GlasswallWordSearchResult` object for Java. #### Python To integrate Glasswall Word Search in Python the **Glasswall Python wrapper is required**. ```py # glasswall\libraries\word_search\word_search.py def redact_file(self, input_file: Union[str, bytes, bytearray, io.BytesIO], content_management_policy: Union[str, bytes, bytearray, io.BytesIO], output_file: Union[None, str] = None, output_report: Union[None, str] = None, homoglyphs: Union[None, str, bytes, bytearray, io.BytesIO] = None, raise_unsupported: bool = True): """ Redacts text from input_file using the given content_management_policy and homoglyphs file, optionally writing the redacted file and analysis report to the paths specified by output_file and output_report. Args: input_file (Union[str, bytes, bytearray, io.BytesIO]): The input file path or bytes. content_management_policy (Union[str, bytes, bytearray, io.BytesIO)]): The content management policy to apply. output_file (Union[None, str], optional): Default None. If str, write output_file to that path. output_report (Union[None, str], optional): Default None. If str, write output_file to that path. homoglyphs (Union[None, str, bytes, bytearray, io.BytesIO)], optional): Default None. The homoglyphs json file path or bytes. raise_unsupported (bool, optional): Default True. Raise exceptions when Glasswall encounters an error. Fail silently if False. Returns: gw_return_object (glasswall.GwReturnObj): An instance of class glasswall.GwReturnObj containing attributes: "status" (int), "output_file" (bytes), "output_report" (bytes) """ def redact_directory(self, input_directory: str, content_management_policy: Union[str, bytes, bytearray, io.BytesIO, glasswall.content_management.policies.policy.Policy], output_directory: Optional[str] = None, output_report_directory: Optional[str] = None, homoglyphs: Union[None, str, bytes, bytearray, io.BytesIO] = None, raise_unsupported: bool = True): """ Redacts all files in a directory and it's subdirectories using the given content_management_policy and homoglyphs file. The redacted files are written to output_directory maintaining the same directory structure as input_directory. Args: input_directory (str): The input directory containing files to redact. output_directory (str): The output directory where the redacted files will be written. output_report_directory (Optional[str], optional): Default None. If str, the output directory where analysis reports for each redacted file will be written. content_management_policy (Union[str, bytes, bytearray, io.BytesIO)]): The content management policy to apply. homoglyphs (Union[None, str, bytes, bytearray, io.BytesIO)], optional): Default None. The homoglyphs file path, str, or bytes. raise_unsupported (bool, optional): Default True. Raise exceptions when Glasswall encounters an error. Fail silently if False. Returns: redacted_files_dict (dict): A dictionary of file paths relative to input_directory, and glasswall.GwReturnObj with attributes: "status" (int), "output_file" (bytes), "output_report" (bytes) """ ``` **Note**: Unlike some other supported languages, all output is returned in the `GwReturnObj` object for Python. #### JavaScript ```jsx /** * Perform word search on input buffer, using the applied config and homoglyphs * @param {Buffer} inputBuffer A buffer containing the contents of the document to be processed. * @param {String} homoglyphs A homoglyphs file that will be used as part of the Word Search process (UTF-8 string). * @param {String} configXml The content management XML policy (utf-8 string). * @returns {WordSearchResult} The result from Word Search. */ wordSearch(inputBuffer: Buffer, homoglyphs: string, configXml: string): WordSearchResult ``` **Note**: Unlike some other supported languages, all output is returned in the `WordSearchResult` object for JavaScript. ### GWWordSearchDone This is used to release any resources that have been allocated by the Word Search engine. This function needs to be called after each call made to the `GwWordSearch` function otherwise memory leaks will occur. This API call is only required in C++. #### C++ ```cpp Status GwWordSearchDone( void** output_buffer, size_t* output_buffer_len, void** output_report_buffer, size_t* output_report_buffer_len) ``` Name | Type | Direction | Description ---- | ---- | --------- | ----------- output_buffer | `void **` | Out | A pointer to a pointer to the buffer containing the processed file that will be freed by the Word Search library output_buffer_len | `size_t *` | Out | A pointer to the size of the output file buffer output_report_buffer | `void **` | Out | A pointer to a pointer to the buffer containing the Word Search analysis report that will be freed by the Word Search library output_report_buffer_len | `size_t *` | Out | A pointer to the size of the Word Search analysis report #### Other languages For all languages covered by the Glasswall wrappers, the **GwWordSearchDone** API function is internally called within the wrapper, meaning the API is not exposed to the user. ### GwWordSearchVersion This is used to retrieve the current library version number. #### C++ ```cpp const char* GwWordSearchVersion(void) ``` ### GwWordSearchTranslateStatus Translate the provided error code into a user friendly error message. #### C++ ```cpp const char* GwWordSearchTranslateStatus(Status errorCode) ``` | Name | Type | Direction | Description | | --------- | ----------------------| --------- | ---------------------------------------- | | errorCode | Status | In | The return code that should be translated | ## Common issues ### Word Search is not processing files When running Word Search, please ensure that **all Embedded Engine libraries** are located in the same directory, which also needs to be set as the current working directory. Glasswall searches within the current working directory for it's dependencies and if they are not found then files will not be processed correctly. Also ensure a **valid licence key** is present. ## Example usage Here we have an example application that takes an input file, processes it using the Glasswall Word Search engine, and then produces an output file alongside a Word Search analysis report. This example application expects the following command line parameters: 1. Path to the content management configuration XML. 2. Path to the homoglyphs file. 3. Path to the input file to be processed. 4. Path to the output file where the processed file will be stored. #### C++ ```cpp #include #include #include #include #include #include #include "api.h" using namespace std; // Read the file into a buffer vector readFile(ifstream &fileHandle, const string &filePath, bool nullTerminator) { fileHandle.exceptions(ifstream::failbit | ifstream::badbit); fileHandle.open(filePath.c_str(), ios::binary | ios::ate); vector data; streamsize size = fileHandle.tellg(); fileHandle.seekg(0, ios::beg); data.resize(size + 1); fileHandle.read(reinterpret_cast(data.data()), size); if (nullTerminator) { data.push_back(0); } return data; } int main(int argc, char **argv) { if (argc != 5) { cerr " xmlBuffer = readFile(xmlFileHandle, xmlFilePath, true); // Buffer containing the XML content management settings. This is null terminated vector homoglyphsBuffer = readFile(homoglyphsFileHandle, homoglyphsFilePath, true); // Buffer containing the homoglyphs. This is null terminated vector inputBuffer = readFile(inputFileHandle, inputFilePath, false); // Buffer containing the input file to be processed // Create variables for output buffers void * outputBuffer = nullptr; // Output buffer for processed file size_t outputBufferSize = 0; // Output buffer size void * outputReportBuffer = nullptr; // Output buffer for analysis report file size_t outputReportBufferSize = 0; // Output analysis report buffer size // Run Word Search and redact Status status = GwWordSearch(inputBuffer.data(), inputBuffer.size(), &outputBuffer, &outputBufferSize, &outputReportBuffer, &outputReportBufferSize, reinterpret_cast(homoglyphsBuffer.data()), reinterpret_cast(xmlBuffer.data())); if (status == Status::eSuccess) { // Write out the processed output file if the Word Search and redact was successful ofstream outputFileHandle(outputFilePath, ios::binary | ios::trunc); if (outputFileHandle.is_open()) { outputFileHandle.write(static_cast(outputBuffer), outputBufferSize); } outputFileHandle.close(); } // Write out the analysis report file ofstream analysisFileHandle(outputFilePath + ".xml", ios::binary | ios::trunc); if (analysisFileHandle.is_open()) { analysisFileHandle.write(static_cast(outputReportBuffer), outputReportBufferSize); } analysisFileHandle.close(); // Call done to release any allocated resources GwWordSearchDone(&outputBuffer, &outputBufferSize, &outputReportBuffer, &outputReportBufferSize); return 0; } ``` #### C# ```csharp using System; using System.IO; namespace glasswall.word.search.csharp.testing { internal class Program { static void Main(string[] args) { Console.WriteLine("Word Search test"); if (args.Length != 4) { Console.WriteLine("usage: "); Console.WriteLine("Parameters specified: \n{0}", string.Join("\n", args)); return; } string xmlConfigPath = args[0]; string homoglyphsPath = args[1]; string inputDirectory = args[2]; string outputDirectory = args[3]; if (!File.Exists(xmlConfigPath)) { Console.Error.WriteLine("Xml config does not exist: {0}", xmlConfigPath); return; } if (!File.Exists(homoglyphsPath)) { Console.Error.WriteLine("Homoglyphs does not exist: {0}", homoglyphsPath); return; } if (!Directory.Exists(inputDirectory)) { Console.Error.WriteLine("Input directory does not exist: {0}", inputDirectory); return; } Directory.CreateDirectory(outputDirectory); using (FileStream fileStream = new FileStream(Path.Combine(outputDirectory, "ProcessLog.txt"), FileMode.OpenOrCreate, FileAccess.Write)) { using (StreamWriter writer = new StreamWriter(fileStream)) { writer.WriteLine("> Word Search Library version: {0}", GlasswallWordSearch.GwWordSearchVersion()); string xmlConfig = File.ReadAllText(xmlConfigPath); string homoglyphs = File.ReadAllText(homoglyphsPath); foreach (string path in Directory.EnumerateFiles(inputDirectory, "*", SearchOption.AllDirectories)) { writer.WriteLine("> Processing file: {0}", path); string inputDirectoryPath = path.Substring(inputDirectory.Length + 1); string directory = Path.Combine(outputDirectory, inputDirectoryPath); Directory.CreateDirectory(directory); processFile(path, directory, homoglyphs, xmlConfig); } } } return; } static void WriteAllBytes(string path, byte[] data) { if (data == null) { File.Create(path); } else { File.WriteAllBytes(path, data); } } public static void processFile(string inputFile, string outputDirectory, string homoglyphs, string xmlConfig) { using (FileStream fileStream = new FileStream(Path.Combine(outputDirectory, Path.GetFileName(inputFile) + ".log"), FileMode.OpenOrCreate, FileAccess.Write)) { using (StreamWriter writer = new StreamWriter(fileStream)) { // Word Search writer.WriteLine(">> Run Word Search"); byte[] inputFileBuffer = File.ReadAllBytes(inputFile); byte[] outputBuffer, outputReportBuffer; GlasswallWordSearch.WordSearchStatus status = GlasswallWordSearch.GwWordSearch(inputFileBuffer, out outputBuffer, out outputReportBuffer, homoglyphs, xmlConfig); writer.WriteLine("Status is: {0}", status); if (outputBuffer != null) { WriteAllBytes(Path.Combine(outputDirectory, Path.GetFileName(inputFile)), outputBuffer); } if (outputReportBuffer != null) { WriteAllBytes(Path.Combine(outputDirectory, Path.GetFileName(inputFile)) + ".xml", outputReportBuffer); } } } } } } ``` #### Java ```java package com.glasswallsolutions; import java.lang.System; import java.io.*; import com.glasswallsolutions.*; import java.nio.file.Paths; public class MainTest { public static byte[] readAllBytes(InputStream inputStream) throws IOException { final int bufLen = 4 * 0x400; // 4KB byte[] buf = new byte[bufLen]; int readLen; try (ByteArrayOutputStream outputStream = new ByteArrayOutputStream()) { while ((readLen = inputStream.read(buf, 0, bufLen)) != -1) outputStream.write(buf, 0, readLen); return outputStream.toByteArray(); } } public static void main(String[] args) throws Exception { if (args.length != 4) { System.out.println("Usage: "); System.exit(-1); } File inputDirectory = new File(args[0]); File outputDirectory = new File(args[1]); outputDirectory.delete(); outputDirectory.mkdir(); String homoglyphsFile = args[2]; String configXmlFile = args[3]; String homoglyphs = null; String configXML = null; GlasswallWordSearch glasswallWordSearch = new GlasswallWordSearch(); try(FileInputStream homoglyphsInputStream = new FileInputStream(homoglyphsFile)) { homoglyphs = new String(readAllBytes(homoglyphsInputStream)); } try(FileInputStream configXmlInputStream = new FileInputStream(configXmlFile)) { configXML = new String(readAllBytes(configXmlInputStream)); } System.out.println("Word Search version: " + glasswallWordSearch.version()); for (File inputFile : inputDirectory.listFiles()) { try { System.out.println("Processing file: " + inputFile.getAbsolutePath()); File fileOutputDirectory = new File(Paths.get(outputDirectory.getAbsolutePath(), inputFile.getName()).toString()); fileOutputDirectory.mkdir(); String fileOutputPath = Paths.get(fileOutputDirectory.getAbsolutePath(), inputFile.getName()).toString(); try(FileInputStream inputStream = new FileInputStream(inputFile)) { byte[] fileData = readAllBytes(inputStream); GlasswallWordSearchResult result = glasswallWordSearch.wordSearch(fileData, homoglyphs, configXML); System.out.println("Status: " + result.success); if (result.outputDocument != null) { try(FileOutputStream fileOutputStream = new FileOutputStream(fileOutputPath)) { fileOutputStream.write(result.outputDocument); } } if (result.report != null) { try(FileOutputStream fileOutputStream = new FileOutputStream(fileOutputPath + ".xml")) { fileOutputStream.write(result.report.getBytes()); } } } } catch(Exception ex) { System.err.println("Exception occurred: " + ex.getMessage()); ex.printStackTrace(System.err); } } } } ``` #### Python For further examples see [Python Word Search & Redaction](/embedded-engine/embedded-engine-python-word-search-and-redaction) #### JavaScript ```jsx import fs from 'fs'; import path from 'path'; import { GlasswallWordSearch, GlasswallWordSearchNative, WordSearchResult, WordSearchStatus } from '../index' let main = function() { const args = process.argv; if (args.length === 7) { let wordSearchDllPath = path.resolve(args[2]); let inputDirectory = path.resolve(args[3]); let outputDirectory = path.resolve(args[4]); let homoglyphsPath = path.resolve(args[5]); let configXmlPath = path.resolve(args[6]); let handler = new GlasswallWordSearchNative(wordSearchDllPath, { enableLogging: true}); let glasswallWordSearch = new GlasswallWordSearch(handler); console.log("Glasswall Word Search version: " + glasswallWordSearch.version()) if (!fs.existsSync(inputDirectory)) { console.log('Input Directory does not exist: ' + inputDirectory); process.exit(-1); } if (!fs.existsSync(homoglyphsPath)) { console.log('Homoglyphs file does not exist: ' + homoglyphsPath); process.exit(-1); } if (!fs.existsSync(configXmlPath)) { console.log('Config XML file does not exist: ' + configXmlPath); process.exit(-1); } let homoglyphs = fs.readFileSync(homoglyphsPath, 'utf8'); let configXml = fs.readFileSync(configXmlPath , 'utf8'); fs.mkdirSync(outputDirectory, {recursive: true}); fs.readdirSync(inputDirectory).forEach(file => { try { let fullFilePath = path.join(inputDirectory, file); if (fs.statSync(fullFilePath).isFile()) { console.log('Processing file: ' + fullFilePath); let outputFileDirectory = path.join(outputDirectory, file); fs.mkdirSync(outputFileDirectory); let inputBuffer = fs.readFileSync(fullFilePath); let wordSearchResult = glasswallWordSearch.wordSearch(inputBuffer, homoglyphs, configXml); console.log("Status: " + wordSearchResult.status); if (wordSearchResult.outputBuffer != undefined && wordSearchResult.outputBuffer != null) { fs.writeFileSync(path.join(outputFileDirectory, file), wordSearchResult.outputBuffer); } if (wordSearchResult.analysisXmlReport != undefined && wordSearchResult.analysisXmlReport != null) { fs.writeFileSync(path.join(outputFileDirectory, file + ".xml"), wordSearchResult.analysisXmlReport); } } } catch(error) { console.log("Exception occurred: " + error); console.trace(error); } }) } else { console.log("Usage: Application "); process.exit(-1); } } if (require.main === module){ main(); } ``` **Glasswall Conform** is a command-line tool designed to reconstruct malformed or corrupt PDF files that cannot be processed by the **Glasswall Embedded Engine**. This tool extracts visual content such as text, graphics, and images from input PDFs, generating a newly reconstructed document that adheres to PDF standards. The tool is particularly useful for addressing PDF files that deviate from the ISO 32001 family of specifications, making them incompatible with the **Glasswall Embedded Engine** and otherwise subject to being blocked. By restoring structural integrity, **Glasswall Conform** makes these files suitable for further processing by the **Glasswall Embedded Engine** for **Content Disarm and Reconstruction (CDR)**. The reconstruction process allows for some loss of visual fidelity to achieve a conforming, processable file. In some cases, the tool may not be able to process certain documents. For more information on current features, constraints, and known limitations, please refer to [**Features, Constraints, and Limitations**](/embedded-engine/glasswall-conform-features-constraints-and-limitations). ## Components & services Glasswall Conform is packaged as an installer within a ZIP archive, which also includes documentation in Markdown format. ### Contents of the ZIP archive: The ZIP archive size is approximately **100 MiB**. An example of the structure for the Linux RPM distribution is below. ``` glasswall_conform_1.1.0_linux_amd64_rpm.zip │ glasswall_conform-1.1.0-1.x86_64.rpm └───docs Changelog.md Features-Constraints-and-Limitations.md Overview.md User-Guide.md ``` ## User guide For instructions on installation, configuration, and usage, including advanced options, please refer to the [**User Guide**](/embedded-engine/glasswall-conform-user-guide). The User Guide includes examples of command-line usage and describes the various processing modes and arguments available. ## Platform support Glasswall Conform is available for both **Windows** and **Linux**. For Windows we provide an `.exe` installer. For Linux we provide `.rpm` and `.deb` packages which support Linux distributions such as **Rocky 9**, **Rocky 8**, **Ubuntu 24**, and **Ubuntu 22**. ## Features Glasswall Conform is a command-line tool designed for preprocessing PDF documents. It extracts and reconstructs visual content to ensure documents meet PDF standards, preparing them for further processing by the **Glasswall Embedded Engine**, which provides comprehensive Content Disarm and Reconstruction (CDR) protection. **Key Features:** - **Text, Graphic, and Image Extraction:** Extracts and reconstructs text, graphics, and images from PDFs, producing a clean, standards-compliant output document. - **Handling Rate Threshold:** Allows setting a minimum handling rate for graphics, images, or text. Files that fail to meet this threshold are classified as failures and will not be saved. - **Custom Watermarking:** Supports adding custom watermark text on each page of the reconstructed PDF, enabling personalised branding or messaging. - **Character Identifier (CID) and Glyph Suppression:** Suppresses unsupported glyphs and character identifiers (CIDs), replacing them with the default question mark character (?). - **Font Replacement:** Converts custom embedded fonts to known-good Microsoft fonts or defaults to Cambria Math when necessary. This process aims to provide the best possible text display, even when custom fonts are not supported. - **Standards Compliance:** Produces a reconstructed PDF that adheres to PDF standards, allowing for subsequent CDR processing by the Glasswall Embedded Engine for full Content Disarm and Reconstruction (CDR) protection. - **Fast Mode:** Enabled by default, Fast Mode processes files quickly while maintaining accurate visual appearance. - Fastest processing speed. - Best visual appearance. - Custom embedded fonts are not replaced. - May not be suitable for scenarios requiring very strict compliance with PDF standards. - **Cautious Mode:** This fallback mode is automatically used when Fast Mode cannot process a file or is disabled. - Slower processing speed. - In a small number of cases, may result in reduced visual appearance, such as: - Degraded or missing images and graphics. - Differences in text appearance (e.g. size, font style, or spacing). - Missing text when unknown embedded fonts are in use. - Processes PDFs with stricter compliance to specifications. - Replaces custom embedded fonts with known-good fonts. - Preferable only for scenarios requiring very strict compliance with PDF standards, even at the cost of visual fidelity. - **File Inclusion and Exclusion Filtering**: Specify which files to process or exclude using absolute paths or wildcard patterns. - **Output File Categorisation:** Defines how output files are organised. `categorised` organises output files into subdirectories based on processing status (`engine_success`, `conform_success`, `failure`). `mirrored` places successfully processed output files directly in the output directory, maintaining the original input directory structure, and failed files will not be copied. - **Post Processing Summary:** Provides detailed information on processing results, including file statuses, memory usage, and processing time. - **In-Memory Processing**: Supports `engine_memory` and `conform_only_memory` modes, allowing files to be processed entirely in memory using base64-encoded input via standard input, and returning the base64-encoded output files via standard output. Ideal for integration with systems that avoid disk-based I/O. ## Constraints and limitations While Glasswall Conform is a powerful tool, certain constraints and limitations should be considered: - **Image Handling:** Some image colour spaces are unsupported and may be ignored. Additionally, image processing may convert compressed images to a lossless format, which can increase file size. - **Font Handling:** Glasswall Conform supports Base 14 and many Microsoft fonts, but unsupported custom fonts are replaced to mitigate potential risks. - **PDF Structure:** PDFs missing essential structural elements (e.g., root catalog, cross-reference tables) may not be recoverable. - **Memory Usage:** PDFs with many images may consume significant memory. While the tool has been tested with files up to 50 MB, larger files may experience performance issues. - **Color Spaces:** The CalRGB colour space is not supported. - **Graphics Handling:** Support for complex graphics, such as shapes, charts, and graphs, is limited. This version prioritises text integrity. - **Document Recovery:** Severely corrupted PDFs or those with missing structural elements may be unrecoverable. - **Platform Support:** Glasswall Conform is available for both **Windows** and **Linux**. For Windows we provide an `.exe` installer. For Linux we provide `.rpm` and `.deb` packages which support Linux distributions such as **Rocky 9**, **Rocky 8**, **Ubuntu 24**, and **Ubuntu 22**. - **Timeout and Memory Configuration:** The following table presents our findings on how configurable timeout and memory settings impact the overall processing success rate and total runtime when running on **d16-v3 VMs**, each with **16 vCPUs and 64GB RAM**: | **Timeout** | **Memory Limit** | **Runtime (7 VMs)** | **Aggregate Processing Time** | **Processing Time Increase** | **Files Processed** | **Success Rate** | **Success Increase** | |-------------|------------------|---------------------|-------------------------------|------------------------------|---------------------|------------------|----------------------| | **180s** | **4GB** | 65 minutes | 350 minutes | *Baseline* | 2,875 / 3,073 | 93.56% | *Baseline* | | **300s** | **8GB** | 79 minutes | 428 minutes | +23% | 2,939 / 3,073 | 95.64% | +2.08% | | **600s** | **12GB** | 96 minutes | 514 minutes | +47% | 2,947 / 3,073 | 95.90% | +2.34% | | **1200s** | **20GB** | 145 minutes | 689 minutes | +97% | 2,952 / 3,073 | 96.06% | +2.50% | - Increasing timeout and memory results in a **higher success rate** but comes at the cost of increased runtime. - The **300s / 8GB configuration** improves success by **+2.08% over 180s / 4GB**, with a **23% increase in processing time**. - The **600s / 12GB configuration** improves success by **+2.34% over 180s / 4GB**, with a **47% increase in processing time**. - The **1200s / 20GB configuration** provides **only a marginal increase** in processed files (+5 over 600s). - The optimal configuration depends on whether speed or processing success rate is the higher priority. The **300s / 8GB configuration** offers a well-balanced choice when at least **64GB RAM** is available, allowing for **8+ files to be processed in parallel** while delivering a **strong success rate improvement (+2.08%) over 180s / 4GB** and maintaining a **reasonable 23% processing time increase**, making it an efficient middle ground between speed and processing success. ## Licensing Glasswall Conform includes [PyMuPDF](https://pymupdf.readthedocs.io/en/latest/about.html#license-and-copyright) software which is available under both open-source [AGPL](https://www.gnu.org/licenses/agpl-3.0.html) and commercial license agreements via [Artifex](https://artifex.com/licensing/). Glasswall holds a commercial distribution license agreement for the context of Glasswall Conform. **Glasswall Conform** is designed to preprocess PDF files to meet standards for further processing. It extracts and reconstructs visual content and should be used in conjunction with the **Glasswall Embedded Engine** for complete Content Disarm and Reconstruction (CDR) protection. This document offers instructions on using Conform for reconstructing PDF documents, along with several [examples](#examples) for invoking the command-line tool. --- ## Installation Conform is installed as a system-wide command accessible from your terminal. Once installed, the `glasswall_conform` command will be available on your system `PATH`. You may need to restart your terminal session for this change to take effect. ### Windows Conform for Windows is distributed as an `.exe` installer. It installs to `C:\Program Files (x86)\Glasswall Conform` and adds this folder to your system PATH. Install Conform by running the installer and following the instructions: ```bash .\glasswall-conform-1.1.0.exe ``` Or install it silently for automation or CI environments: ```bash .\glasswall-conform-1.1.0.exe /VERYSILENT ``` ### Linux Conform for Linux is distributed as both `.rpm` and `.deb` packages. Both install files to `/opt/glasswall_conform` and create a symbolic link in `/usr/local/bin` to allow running `glasswall_conform` from the command line. #### RPM (e.g. Rocky 9, Rocky 8) ```bash sudo yum -y install ./glasswall_conform-1.1.0-1.x86_64.rpm ``` #### DEB (e.g. Ubuntu 24.04, Ubuntu 22.04) ```bash sudo apt-get -y install ./glasswall-conform_1.1.0_amd64.deb ``` --- ## Setup Before calling `glasswall_conform`, ensure that your environment is set up correctly. For modes that use the Embedded Engine, the directory passed to `--library-directory` must contain the Embedded Engine binaries. Conform verifies this at startup and stops with a short message if they are not present. ### Linux For processing modes that utilise the Embedded Engine, `LD_LIBRARY_PATH` must be set to include the directory containing the Embedded Engine. For example, if the Embedded Engine is at path `/home/azureuser/glasswall/Release-16.2.0` you can temporarily modify `LD_LIBRARY_PATH`: ```bash export LD_LIBRARY_PATH=/home/azureuser/glasswall/Release-16.2.0:$LD_LIBRARY_PATH ``` #### Ubuntu On Ubuntu-based systems, if you encounter the error message `libgthread-2.0.so.0: cannot open shared object file: No such file or directory`, you can resolve it by installing the necessary package with the following command: ```bash DEBIAN_FRONTEND=noninteractive && apt update && apt install -y libglib2.0-0 ``` ### Windows We recommend installing Windows dependencies using [chocolatey](https://chocolatey.org). For all processing modes, [Microsoft Visual C++ Redistributable](https://community.chocolatey.org/packages/vcredist140) must be installed. For processing modes that utilise the Embedded Engine: 1. `PATH` must be set to include the directory containing the Embedded Engine. For example, if the Embedded Engine is at path `C:/glasswall/Release-16.2.0` you can temporarily modify `PATH`: ```sh SET "PATH=%PATH%;C:/glasswall/Release-16.2.0" ``` 2. [OpenSSL light](https://community.chocolatey.org/packages/openssl.light) or [OpenSSL](https://community.chocolatey.org/packages/OpenSSL) must be installed. Example Windows docker installation of vcredist140 and openssl.light using chocolatey: ```docker # escape=` FROM mcr.microsoft.com/windows/servercore:ltsc2022 USER ContainerAdministrator WORKDIR C:\temp\ SHELL ["powershell", "-Command", "$ErrorActionPreference = 'Stop'; $ProgressPreference = 'SilentlyContinue';"] # Download and install Chocolatey, to install OpenSSL and Visual C++ Redistributable RUN Invoke-WebRequest -Uri 'https://chocolatey.org/install.ps1' -OutFile 'install.ps1'; ` ./install.ps1; ` Remove-Item install.ps1; ` Import-Module "$env:ChocolateyInstall/helpers/chocolateyProfile.psm1"; ` choco install -y --fail-on-unfound --no-progress --stop-on-first-package-failure vcredist140; ` choco install -y --fail-on-unfound --no-progress --stop-on-first-package-failure openssl.light; ``` --- ## Licensing Glasswall Conform requires a valid license to operate. To obtain a license file, contact support@glasswall.com. The `--license` argument can be used in all processing modes to specify either a direct path to a license file, or a path to a directory containing a `gwkey.lic` file. - **Engine modes** (`engine`, `engine_memory`): `--license` is optional. If not specified, Conform will automatically search the `--library-directory` for a `gwkey.lic` file. If one is found, it is used as the license. If not found, an error is raised. - **Conform-only modes** (`conform_only`, `conform_only_memory`): `--license` is required, as there is no library directory to search. If the license is missing, invalid or does not contain the required entitlements, Conform will exit with error code 2. ``` --license LICENSE Path to the Glasswall Conform license file, or a directory containing 'gwkey.lic'. Required for conform_only and conform_only_memory modes. In engine modes, defaults to 'gwkey.lic' in the library directory if not specified. ``` --- ## Processing modes Conform is run from the command line and offers several **processing modes** for processing files. When calling `glasswall_conform`, the first positional argument specifies the processing mode. Available processing modes are: - **engine**: Protects files using the Engine. Non-conforming files are reconstructed by Conform and then processed by the Engine. - **conform_only**: Reconstructs files using Conform only, without providing CDR protection. - **engine_memory**: Accepts a base64-encoded file via standard input. Protects a single file in memory using the Engine. If the file is non-conforming, it is reconstructed using Conform and then processed by the Engine. The processed file is returned via standard output, or an error is returned via standard error. - **conform_only_memory**: Accepts a base64-encoded file via standard input. Reconstructs a single file using Conform only, without providing CDR protection. The reconstructed file is returned via standard output, or an error is returned via standard error. To show available processing modes: ``` glasswall_conform -h ``` ### engine This processing mode is the intended default and cleans files using Glasswall CDR technology. It requires access to the Embedded Engine and a valid licence. For an example of invoking this processing mode, see: [End to end protection](#end-to-end-protection). Processed files are sorted into one of three output subdirectories: 1. **01_engine_success**: Files successfully processed by the Embedded Engine without the need for reconstruction by Conform. 2. **02_conform_engine_success**: PDF files that were initially unable to be processed by the Embedded Engine, but were reconstructed by Conform and then successfully processed by the Embedded Engine. 3. **03_failure**: Files that failed to be processed using both the Embedded Engine and Conform, or that contain content that has been set to disallow using a custom content management policy. To show the command line arguments for the `engine` processing mode: ``` glasswall_conform engine -h ``` ### conform_only This processing mode reconstructs files without utilising the Embedded Engine. It does not provide CDR protection. For an example of invoking this processing mode, see: [Reconstructing files without CDR protection](#reconstructing-files-without-cdr-protection) Processed files are sorted into one of two output subdirectories: 1. **01_conform_success**: Files successfully reconstructed by Conform. 3. **02_failure**: Files that failed to be reconstructed by Conform. To show the command line arguments for the `conform_only` processing mode: ``` glasswall_conform conform_only -h ``` ### engine_memory This mode accepts a base64-encoded file via standard input and processes it using the Embedded Engine. If the file is non-conforming, it is reconstructed by Conform, then processed by the Engine. The final output is returned via standard output, or an error is returned via standard error. No files are written to disk. This mode is ideal for integrating with systems that hold files in memory and do not rely on filesystem input or output. For an example of invoking this processing mode, see: [Processing files in memory without reading from or writing to disk](#processing-files-in-memory-without-reading-from-or-writing-to-disk). To show the command line arguments for the `engine_memory` processing mode: ``` glasswall_conform engine_memory -h ``` The `--file-name` optional argument can be used to specify the name of the in-memory file. This is used when writing logs and the post processing summary, and defaults to the first 8 characters of the base64 encoded data if not specified. ### conform_only_memory This mode accepts a base64-encoded file via standard input and reconstructs it using Conform only (without CDR protection). The reconstructed file is returned via standard output, or an error is returned via standard error. No files are written to disk. For an example of invoking this processing mode, see: [Processing files in memory without reading from or writing to disk](#processing-files-in-memory-without-reading-from-or-writing-to-disk). To show the command line arguments for the `conform_only_memory` processing mode: ``` glasswall_conform conform_only_memory -h ``` --- ## Testing A dataset of PDF test files for evaluating Conform is available upon request. Please contact us to request access to the test files via Kiteworks. --- ## Examples - [Examples](#examples) - [End to end protection](#end-to-end-protection) - [Reconstructing files without CDR protection](#reconstructing-files-without-cdr-protection) - [Processing files in memory without reading from or writing to disk](#processing-files-in-memory-without-reading-from-or-writing-to-disk) - [Fast mode and cautious mode](#fast-mode-and-cautious-mode) - [Glasswall Python Wrapper functionality](#glasswall-python-wrapper-functionality) - [Multiprocessing](#multiprocessing) - [Logging](#logging) - [Customise content handling rates](#customise-content-handling-rates) - [Watermarking](#watermarking) - [CID suppression](#cid-suppression) - [Font replacement](#font-replacement) - [File inclusion and exclusion filtering](#file-inclusion-and-exclusion-filtering) - [Output file structure and categorisation](#output-file-structure-and-categorisation) - [Example `categorised` output structure](#example-categorised-output-structure) - [Example `mirrored` output structure](#example-mirrored-output-structure) - [Post processing summary](#post-processing-summary) --- ### End to end protection This example demonstrates using the [engine](#engine) processing mode at its most basic level. ``` glasswall_conform engine -i /home/azureuser/input_files -o /home/azureuser/output_files -l /home/azureuser/glasswall/Release-16.2.0 ``` Example input directory: ``` /home/azureuser/input_files conforming_docx.docx conforming_pdf.pdf corrupt_docx.docx nonconforming_pdf.pdf unsupported_filetype.txt ``` Example output directory after processing: ``` /home/azureuser/output_files ├───01_engine_success │ conforming_docx.docx │ conforming_pdf.pdf │ ├───02_conform_engine_success │ nonconforming_pdf.pdf │ └───03_failure corrupt_docx.docx unsupported_filetype.txt ``` Note that the subdirectory names can be customised using the following arguments: - **--engine-success-path**: Optional. Output subdirectory name for files that were successfully processed by the Embedded Engine without the need for reconstruction by Conform. Default 01_engine_success - **--conform-success-path**: Optional. Output subdirectory name for files that were initially unable to be processed by the Embedded Engine, but were reconstructed by Conform and then successfully processed by the Embedded Engine. Default 02_conform_engine_success - **--failure-path**: Optional. Output subdirectory name for files that failed to be processed using both the Embedded Engine and Conform. Default 03_failure If it is desired that all successfully protected files are written to the same output directory, regardless of whether or not Conform was used to reconstruct the file, you can specify to write files to the same success subdirectory path. For example: ``` glasswall_conform engine -i /home/azureuser/input_files -o /home/azureuser/output_files -l /home/azureuser/glasswall/Release-16.2.0 --engine-success-path success --conform-success-path success --failure-path failure ``` Example truncated terminal output after processing: ``` Glasswall Conform processed 3/5 files (60.00%) Glasswall Conform failed to process 2/5 files. (40.00%) Exceptions: PdfExtractionError (Total: 2) - 1x Unable to extract content from PDF: '/home/azureuser/input_files/corrupt_docx.docx' - 1x Unable to extract content from PDF: '/home/azureuser/input_files/unsupported_filetype.txt' 2024-11-06 14:28:50.242 glasswall_conform.config.logging INFO engine_mode Total elapsed time: 5.55 seconds ``` --- ### Reconstructing files without CDR protection The `conform_only` processing mode does not provide CDR protection, and requires an input directory `-i`, an output directory `-o` and a `--license` path. See [conform_only](#conform_only). ``` glasswall_conform conform_only -i /home/azureuser/input_files -o /home/azureuser/output_files --license /home/azureuser/gwkey.lic ``` --- ### Processing files in memory without reading from or writing to disk The [engine_memory](#engine_memory) and [conform_only_memory](#conform_only_memory) processing modes can be used to process files in memory without the use of I/O. If processing is successful, the base64-encoded output file is returned via standard output. If an error occurred during processing, an error message and the post processing summary will be written to standard error. Example standard error for a timeout failure: ``` Error: Processing failed for file: 'hus11976.pdf'. Summary: {'conform_version': '0.11.2', 'operating_system': 'Windows', 'summary_verbosity': 'all', 'processing_rates': {'success': 0.0, 'failure': 100.0}, 'processing_counts': {'success': 0, 'failure': 1, 'total': 1}, 'processing_time': {'elapsed_seconds': 8.25, 'files_per_sec': 0.12, 'secs_per_file': 8.25}, 'processing_arguments': {'mode': 'engine_memory', 'library_directory': 'C:/azure/sdk.editor/2.1394.0/build-sdk-editor-windows-amd64-dev_license', 'cautious_mode': False, 'max_workers': 1, 'timeout_seconds': 5.0, 'memory_limit_gib': 11.96, 'function_name': 'protect_file', 'content_management_policy': None}, 'processing_success': [], 'processing_failure': [{'file_name': 'hus11976.pdf', 'timed_out': True, 'out_of_memory': False, 'max_memory_used_in_gib': 0.32227325439453125, 'elapsed_time': 5.0007593631744385, 'exception': 'TimeoutError()', 'success': False}]} ``` Basic usage examples: **Pipe a base64-encoded string of an in memory PDF file directly into glasswall_conform** ```cmd echo "U29tZUJhc2U2NERhdGE=" | glasswall_conform.exe engine_memory -l "C:/azure/sdk.editor/2.1394.0" --file-name "SomeBase64Data" ``` **Or using Python via subprocess** ```python import base64 import os import subprocess # File in memory, for this example simply loaded from a file path file_path = r"C:\conform\input\Set-08-016599.pdf" with open(file_path, "rb") as f: file_bytes = f.read() # Convert to base64-encoded string encoded_file_bytes = base64.b64encode(file_bytes).decode("utf-8") file_name = os.path.basename(file_path) command = " ".join( [ "glasswall_conform", "engine_memory", '-l "C:/azure/sdk.editor/2.1394.0"', f'--file-name "{file_name}"', # Optional, used for summary and logs f'--summary-path "C:/conform/summary_{file_name}.json"', # Optional '--watermark "Processed for security: Visual elements may vary"', # Optional ] ) # Run Conform with the base64-encoded string as an input result = subprocess.run(command, input=encoded_file_bytes, text=True, capture_output=True, shell=True) if result.stderr: # Conform failed, handle error gracefully here print(result.stderr) else: # Conform succeeded, convert the conformed file from base64 to bytes conformed_file_bytes = base64.b64decode(result.stdout) # Do something with the conformed file bytes, e.g. write to a file with open("conformed_file.pdf", "wb") as f: f.write(conformed_file_bytes) ``` Similarly, `conform_only_memory` mode can be used by replacing `engine_memory` in the above examples, omitting the `-l` argument and specifying `--license` as it is required for this mode. --- ### Fast mode and cautious mode Fast mode is the default processing mode in Conform. It offers the fastest processing speed and the best visual appearance for PDF files, but may not be suitable for scenarios that require very strict compliance with PDF specifications. If fast mode is disabled or cannot process a file, Conform automatically falls back to cautious mode. This mode prioritises compliance and risk reduction by replacing embedded fonts, which helps mitigate issues associated with custom or unknown fonts. Cautious mode may result in lower visual fidelity, such as degraded or missing images, inconsistent font sizes, or missing text. - Disabling fast mode is only recommended when very strict compliance with PDF standards is essential, even at the cost of visual fidelity. - Disabling cautious mode is only recommended when preserving embedded fonts is essential, or when visual appearance is more important than Conform being able to successfully process a wider range of PDFs. Fast mode can be disabled using the optional `--disable-fast-mode` command line argument. Cautious mode can be disabled using the optional `--disable-cautious-mode` command line argument. When Conform processes a file successfully using fast mode: - Fastest processing speed. - Best visual appearance. - Custom embedded fonts are not replaced. - May not be suitable for scenarios requiring very strict compliance with PDF standards. When Conform uses the cautious mode fallback: - Slower processing speed. - In a small number of cases, may result in reduced visual appearance, such as: - Degraded or missing images and graphics. - Differences in text appearance (e.g. size, font style, or spacing). - Missing text when unknown embedded fonts are in use. - Processes PDFs with stricter compliance to specifications. - Replaces custom embedded fonts with known-good fonts. --- ### Glasswall Python Wrapper functionality In the [engine](#engine) processing mode, the `protect_file` function from the Glasswall Python Wrapper is used by default to process files using the Embedded Engine. This can be changed using the optional `-f` command line argument. A default `sanitise` content management policy is applied if a policy file is not specified using the optional `-c` command line argument. The required `-l` command line argument should point to a directory containing the Embedded Engine. The following arguments relate to the Glasswall Python Wrapper: ``` -l LIBRARY_DIRECTORY, --library-directory LIBRARY_DIRECTORY Required. Path to directory containing the Embedded Engine. -f FUNCTION_NAME, --function-name FUNCTION_NAME Optional. Glasswall Python Wrapper function name to call during multiprocessing, such as 'protect_file' or 'export_file'. Default: 'protect_file'. -c CONTENT_MANAGEMENT_POLICY, --content-management-policy CONTENT_MANAGEMENT_POLICY Optional. Path to Embedded Engine content management policy file. If not provided, the default 'sanitise' policy is used. --log-level-console-wrapper {CRITICAL,ERROR,WARNING,INFO,DEBUG,NOTSET} Optional. Set logging level for writing Glasswall Python Wrapper logs to console. Default INFO. ``` Example: ``` glasswall_conform engine -i /home/azureuser/input_files -o /home/azureuser/output_files -l /home/azureuser/glasswall/Release-16.2.0 -f protect_file -c /home/azureuser/glasswall/config.xml ``` --- ### Multiprocessing All processing modes leverage the Glasswall Python Wrapper's `GlasswallProcessManager` to efficiently process files concurrently. The following arguments relate to multiprocessing: ``` -w MAX_WORKERS, --max-workers MAX_WORKERS Optional. Maximum workers for multiprocessing, 0=auto. Default: 0. -t TIMEOUT_SECONDS, --timeout-seconds TIMEOUT_SECONDS Optional. Multiprocessing timeout per file in seconds. Default: 180. --overall-timeout-seconds OVERALL_TIMEOUT_SECONDS Optional. Kill the whole process after this many seconds and report on stderr. Unlike --timeout-seconds, which bounds each file, this bounds the run itself and so also catches a parent that is stuck waiting on a worker. 0 disables it. Defaults to --timeout-seconds plus 120 in the memory modes, which handle one file, and disabled in the directory modes, where the total work is unbounded. -m MEMORY_LIMIT_GIB, --memory-limit-gib MEMORY_LIMIT_GIB Optional. Multiprocessing memory limit per file in GiB, 0=auto (4GiB min, worker distributed max). Default: 0. ``` `--timeout-seconds` and `--overall-timeout-seconds` bound different things. `--timeout-seconds` limits the work done on each file, and is enforced per file. `--overall-timeout-seconds` limits the run as a whole, so it also covers time spent outside processing a file, including a run that stops making progress while waiting on a worker. When it fires, Conform writes a message to standard error, stops any workers it started and exits with a non-zero code. Example: ``` glasswall_conform engine -i /home/azureuser/input_files -o /home/azureuser/output_files -l /home/azureuser/glasswall/Release-16.2.0 -t 300 -m 12 ``` --- ### Logging The default logging level for Conform and the Glasswall Python Wrapper is `INFO`. The following arguments relate to logging: ``` --log-level-console {CRITICAL,ERROR,WARNING,INFO,DEBUG,NOTSET} Optional. Set logging level for writing logs to console. Default INFO. --log-level-file {CRITICAL,ERROR,WARNING,INFO,DEBUG,NOTSET} Optional. Set logging level for writing logs to file. If not provided, logs will not be written to file. --log-path LOG_PATH Optional. Path to output log file. Default is a timestamp-named file located at: '%TEMP%/glasswall_conform/logs'. --log-level-console-wrapper {CRITICAL,ERROR,WARNING,INFO,DEBUG,NOTSET} Optional. Set logging level for writing Glasswall Python Wrapper logs to console. Default INFO. ``` To suppress most logging: ``` glasswall_conform engine -i /home/azureuser/input_files -o /home/azureuser/output_files -l /home/azureuser/glasswall/Release-16.2.0 --log-level-console CRITICAL --log-level-console-wrapper CRITICAL ``` --- ### Customise content handling rates This section is only applicable when [fast mode](#fast-mode-and-cautious-mode) is disabled. By default, Conform generates an output file whenever possible, even if only a portion of the original document's content has been successfully handled. This behaviour might not always be desirable, and can be customised for different types of content within each document. Conform uses "best guesses" when handling malformed, corrupt, or unsupported text content to ensure that as much text as possible is transferred from the original document to the conformed document. For example, if the stroke colour of the text is malformed or in an unsupported colour format, the text is retained in the output document, with the stroke colour defaulting to black. This "best guess" approach may result in text that appears similar to the original, or in some cases, text that is not visible but still present in the output document. As we cannot guarantee that our best guess will handle the text in the same way as in the original document, the handling rate reflects this as content that has not been fully handled. Consequently, a low handling rate for text does not always indicate that the document will look visually different when best guesses are applied. There are three arguments available to set the minimum success rates when handling content: ``` --text-min-success-rate TEXT_MIN_SUCCESS_RATE Optional. The minimum success rate for processing text. Default: 0.0. --image-min-success-rate IMAGE_MIN_SUCCESS_RATE Optional. The minimum success rate for processing images. Default: 0.0. --graphic-min-success-rate GRAPHIC_MIN_SUCCESS_RATE Optional. The minimum success rate for processing graphics. Default: 0.0. ``` If the minimum content handling rate value is not met then processing for the given file will be deemed a failure and the output file will not be written. --- ### Watermarking Watermarking is disabled by default, but can be enabled using the `--watermark` argument. Text will be added with font size 12 in a semi-transparent dark grey colour. The watermark is usually positioned at the top-right of the document, however depending on the rotation that has been applied to the page, the orientation may differ. The maximum text length for a watermark is currently 256 characters. ``` --watermark WATERMARK Optional. Adds a watermark to each page of the reconstructed document. Default '' (disabled). ``` Example: ``` glasswall_conform engine -i /home/azureuser/input_files -o /home/azureuser/output_files -l /home/azureuser/glasswall/Release-16.2.0 --watermark "Glasswall Conform" ``` --- ### CID suppression This section is only applicable when [fast mode](#fast-mode-and-cautious-mode) is disabled. In PDFs, some fonts use a system called CID (Character Identifier) to manage large sets of characters. When constructing a new PDF, if the tool encounters characters that cannot be processed, it replaces them with a default question mark character (?). You can adjust how unprocessable CIDs are represented in your PDFs using the `--suppress-cid` argument: ``` --suppress-cid SUPPRESS_CID Optional. Replace CID metadata that may be printed to the visual layer due to font array omissions with the supplied string, with placeholder text. Glasswall Conform restricts the processing of PDFs to only known secure fonts. This is a deliberate security feature to make the PDF conform safely. Default '■'. ``` Example: ``` glasswall_conform engine -i /home/azureuser/input_files -o /home/azureuser/output_files -l /home/azureuser/glasswall/Release-16.2.0 --suppress-cid "?" ``` --- ### Font replacement This section is only applicable when [fast mode](#fast-mode-and-cautious-mode) is disabled. Conform supports bold, italic, and bold italic variants of the base 14 Type1 fonts and the Cambria font. Conform also supports some custom fonts. The base 14 Type1 fonts are: - Courier, Courier-Bold, Courier-Oblique, Courier-BoldOblique - Helvetica, Helvetica-Bold, Helvetica-Oblique, Helvetica-BoldOblique - Times-Roman, Times-Bold, Times-Italic, Times-BoldItalic - Symbol - ZapfDingbats Embedded fonts that are not supported may be replaced with the Cambria font. If Cambria does not support a glyph from an embedded font, the character is suppressed. For more information on this, see [CID suppression](#cid-suppression). By default, some commonly embedded sans serif fonts are replaced with Helvetica instead of Cambria for visual similarity. This, and other font replacement features, can be modified using these arguments: ``` --disable-base-14-fonts Optional. Disable matching embedded fonts to base 14 fonts. This will result in more fonts being replaced by the fallback font, Cambria. Default False. --disable-custom-fonts Optional. Disable matching embedded fonts to custom fonts. This will result in lower support for custom embedded fonts, and more fonts being replaced by the fallback font, Cambria. Default False. --disable-sans-serif-replacement Optional. Disable replacing some sans serif fonts with Helvetica instead of the fallback font, Cambria. This will result in some replaced sans serif fonts looking more visually different when compared to the original file. Default False. ``` Example: ``` glasswall_conform engine -i /home/azureuser/input_files -o /home/azureuser/output_files -l /home/azureuser/glasswall/Release-16.2.0 --disable-custom-fonts ``` --- ### File inclusion and exclusion filtering Conform allows additional control over which files in the input directory are processed by using **include and exclude filters**. These filters let you specify which files to process or ignore using basic **Unix shell-style wildcards** directly from the command line. If a file matches both an inclusion and an exclusion rule, it will be **excluded**. By default, if the `--include-files` and `--exclude-files` arguments are omitted, Conform will process all files that are present in the input directory. The following arguments relate to file inclusion and exclusion: ``` --include-files INCLUDE_FILES Optional. Can be either a path to a file containing file paths/patterns or a semicolon-separated list of patterns (e.g. '*.pdf;*/SET_03/*'). Only matching files will be processed. If None, all files are included. Default: None. --exclude-files EXCLUDE_FILES Optional. Can be either a path to a file containing file paths/patterns or a semicolon-separated list of patterns. Any matching files will be excluded from processing. If None, no files are excluded. Default: None. ``` The following table demonstrates examples of some patterns that can be used: | Pattern | Meaning | Example | Matches | Does Not Match | | -------- | -------------------------------------- | ---------------- | -------------------------- | -------------------------- | | `*` | Matches everything | `*.pdf` | `file.pdf`, `report.pdf` | `file.docx` | | `?` | Matches any single character | `file_?.pdf` | `file_1.pdf`, `file_A.pdf` | `file_10.pdf` | | `[seq]` | Matches any character in `seq` | `file_[AB].pdf` | `file_A.pdf`, `file_B.pdf` | `file_C.pdf` | | `[!seq]` | Matches any character **not** in `seq` | `file_[!AB].pdf` | `file_C.pdf`, `file_D.pdf` | `file_A.pdf`, `file_B.pdf` | #### Case sensitivity considerations File names are **case-sensitive on Linux** but **case-insensitive on Windows**. This affects how file paths or patterns are interpreted across different operating systems. - On **Linux**, `report.pdf` and `Report.pdf` are treated as different files. - On **Windows**, both are considered the same file. **Recommendation:** To ensure consistency across platforms, use **consistent casing** in file names and patterns. If working across multiple environments, consider using wildcard patterns (`*`) where appropriate to avoid mismatches. #### Handling single file inclusions If specifying a single file with `--include-files`, be aware that Conform first checks whether the provided value is a **file on disk**, and if it is not then the value is treated as a **pattern**. **Potential issue:** If a user specifies: ```sh --include-files "/home/azureuser/input_files/first.pdf" ``` Conform will see that `/home/azureuser/input_files/first.pdf` exists as a file, and attempt to read from it as a list file that contains multiple paths or patterns. **Solution:** To explicitly indicate that this is a pattern for a single file, append a **trailing semicolon**: ```sh --include-files "/home/azureuser/input_files/first.pdf;" ``` This ensures that Conform treats the path as a pattern rather than a list file. #### **Include specific PDF files** To process only PDFs with "report" in the filename: ```sh glasswall_conform engine -i /home/azureuser/input_files -o /home/azureuser/output_files -l /home/azureuser/glasswall/Release-16.2.0 --include-files "*report*.pdf" ``` **Result:** Only files like `annual_report.pdf`, `summary_report_2023.pdf`, etc., are processed. #### **Exclude specific PDF files** To process all PDFs **except** ones containing "draft" in the name: ```sh glasswall_conform engine -i /home/azureuser/input_files -o /home/azureuser/output_files -l /home/azureuser/glasswall/Release-16.2.0 --exclude-files "*draft*.pdf" ``` **Result:** All PDFs are processed, except files like `proposal_draft.pdf` and `internal_draft_v2.pdf`. #### **Exclude an entire directory** To exclude all files inside `/home/azureuser/input_files/archive/`: ```sh glasswall_conform engine -i /home/azureuser/input_files -o /home/azureuser/output_files -l /home/azureuser/glasswall/Release-16.2.0 --exclude-files "*/archive/*" ``` **Result:** Everything inside `/home/azureuser/input_files/archive/` is skipped. #### **Include and exclude together** If a file matches both an inclusion and an exclusion rule, it will be **excluded**. To process all files from `SET_03`, but exclude files containing "error_log": ```sh glasswall_conform engine -i /home/azureuser/input_files -o /home/azureuser/output_files -l /home/azureuser/glasswall/Release-16.2.0 --include-files "*/SET_03/*" --exclude-files "*error_log*" ``` **Result:** Only files from `SET_03/` are processed, except any containing "error_log" in the filename. #### **Using a file for large lists** For more complex filtering, you can provide a file containing multiple patterns or absolute file paths instead of specifying them directly. **Example using an inclusion list file:** ```sh glasswall_conform engine -i /home/azureuser/input_files -o /home/azureuser/output_files -l /home/azureuser/glasswall/Release-16.2.0 --include-files "include_list.txt" ``` **Example `include_list.txt`:** ``` */SET_03/*.pdf *reports_2023_*.pdf /home/azureuser/input_files/SET_02/splat.pdf ``` **Result:** Processes only files from `SET_03/`, files containing `reports_2023_`, and the specific file `/home/azureuser/input_files/SET_02/splat.pdf`. --- ### Output file structure and categorisation The directory structure for output files can be customised for both the [engine](#engine) and [conform_only](#conform_only) processing modes using the `--output-structure` command line argument. ``` --output-structure {categorised,mirrored} Optional. Defines the directory structure of output files. 'categorised' organises output files into subdirectories based on processing status ('engine_success', 'conform_success', 'failure'). 'mirrored' places successfully processed output files directly in the output directory, maintaining the original input directory structure, and failed files will not be copied. Default: categorised. ``` If omitted, the default `categorised` structure is used. Additional options are available to customise the category subdirectory names: - **--engine-success-path**: Optional. Output subdirectory name for files that were successfully processed by the Embedded Engine without the need for reconstruction by Conform. Default 01_engine_success - **--conform-success-path**: Optional. Output subdirectory name for files that were initially unable to be processed by the Embedded Engine, but were reconstructed by Conform and then successfully processed by the Embedded Engine. Default 02_conform_engine_success - **--failure-path**: Optional. Output subdirectory name for files that failed to be processed using both the Embedded Engine and Conform. Default 03_failure #### Example `categorised` output structure ``` glasswall_conform engine -i /home/azureuser/input_files -o /home/azureuser/output_files -l /home/azureuser/glasswall/Release-16.2.0 ``` Example input directory: ``` /home/azureuser/input_files conforming_docx.docx conforming_pdf.pdf corrupt_docx.docx nonconforming_pdf.pdf unsupported_filetype.txt ``` Example output directory after processing: ``` /home/azureuser/output_files ├───01_engine_success │ conforming_docx.docx │ conforming_pdf.pdf │ ├───02_conform_engine_success │ nonconforming_pdf.pdf │ └───03_failure corrupt_docx.docx unsupported_filetype.txt ``` When using the `categorised` output structure, if it is desired that all successfully protected files are written to the same output directory, regardless of whether or not Conform was used to reconstruct the file, you can specify to write files to the same success subdirectory path. For example: ``` glasswall_conform engine -i /home/azureuser/input_files -o /home/azureuser/output_files -l /home/azureuser/glasswall/Release-16.2.0 --engine-success-path success --conform-success-path success --failure-path failure ``` #### Example `mirrored` output structure ``` glasswall_conform engine -i /home/azureuser/input_files -o /home/azureuser/output_files -l /home/azureuser/glasswall/Release-16.2.0 --output-structure mirrored ``` Example input directory: ``` /home/azureuser/input_files conforming_docx.docx conforming_pdf.pdf corrupt_docx.docx nonconforming_pdf.pdf unsupported_filetype.txt ``` Example output directory after processing: ``` /home/azureuser/output_files conforming_docx.docx conforming_pdf.pdf nonconforming_pdf.pdf ``` --- ### Post processing summary By default, a summary is written after Conform has finished processing files. The summary provides detailed information such as return statuses, processing time, and memory usage for each file. The `--summary-verbosity` argument controls which files are included in the summary. This setting is independent of the logging level and does not affect detailed log outputs. Available Options - `all` (default) - Includes both successfully processed and failed files. - `failure` - Includes only failed files. - `success` - Includes only successfully processed files. - `none` - Disables the summary output completely. The `--summary-path` argument can be used to write the summary to disk as a JSON file instead of only displaying it in the terminal. Example to include only failed files in the summary output: ``` glasswall_conform engine -i /home/azureuser/input_files -o /home/azureuser/output_files -l /home/azureuser/glasswall/Release-16.2.0 --summary-verbosity failure --summary-path /home/azureuser/conform_summary.json ``` Example to disable the summary output entirely: ``` glasswall_conform engine -i /home/azureuser/input_files -o /home/azureuser/output_files -l /home/azureuser/glasswall/Release-16.2.0 --summary-verbosity none --summary-path /home/azureuser/conform_summary.json ``` Example summary JSON output (Windows): ```json { "conform_version": "0.10.1", "operating_system": "Windows", "summary_verbosity": "all", "processing_rates": { "success": 50.0, "failure": 50.0 }, "processing_counts": { "success": 2, "failure": 2, "total": 4 }, "processing_time": { "elapsed_seconds": 43.61, "files_per_sec": 0.09, "secs_per_file": 10.9 }, "processing_arguments": { "mode": "engine", "input_directory": "C:\\conform\\input", "output_directory": "C:\\conform\\output", "library_directory": "C:\\azure\\sdk.editor\\2.1394.0", "cautious_mode": false, "max_workers": 3, "timeout_seconds": 180, "memory_limit_gib": 4.35, "function_name": "protect_file", "content_management_policy": null, "include_files": null, "exclude_files": null, "output_structure": "categorised" }, "processing_success": [ { "input_file": "C:\\conform\\input\\pal1.bmp", "output_file": "C:\\conform\\output\\01_engine_success\\pal1.bmp", "engine_status": "OK(0)", "max_memory_used_in_gib": 0.11124420166015625, "elapsed_time": 0.9149298667907715, "success": true }, { "input_file": "C:\\conform\\input\\Set-08-016599.pdf", "output_file": "C:\\conform\\output\\02_conform_engine_success\\Set-08-016599.pdf", "engine_status": "GeneralFail(-1)", "engine_GW2FileErrorMsg": "[FAILURE_LOG_SEM_FONTS_0021897368] Key /FirstChar must be present in a Type 1 Font dictionary other than for standard 14. fonts.", "engine_conform_fast_status": "GeneralFail(-1)", "engine_conform_fast_GW2FileErrorMsg": "[FAILURE_LOG_SEM_FONTS_0021897368] Key /FirstChar must be present in a Type 1 Font dictionary other than for standard 14. fonts.", "engine_conform_cautious_status": "OK(0)", "max_memory_used_in_gib": 0.22198104858398438, "elapsed_time": 1.8940067291259766, "success": true } ], "processing_failure": [ { "input_file": "C:\\conform\\input\\pal1_corrupt.bmp", "engine_status": "FileTypeUnknown(-7)", "engine_GW2FileErrorMsg": "Unable to determine file type", "engine_conform_fast_status": "PdfFastProcessError()", "engine_conform_cautious_status": "PdfExtractionError(Unable to extract content from PDF: 'C:\\conform\\input\\pal1_corrupt.bmp')", "exit_code": 0, "timed_out": false, "out_of_memory": false, "max_memory_used_in_gib": 0.13513565063476562, "elapsed_time": 0.8690056800842285, "success": false }, { "input_file": "C:\\conform\\input\\Straw120556398.pdf", "timed_out": false, "out_of_memory": true, "max_memory_used_in_gib": 4.3571624755859375, "elapsed_time": 41.976775884628296, "exception": "MemoryError()", "success": false } ] } ``` # Overview Image Resizer is a command-line application built with .NET designed to resize images. It allows you to resize a provided image by specifying a target size in bytes and an output folder. The tool aims to maintain the original image's likeness. Any differences should stay within the tolerances of human vision. The output of this tool is targeting a value of 90% or above visuality similarity score in comparison tests using a cross correlation algorithm. ## Import / Export integration Supported files can be processed using the export function of the Glasswall Engine. This function outputs a ZIP file containing all images extracted from the original document. This feature allows for custom processing of the images, which can then be reinserted into the original document using the import function. However, there are some integration limitations to be aware of: - Changing the file names of images to anything different from the original names will result in a failure to place the images back into the document during the import process. - File names are case sensitive. - If the file size of an image exceeds the original size, the import may not be successful. The tool enables users to resize images to a target size that is less than or equal to the original file size, ensuring a smooth import process. ## Usage: gw_image_resizer [options] ## Options: ``` -i, --in (REQUIRED) Input file name -o, --out (REQUIRED) Output file name -t, --target (REQUIRED) Target maximum output file size -l, --memory-limit (OPTIONAL) An optional number that specifies the maximum amount of system memory percent the image resize is limited to [default: 50] -T, --temp-dir (OPTIONAL) Defines the path to an existing directory where temporary files are created during processing [default: $TMPDIR/.gw-image-resizer] -?, -h, --help Show help and usage information --version Show version information ```` ## Exit codes | Exit Code | Description | | --------- | ------------------------------------------------------------------ | | 0 | Successfully resized or the file was already the target size | | 1 | The input file target size is less than zero | | 2 | The input file could not be found | | 3 | The input file could not be loaded as it is corrupt or unsupported | | 4 | The input file could not be loaded for an unknown reason | | 5 | Part of the output path does not exist | | 6 | Could not write the output file for an unknown reason | | 7 | General error in the magick library when trying to resize | | 8 | Could not reach the desired target size | | 9 | Unexpected error | | 10 | Memory limit option was not between 1-100 | | 11 | Temporary directory path could not be found | | 12 | Could not write to temporary directory | ## Temporary directory In order to ensure the successful resizing of files, during processing the image resizer tool may use a temporary directory located at "$TMPDIR/.gw-image-resizer" or "/tmp/.gw.image-resizer". This can be overridden to an existing writeable directory with the optional command line option "--temp-dir" (or "-T" for shorthand.) **NOTE:** If the temporary directory is not writeable (e.g. the current user does not have adequate permission) then the application will fail when image processing data needs to be cached. It is recommended to test that the user is able to write to the directory to avoid sporadic issues during operation. ## Limiting memory By default the memory allocation used for resizing images is limited to 50% of system memory. This can be overridden to a value between 1-100 with the optional command line option "-l". ## Support The tool currently supports the following: - JPEG - PNG - BMP - TIFF (see GeoTiff known issue) - GIF Other files supported by ImageMagick may load correctly but have not explicitly been implemented. ## Resizing strategy All supported image files go through a set of methods to reach the target size. For example: #### Compression quality: JPEG / JPG files have quality set to 25. PNG files are set to 100. Other file types remain unchanged, however the compression method is always set to LZW. #### GIF optimization Since GIFs can be comprised of multiple images and layers, there are potential optimizations to be made in order to reduce file size. If the different layers of an image are of different sizes, Optimization is done by Image Magick to reduce the file size. Otherwise the GIF is Coalesced, which merges similarities in images in the sequence to reduce size. Transparency is also optimized at this point, replacing duplicate pixels data across the sequence with transparency. #### Quantize All images will then be Quantized, which will cap the number of colors to 128, not changing the value if less than this already. #### Downscaling If the target is not yet reached, the image will then be resized until it is. Resolution is reduced to 95% of the original continuously until the target is reached or it cannot reduce the resolution any further. ## Known issues Processing can remove locational data from the pixels in GeoTIFF. In addition, regular TIFF's may have colors flipping in grayscale images. # Third party libraries | Library Name | Library Version | Type | License Type | License Url | Project Url | Source Url | | -------------------------- | ------------------- | ------ | ------------ | ----------------------------------------------------------------------- | -------------------------------------------------- | -------------------------------------------------- | | System.CommandLine | 2.0.0-beta4.22272.1 | Library | MIT | [Link](https://github.com/dotnet/command-line-api/blob/main/LICENSE.md) | [Link](https://github.com/dotnet/command-line-api) | [Link](https://github.com/dotnet/command-line-api) | | System.CommandLine.Hosting | 0.4.0-alpha.22272.1 | Library | MIT | [Link](https://github.com/dotnet/command-line-api/blob/main/LICENSE.md) | [Link](https://github.com/dotnet/command-line-api) | [Link](https://github.com/dotnet/command-line-api) | | Magick.NET-Q8-AnyCPU | 13.10.0 | Library | Apache-2.0 | [Link](https://github.com/dlemstra/Magick.NET/blob/main/License.txt) | [Link](https://github.com/dlemstra/Magick.NET) | [Link](https://github.com/dlemstra/Magick.NET) | Glasswall's Image Resizer tool utilizes the "Magick.NET-Q8-AnyCPU" NuGet package which is a dotnet wrapper for ImageMagick. ImageMagick itself is a free open source tool that is used for editing and manipulating digital images. # Use Case: Import/Export with the Glasswall Embedded Engine Image resizer can sit between the Embedded Engine content export and import functionality as a method of reducing the size in bytes of embedded images before importing back into their parent document. Some users may perform extra operations on these embedded images before importing them back into the clean file. The image resizer can help manage the size of these images and even reduce the overall size of the document in some cases. To read more about the Glasswall Embedded Engine content export and import, see [Content Export and Import](/embedded-engine/embedded-engine-content-export-and-import). In this guide, we will use the Embedded Engine wrappers to perform export and import on a sample PDF file, then resize the images before importing back into the original format. Examples and a breakdown of the various Embedded Engine API functions can be found [here](/embedded-engine/embedded-engine-api-functions). ## Getting started ### Prerequisites - The Embedded Engine SDK - A valid license with the Export and Import capabilities - The relevant libraries for your chosen Embedded Engine wrapper - The Image Resizer CLI ### Sample file Attached is a PDF sample file, the document contains 2 JPEG embedded images: [Sample.pdf](/.attachments/Sample-501960af-a956-4cff-9b08-eff08d69d5e6.pdf) ### Configuration The following XML file includes a policy set to sanitise all content. It is also configured to export images to their raw format under the `sysconfig` section, see `export_embedded_images`. For more on engine configuration info see [Configuration Management](/embedded-engine/embedded-engine-system-configuration). ```xml sanitise sanitise sanitise sanitise sanitise sanitise sanitise sanitise sanitise sanitise sanitise sanitise sanitise sanitise sanitise sanitise sanitise sanitise sanitise sanitise sanitise sanitise sanitise sanitise sanitise sanitise sanitise sanitise sanitise sanitise sanitise sanitise sanitise sanitise sanitise sanitise sanitise sanitise sanitise sisl false ``` ## Step 1: Export Let's first export the original file. Ensure paths to the XML configuration file and your Glasswall license are to hand. > Note: These examples do not contain error handling logic, see [Return Types](/embedded-engine/embedded-engine-api-overview#return-types) for more information. C# ```csharp using glasswall_core2; var filePath = ""; var pathToConfig = ""; var licenseFilePath = ""; var outputDirectory = ""; Glasswall glasswall = new Glasswall(); int session = 0; session = glasswall.OpenSession(); glasswall.RegisterLicenseFile(session, licenseFilePath); glasswall.RegisterPoliciesFile(session, pathToConfig, 0); glasswall.RegisterInputFile(session, filePath); glasswall.RegisterExportFile(session, Path.Combine(outputDirectory, $"export.zip")); glasswall.RunSession(session); glasswall.CloseSession(session); ``` Upon successful Export of the sample file, an archive named `export.zip` should have been created in your chosen output directory. Inspecting this ZIP file should reveal: - Several SISL files - 2 JPEG images - 41 KB file - 2 KB file - 2 JSON files For more detail on the contents of the export, see [Export Package Content](/embedded-engine/embedded-engine-content-export-and-import#export-package-content) Next, extract the JPEG images from the ZIP for use with the Image Resizer CLI. ## Step 2: resize images Now the embedded images from our sample PDF file have been extracted, we can resize them before reconstructing the clean file. Details on how to use the CLI can be found by running the `-h` command. There is also a reference [here](/embedded-engine/image-resizer-overview). ### Resize an image Lets resize the 41 KB JPEG file. Ensure the filename remains unchanged as the Embedded Engine will look for JPEG files with the same names from the ZIP in the next step. Below we are resizing the 41KB extracted JPEG image from the export ZIP, with a target of 20 KB (20,000 bytes). ```bash mkdir output ./gw_image_resizer -i .jpeg -o output/.jpeg -t 20000 ``` We should now have another JPEG file with the exact same name but with a size of ~20KB. ### Replace the image in the export zip Now we can replace the original 41 KB JPEG file with our resized one by dropping it into `export.zip`. ## Step 3: import Using the .zip file containing our resized images, let's import it back into a PDF file. C# ```csharp using glasswall_core2; var exportZipPath = ""; var pathToConfig = ""; var licenseFilePath = ""; var outputDirectory = ""; Glasswall glasswall = new Glasswall(); int session = 0; session = glasswall.OpenSession(); glasswall.RegisterLicenseFile(session, licenseFilePath); glasswall.RegisterPoliciesFile(session, pathToConfig, 0); glasswall.RegisterImportFile(session, exportZipPath); glasswall.RegisterOutFile(session, Path.Combine(outputDirectory, $"Sample.Clean.pdf")); glasswall.RunSession(session); glasswall.CloseSession(session); ``` Our sample file becomes `Sample.Clean.Pdf` and has been sanitised. It also includes the resized JPEG file. ## Overview When regenerating a file, content management policies allow control of various content types such as executable code, interactive form content and metadata. These file elements are known to be common attack vectors, hidden data risks and data disclosure risks when encountered within a file. A content management policy will define how the Glasswall Embedded Engine should handle applicable structures. Content management policies can be updated on a file by file basis, but must be set prior to processing a file. Where policies are not provided, the Glasswall default settings are applied, and all configurable content is **sanitised by default**. Content management policies differ across supported file types. See [Content Management](/embedded-engine/embedded-engine-content-management). ### Protect process Input files are read in and the manufacturer's specification is used to validate each byte as it is processed. This allows all the data structures (content items) within the file to be syntactically validated. If the data structures pass syntax validation, they are subjected to further semantic checks. Those data structures that have passed both syntactic and semantic validation are then written out to a new version of the input file that is regenerated on a data structure by data structure basis. If a data structure does not pass validation, an attempt is made to repair the structure in accordance with the manufacturer's specification (remediation) before it is written to the regenerated file. If this cannot be done, an issue is reported, as the file cannot be safely regenerated. Whilst the input file is being read in, the content management policies are used to determine whether content items (e.g. macros or embedded files) are permitted in the regenerated file. If these items are not permitted, they are not written to the regenerated file and the item is marked as a `SanitisationItem` in analysis reports. The result is a clean and compliant file which is **visually identical** to the original. ### Protected files Regenerated files are fully compliant with the manufacturer's specification and, depending on the content management policies applied, may have had a number of content items removed from the file. > **Note:** If all applicable content management policies were applied with a `sanitise` setting and any regenerated files were subsequently reprocessed; there should be no issue, remedy or sanitisation items present. ## Embedded Engine release versioning The Embedded Engine is released as a package of multiple libraries, wrappers and scripts. As such, defining typical semantic versioning (e.g. based on discrete API changes) can be a challenge. Currently the Embedded Engine package adheres to the following semantic-esque versioning: `..` e.g. `Release 16.1.0` (or shortened to `R16.1.0`) - `major` - A new software component is added or a known breaking change to major public APIs and/or associated functionality. A customer will need to make significant changes to integrations - `minor` - New functionality and/or collection of fixes for individual components that is configurable - e.g. additional content management policies. A customer may only need to make configuration changes to adopt new functionality (unless otherwise explicitly stated in release notes) - `patch` - A minor fix or security patch. A customer will not need to make changes to integrations in between patch releases (unless otherwise explicitly stated in release notes) import ThemedImage from '@theme/ThemedImage'; import useBaseUrl from '@docusaurus/useBaseUrl'; ## Example performance data The following tables show peak memory consumption and processing times across a range of synthetic and real world files. For typical production workloads, processing completes in less than one second and uses under 4GB RAM. ### Complexity Memory consumption depends on how complex a file is, based on the number and type of elements the engine evaluates. For example, a small but complex Excel workbook with many sheets may consume more memory than a large but simple image file. File complexity scores are included in the engine’s analysis report. In most cases, real files score no higher than 7, with a maximum value of 10. The table below shows some examples, from synthetically generated files, of how complexity scales with the number of objects. | Format | File size (MB) | Complexity | Description of file contents | |--------|----------------|------------|---------------------------------------| | DOCX | 0.038 | 5.2 | 64 lines | | DOCX | 0.043 | 5.22 | 512 lines | | DOCX | 0.094 | 5.43 | 5000 lines | | DOCX | 0.606 | 6.1 | 50000 lines | | DOCX | 2.311 | 6.66 | 200000 lines | | PPTX | 0.239 | 5.13 | 2000 external hyperlinks | | PPTX | 0.451 | 5.4 | 4000 external hyperlinks | | PPTX | 1.725 | 5.98 | 16000 external hyperlinks | | PPTX | 3.428 | 6.28 | 32000 external hyperlinks | | PPTX | 13.663 | 6.88 | 128000 external hyperlinks | | XLSX | 0.007 | 3.39 | 2 images, each 10x10 pixels | | XLSX | 0.019 | 3.76 | 32 images, each 10x10 pixels | | XLSX | 0.205 | 4.77 | 512 images, each 10x10 pixels | | XLSX | 1.600 | 5.66 | 4096 images, each 10x10 pixels | | XLSX | 12.807 | 6.56 | 32768 images, each 10x10 pixels | | PDF | 0.002 | 3.09 | 2 images, each 10x10 pixels | | PDF | 0.024 | 4.12 | 32 images, each 10x10 pixels | | PDF | 0.380 | 5.65 | 512 images, each 10x10 pixels | | PDF | 3.063 | 7.26 | 4096 images, each 10x10 pixels | | PDF | 6.149 | 7.85 | 8912 images, each 10x10 pixels | ### Real world files examples | Format | File size (MB) | Complexity | Protect - Processing time (seconds) | Protect - Peak Memory Usage (GB) | Export/Import - Processing time (seconds) | Export/Import - Peak Memory (GB) | |--------|----------------|------------|-------------------------------------|----------------------------------|-------------------------------------------|----------------------------------| | PNG | 0.402 | 0.90 | 0.028 | 0.143 | 0.048 | 0.114 | | JPEG | 0.251 | 1.28 | 0.037 | 0.091 | 0.027 | 0.098 | | JPEG | 0.112 | 1.32 | 0.032 | 0.093 | 0.017 | 0.116 | | PNG | 1.008 | 1.54 | 0.049 | 0.126 | 0.109 | 0.192 | | JPEG | 0.251 | 1.67 | 0.087 | 0.104 | 0.031 | 0.091 | | MP4 | 2.411 | 2.10 | 0.055 | 0.123 | 0.242 | 0.203 | | PNG | 0.603 | 2.43 | 0.033 | 0.141 | 0.071 | 0.135 | | MP4 | 23.705 | 2.84 | 0.367 | 0.275 | 2.248 | 0.327 | | XLSX | 0.012 | 3.66 | 0.035 | 0.147 | 0.105 | 0.190 | | MP4 | 103.222 | 4.06 | 1.616 | 0.500 | 9.910 | 0.905 | | PPTX | 0.143 | 4.29 | 0.111 | 0.197 | 0.384 | 0.161 | | PDF | 8.691 | 4.50 | 0.587 | 0.234 | 1.695 | 0.302 | | DOCX | 0.038 | 4.55 | 0.154 | 0.113 | 0.607 | 0.199 | | DOCX | 0.040 | 4.65 | 0.172 | 0.140 | 0.759 | 0.183 | | XLSX | 0.034 | 4.72 | 0.151 | 0.168 | 0.957 | 0.293 | | PPTX | 1.128 | 4.80 | 0.485 | 0.173 | 1.240 | 0.273 | | PDF | 0.221 | 5.16 | 0.160 | 0.159 | 0.664 | 0.205 | | XLSX | 0.376 | 5.87 | 1.745 | 0.317 | 14.111 | 1.186 | | PPTX | 0.643 | 5.87 | 1.144 | 0.194 | 11.635 | 0.664 | | DOCX | 0.206 | 5.92 | 2.855 | 0.305 | 12.701 | 0.936 | | PDF | 1.462 | 6.03 | 0.699 | 0.232 | 6.697 | 0.470 | ### Complex real world file examples | Format | File size (MB) | Complexity | Protect - Processing time (seconds) | Protect - Peak Memory (GB) | Export/Import - Processing time (seconds) | Export/Import - Peak Memory (GB) | |--------|----------------|------------|-------------------------------------|-----------------------------|-------------------------------------------|----------------------------------| | PDF | 1.045 | 6.01 | 0.816 | 0.171 | 3.441 | 0.306 | | PPTX | 6.462 | 6.04 | 1.999 | 0.296 | 18.243 | 1.179 | | PPTX | 0.702 | 6.05 | 1.639 | 0.425 | 19.065 | 1.418 | | XLSX | 1.120 | 6.07 | 2.352 | 0.255 | 17.993 | 0.985 | | XLSX | 0.716 | 6.17 | 3.152 | 0.388 | 26.734 | 1.715 | | PDF | 1.315 | 6.20 | 0.814 | 0.182 | 10.764 | 0.611 | | PPTX | 1.988 | 6.21 | 2.315 | 0.193 | 26.198 | 1.316 | | XLSX | 1.681 | 6.33 | 4.632 | 0.434 | 39.617 | 2.213 | | DOCX | 0.477 | 6.47 | 11.581 | 0.892 | 50.325 | 3.208 | | PDF | 1.519 | 6.51 | 0.927 | 0.194 | 19.823 | 0.998 | | XLSX | 2.031 | 6.56 | 7.713 | 0.441 | 64.021 | 3.140 | | PDF | 1.935 | 6.80 | 2.607 | 0.317 | 45.199 | 2.128 | | DOCX | 1.471 | 6.95 | 18.820 | 2.144 | 114.765 | 10.195 | | PDF | 3.553 | 7.08 | 3.795 | 0.479 | 62.265 | 2.964 | | PDF | 3.768 | 7.24 | 3.670 | 0.237 | 96.422 | 4.503 | ### Synthetic file examples | Format | File size (MB) | Complexity | Protect - Processing time (seconds) | Protect - Peak Memory Usage (GB) | Export/Import - Processing time (seconds) | Export/Import - Peak Memory (GB) | Description of file contents | |--------|----------------|------------|-------------------------------------|----------------------------------|-------------------------------------------|----------------------------------|------------------------------| | PDF | 0.001 | 2.88 | 0.014 | 0.113 | 0.020 | 0.079 | 2 external hyperlinks | | PDF | 0.002 | 3.09 | 0.014 | 0.089 | 0.024 | 0.083 | 2 images, each 10x10 pixels | | XLSX | 0.007 | 3.39 | 0.029 | 0.102 | 0.064 | 0.099 | 2 images, each 10x10 pixels | | XLSX | 0.008 | 3.63 | 0.029 | 0.114 | 0.101 | 0.112 | 2 charts | | PDF | 0.005 | 3.70 | 0.015 | 0.078 | 0.044 | 0.113 | 32 external hyperlinks | | XLSX | 0.019 | 3.76 | 0.099 | 0.108 | 0.132 | 0.141 | 32 images, each 10x10 pixels | | PDF | 0.024 | 4.12 | 0.020 | 0.090 | 0.103 | 0.115 | 32 images, each 10x10 pixels | | PPTX | 0.042 | 4.36 | 0.125 | 0.117 | 0.478 | 0.159 | 2 charts | | XLSX | 0.035 | 4.55 | 0.139 | 0.113 | 0.755 | 0.151 | 32 charts | | XLSX | 0.205 | 4.77 | 1.383 | 0.125 | 1.334 | 0.210 | 512 images, each 10x10 pixels | | PDF | 0.075 | 4.87 | 0.046 | 0.079 | 0.440 | 0.137 | 512 external hyperlinks | | PPTX | 0.255 | 5.06 | 0.810 | 0.113 | 2.559 | 0.209 | 32 charts | | PPTX | 0.239 | 5.13 | 0.697 | 0.115 | 2.251 | 0.181 | 2000 external hyperlinks | | XLSX | 0.119 | 5.14 | 0.499 | 0.148 | 2.851 | 0.289 | 128 charts | | DOCX | 0.038 | 5.20 | 0.353 | 0.121 | 2.606 | 0.257 | 64 lines | | DOCX | 0.037 | 5.20 | 0.373 | 0.127 | 2.6 07 | 0.257 | 2 images, each 1x1 pixel | | DOCX | 0.043 | 5.22 | 0.375 | 0.121 | 2.742 | 0.271 | 512 lines | | DOCX | 0.062 | 5.29 | 0.814 | 0.124 | 3.364 | 0.289 | 256 images, each 1x1 pixel | | PPTX | 0.451 | 5.40 | 1.424 | 0.126 | 4.271 | 0.231 | 4000 external hyperlinks | | DOCX | 0.094 | 5.43 | 0.643 | 0.138 | 4.290 | 0.338 | 5000 lines | | PDF | 0.380 | 5.65 | 0.267 | 0.141 | 1.518 | 0.227 | 512 images, each 10x10 pixels | | DOCX | 0.091 | 5.65 | 2.230 | 0.209 | 8.533 | 0.519 | 2048 images, each 1x1 pixel | | XLSX | 1.600 | 5.66 | 91.552 | 0.246 | 15.003 | 0.710 | 4096 images, each 10x10 pixels | | XLSX | 0.457 | 5.73 | 1.988 | 0.191 | 11.183 | 0.601 | 512 charts | | PDF | 0.604 | 5.78 | 0.281 | 0.186 | 3.386 | 0.303 | 4096 external hyperlinks | | PPTX | 1.725 | 5.98 | 8.293 | 0.153 | 18.819 | 0.573 | 16000 external hyperlinks | | DOCX | 0.606 | 6.10 | 3.423 | 0.394 | 19.162 | 1.271 | 50000 lines | | DOCX | 0.198 | 6.20 | 10.333 | 0.462 | 30.985 | 1.636 | 10000 images, each 1x1 pixel | | PPTX | 3.656 | 6.21 | 12.352 | 0.153 | 36.349 | 1.146 | 512 charts | | PPTX | 3.428 | 6.28 | 33.142 | 0.191 | 54.816 | 0.959 | 32000 external hyperlinks | | XLSX | 12.807 | 6.56 | 1041.659 | 1.027 | 508.494 | 4.637 | 32768 images, each 10x10 pixels | | XLSX | 3.626 | 6.63 | 20.287 | 0.592 | 92.785 | 4.079 | 4096 charts | | DOCX | 2.311 | 6.66 | 12.707 | 1.136 | 69.830 | 5.043 | 200000 lines | | PDF | 4.920 | 6.69 | 2.215 | 0.391 | 27.451 | 1.729 | 32768 external hyperlinks | | DOCX | 0.600 | 6.76 | 31.337 | 1.492 | 118.010 | 6.194 | 40000 images, each 1x1 pixel | | PPTX | 13.663 | 6.88 | 560.821 | 0.460 | 609.425 | 3.823 | 128000 external hyperlinks | | PPTX | 29.088 | 7.10 | 139.941 | 0.466 | 330.423 | 9.145 | 4096 charts | | PDF | 3.063 | 7.26 | 48.977 | 0.215 | 58.920 | 0.810 | 4096 images, each 10x10 pixels | | PDF | 6.149 | 7.85 | 363.602 | 0.298 | 383.711 | 1.550 | 8192 images, each 10x10 pixels | ### Synthetic file performance graphs ## Summary functionality The Java wrapper has additional functionality that can generate a file summary from a given analysis report. The file summary provides an easy to see overview of the work that was carried out on the file. It contains the following items: - The filename of the file that was processed - The process message from the [`GW2FileSessionStatus`](/embedded-engine/embedded-engine-gw2filesessionstatus) API - The last error message from [`GW2FileErrorMsg`](/embedded-engine/embedded-engine-gw2fileerrormsg) API, if the file was non-conforming - Additional information that contains a list of sanitised or allowed items with their respective counts. For example, 5 instances of Metadata that have been sanitised. The file summary has a flag for skipping unsupported file types. When this is set to true then unsupported file types will have the `EngineOutcome` set to `Unsupported` with the `LastErrorMessage` set to `Skipping unsupported filetype`. The file summary can be used in the following ways: - A summary XML report can be generated from a list of analysis reports. This is documented on the [Report Summary](/embedded-engine/embedded-engine-java-summary-report) page. - The file summary can be used independently as a way to summarise the analysis report without needing to fully understand the analysis report. This is documented on the [Standalone Summary](/embedded-engine/embedded-engine-java-summary-report-standalone) page. ## Summary report ### Summary report example ```xml External Hyperlinks Metadata Embedded File /home/glasswall/Documents/Example.doc Remedies Applied Managed Metadata /home/glasswall/Documents/Example2.docx Issues Found Non-conforming End of stream 'xl/workbook.xml' not reached /home/glasswall/Documents/UnsupportedFileType.cat Unsupported Skipping unsupported filetype /home/glasswall/Documents/UnsupportedFileType.cat Issues Found Non-conforming Unable to determine file type ``` ### API #### Constructors ```java SummaryReport() throws ParserConfigurationException, DOMException SummaryReport(boolean skipUnsupportedFileTypes) throws ParserConfigurationException, DOMException ``` **Parameters**: - `boolean skipUnsupportedFileTypes` - This sets the skipUnsupportedFileTypes flag. If this flag is not specified then the skip unsupported file types will be set to `false`. #### Instance methods ```java void setSkipUnsupportedFileTypes(boolean skipUnsupportedFileTypes) ``` Setter for the skip unsupported file types flag. **Parameters**: - `boolean skipUnsupportedFileTypes` - This sets the skip unsupported file types flag. --- ```java boolean getSkipUnsupportedFileTypes() ``` Getter for the skip unsupported file types flag. **Returns**: `true` if unsupported file types should be skipped, and `false` otherwise. --- ```java void addAnalysisReport( Path inputFilePath, int sessionStatus, String lastErrorMessage, String processMessage, InputStream analysisInputStream ) throws SAXException, ParserConfigurationException, IOException ``` Add a new Glasswall analysis report to the summary report. **Parameters**: - `Path inputFilePath` - The file path of the file that was processed. - `int sessionStatus` - The return status from [GW2RunSession](/embedded-engine/embedded-engine-gw2runsession). - `String lastErrorMessage` - The error message from [GW2FileErrorMsg]/embedded-engine/embedded-engine-gw2fileerrormsg). If this is null or empty then it won't be included in the summary report. - `String processMessage` - The process message from [GW2FileSessionStatus](/embedded-engine/embedded-engine-gw2filesessionstatus). If this null or empty then it won't be included in the summary report. - `InputStream analysisInputStream` - Input stream containing the analysis report to be included. --- ```java void addAnalysisReport(FileSummary summary) throws SAXException, ParserConfigurationException, IOException ``` Add a new Glasswall file summary to the summary report. **Parameters**: - `FileSummary summary` - The summary information to be added to the summary report. --- ```java public void generateSummaryReport(Writer writer) throws TransformerConfigurationException, TransformerException ``` Generate a XML summary report from the provided analysis reports. **Parameters**: - `Writer writer` - A writer where the summary report will be written to. --- ```java public void generateSummaryReport(Writer writer, Transformer transformer) throws TransformerException ``` Generate a XML summary report from the provided analysis reports using the specified transformer. **Parameters**: - `Writer writer` - A writer where the summary report will be written to. - `Transformer transformer` - The transformer that will be used for generating the XML report. ### API example The summary report functionality is implemented in `com.glasswall.analysissummary.SummaryReport` class. First an instance of the class is created, `addAnalysisReport` is then called for each analysis report that will be included in the summary report, and then `generateSummaryReport` is called at the end to generate the summary report. An example can be seen below: ```java File input_directory = new File("Input"); File output_directory = new File("Output"); output_directory.mkdirs(); SummaryReport summary_report = new SummaryReport(); summary_report.setSkipUnsupportedFileTypes(true); // Optionally set the skip unsupported file types flag for (File file : input_directory.listFiles()) { if (file.isDirectory()) continue; try (Core2JavaBridge gw = new Core2JavaBridge()) { // Create the output path for file and analysis report String file_output_path = Paths.get(output_directory.getAbsolutePath().toString(), file.getName()).toString(); String analysis_output_path = file_output_path + ".xml"; // Run the file through the Glasswall engine int session = gw.GW2OpenSession(); gw.GW2RegisterInputFile(session, file.getAbsolutePath()); gw.GW2RegisterAnalysisFile(session, analysis_output_path, 0); gw.GW2RegisterOutFile(session, file_output_path); int run_status = gw.GW2RunSession(session); // Retrieve the error message if the file is non-conforming String error_message = null; if (run_status < 0) error_message = gw.GW2FileErrorMsgString(session); // Retrieve the session status along with the session description FileSessionStatus session_status = gw.GW2FileSessionStatusResult(session); // Add the analysis report to the summary report try (FileInputStream stream = new FileInputStream(analysis_output_path)) { summary_report.addAnalysisReport(file.toPath(), run_status, error_message, session_status.summaryDescription, stream); } } catch (Exception ex) { System.err.println("Exception occurred: " + ex.getMessage()); } } // Write the summary report to `SummaryReport.xml` file try (BufferedWriter writer = Files.newBufferedWriter( Paths.get(output_directory.getAbsolutePath().toString(), "SummaryReport.xml"), StandardCharsets.UTF_8, StandardOpenOption.TRUNCATE_EXISTING, StandardOpenOption.CREATE, StandardOpenOption.WRITE)) { summary_report.generateSummaryReport(writer); } ``` ## Summary - standalone The standalone file summary can be generated by creating a `com.glasswall.analysissummary.FileSummary` object. ### API #### Constructors ```java public FileSummary( Path inputFilePath, int sessionStatus, String lastErrorMessage, String processMessage, InputStream analysisInputStream, boolean skipUnsupportedFileTypes ) throws SAXException, ParserConfigurationException, IOException ``` **Parameters**: - `Path inputFilePath` - The file path of the file that was processed. - `int sessionStatus` - The return status from [GW2RunSession](/embedded-engine/embedded-engine-gw2runsession). - `String lastErrorMessage` - The error message from [GW2FileErrorMsg](/embedded-engine/embedded-engine-gw2fileerrormsg). If this is null or empty then it won't be included in the summary. - `String processMessage` - The process message from [GW2FileSessionStatus](/embedded-engine/embedded-engine-gw2filesessionstatus). If this null or empty then it won't be included in the summary. - `InputStream analysisInputStream` - Input stream containing the analysis report to be included. - `boolean skipUnsupportedFileTypes` - boolean indicating whether unsupported file types should be skipped. #### Instance methods ```java public String getFileName() ``` Returns the filename of the processed file. **Returns**: The absolute file path of the processed file. --- ```java public String getProcessMessage() ``` Returns the file session process message. This will be null if the process message was not specified. **Returns**: File session process message. --- ```java public EngineOutcome getEngineOutcome() ``` Returns an `EngineOutcome` enum indicating whether the processed file was `Managed`, `NonConforming`, or `Unsupported`. **Returns**: An `EngineOutcome` enum. --- ```java public String getErrorMessage() ``` Returns the error message if the file is non-conforming. This will be null if the file is managed. **Returns**: The error message. --- ```java public Map getSanitisedItems() ``` Returns a map of the sanitised items along with their count. **Returns**: A map of sanitised items with their counts. --- ```java public Map getAllowedItems() ``` Returns a map of the allowed items along with their count. **Returns**: A map of allowed items with their counts. --- ```java public Map getRemedyItems() ``` Returns a map of the remedy items along with their count. **Returns**: A map of remedy items with their counts. ### API example ```java File input_directory = new File("Input"); File output_directory = new File("Output"); output_directory.mkdirs(); for (File file : input_directory.listFiles()) { if (file.isDirectory()) continue; try (Core2JavaBridge gw = new Core2JavaBridge()) { // Create the output path for file and analysis report String file_output_path = Paths.get(output_directory.getAbsolutePath().toString(), file.getName()).toString(); String analysis_output_path = file_output_path + ".xml"; // Run the file through the Glasswall engine int session = gw.GW2OpenSession(); gw.GW2RegisterInputFile(session, file.getAbsolutePath()); gw.GW2RegisterAnalysisFile(session, analysis_output_path, 0); gw.GW2RegisterOutFile(session, file_output_path); int run_status = gw.GW2RunSession(session); // Retrieve the error message if the file is non-conforming String error_message = null; if (run_status < 0) error_message = gw.GW2FileErrorMsgString(session); // Retrieve the session status along with the session description FileSessionStatus session_status = gw.GW2FileSessionStatusResult(session); // Create a file summary try (FileInputStream stream = new FileInputStream(analysis_output_path)) { FileSummary file_summary = new FileSummary( file.toPath(), run_status, error_message, session_status.summaryDescription, stream, true // true to skip unsupported file types ); // TODO - do something with the file summary } } catch (Exception ex) { System.err.println("Exception occurred: " + ex.getMessage()); } } ``` ## Analysis process The Glasswall library receives a file through a published API and passes it through a number of process cycles. The output from each cycle becomes the input for the next, hence maintaining a level of separation between processes. Analysis of the file occurs in each of the cycles. Early cycles elicit the structure of the file and the sizes of its constituent parts. The later cycles are concerned with conducting syntactic and semantic checks, which identify possible sources of risk, out-of-range fields or malformed structures. Where elements of the file are compressed, these are expanded and the results assessed, analysed and verified. This enables the analysis report published at the end of the process to give a thorough assessment of the contents and structure of the file. By stepping through the sanitisation and remediation processes, Glasswall is able to provide an accurate report of the actions that could be carried out by Glasswall's regeneration functionality, thus making the file conformant with the specification. During each cycle, the file being processed is transformed into Glasswall's own internal representation. This simplifies the parsing and traversing processes and helps provide isolation. As the analysis process navigates through the Glasswall structures, the detailed checks are not only made on individual components but also at a higher level on the relationships between file components. These higher level checks enable the semantic structure and consistency of the file to be properly verified. ## Analysis reports Embedded Engine analysis reports provide a detailed, file-type agnostic description of data and is logged in an XML format. The structure of this report follows an Analysis Report XSD, which is designed to simplify parsing and processing. The analysis report XSD is located in the `/schemas/sdk_editor` folder of the Embedded Engine release package. The analysis report contains the following file information: - **[Document Summary](#document-summary)** — high-level file specific information. - **[Content Management Policy](#content-management)** — the settings of content management switches that have been applied to the processed file. - **[Content Groups](#content-groups-and-items)** — the main grouping of content detected in the processed file. - **[Content Items](#content-groups-and-items)** — the low-level structures detected in the processed file. - **[Issue Items](#issue-items)** — the detected structures that do not match the manufacturer's specification or where detected structures are disallowed by policy (in both cases rejecting the file). - **[Sanitisation Items](#sanitisation-items)** — the detected structures under content management that are marked for removal by policy. - **[Allowed Items](#allowed-items)** — the detected structures under content management that are permitted by policy. - **[Remedy Items](#remedy-items)** — automatic corrections applied to the processed file in order to bring the file in line with the manufacturer's specification. ### Sample analysis reports This section contains an abridged Glasswall Analysis Report containing all the principle elements of a typical report. This particular example is based on a PDF file. #### Document summary Each XML report starts with a document summary shown below: ```xml 2293803 pdf PDF-1.6 signature match DEE7CEB7AB57227121FF65F0D8D0878CBEDF90864069D9525698257972498099 83EE59FAB5972429CE65CBB22EBF8B592D53F47927E0FA751584875D4F80CA1E 101 4.41 ``` * TotalSizeInBytes - *Size of input file* * FileType - *Glasswall detected file type* * Version - *File format version (when determinable)* * FileTypeDetectionMethod - *Glasswall detection method employed to determine the file format* * InputSHA256 - *SHA-256 of the original input file* * OutputSHA256 - *SHA-256 of the output file after processing by Glasswall (should the file be regenerated)* * ProcessingTimeMilliseconds - *Glasswall measured time taken to process the file* * ComplexityLevel - *Glasswall estimate of the level of complexity of the file, produced by calculating the log10 to 2 decimal places of the count of artefacts identified in the file* #### Content management The content management policies that were used on the file are then listed. **Note:** All policies for all file types are listed in each report. Where policy has not been configured but is available, the default policy settings that were applied will also be listed. Some of the PDF content management switch settings are shown below: ```xml javascript sanitise acroform sanitise embedded_files sanitise ``` #### Content groups and items The number of different content groups found in the file (16) along with an example of a content item from the first group is shown below: ```xml PDF document has Basic File Section structure instances PDF Header Instances 1 15 15 15 15 > ... ``` #### Sanitisation items In this example, as the **metadata** switch has been set to **sanitise**, a dictionary structure is shown as tagged for removal. ```xml Document information dictionary detected in a document trailer dictionary. 16872998749 1 0 ``` #### Allowed items In this example, as the **metadata** switch has been set to **allow**, a dictionary structure is therefore retained. ```xml Document information dictionary detected in a document trailer dictionary. 16872998749 1 0 ``` #### Remedy Items Not all XML reports include Remedies, as these are automatic corrections made to bring any regenerated file in line with the file specification. In this example, a remedy item has been reported in the file. ```xml PDF Stream is missing an End-Of-Line before the 'EndStream' marker. 1605893787 7 ``` #### Issue Items Very few files have an issue, as this means they are not just non-conformant with the file specification, but Glasswall has been unable to Remedy the issue back to the standards set in the specification. A file with an issue item **cannot be regenerated**. ```xml /Info dictionary contained an unexpected key (/GTS_PDFXConformance). 1670998746 1 ``` Issue items are also reported when a file has been determined to be **non-conformant due to content management policy** (where a setting of `disallow` has been used). Each Sanitisation item, Allowed Item, Remedy item or Issue item has a *unique numeric ID* associated with it, so the item can be uniquely identified by other applications that may wish to process the XML reports. Glasswall provides the ability to export and import content items for supported file types. This allows internal components of processed files to be made available to external processes and applications for additional processing outside of the Glasswall Embedded Engine domain. Once exported, these components can be validated externally before the Glasswall Engine imports the components and recomposes the files. To enable the user to carry out additional analysis on components within files, the files must be processed by the Glasswall Embedded Engine twice; once to extract a package containing the components that make up a file (export), and a second pass to reintegrate the externally analysed and/or modified components back into the file (import). **Files are also automatically protected through the Embedded Engine CDR process throughout both the export and import cycles.** ## Example use cases Example use cases for Export-Import processing include but are no limited to: - **Pattern For Safely Importing Data** - Glasswall exposes the internal file structure in a standard form such as XML, enabling third parties to carry out hardware verification as part of the [pattern for safely importing data](https://www.ncsc.gov.uk/guidance/pattern-safely-importing-data) - **Data Loss Prevention** - Exported content such as text is annotated to allow for all text to be identified, enabling users to carry out DLP processes such as text search and redaction. - **Image analysis** - Additional image processing to detect and/or prevent steganography attacks. ## Exportable content Glasswall provides the ability to export a document object model (DOM), which includes all content, for all supported file formats. The exported DOM is presented in one of two intermediate formats, XML or SISL. Users have the option to extract embedded images in their original form or export them as a DOM representation. ## Importable content Glasswall provides the ability to re-import a document object model (DOM) for all supported file formats, regardless of external modification made to the exported content (provided that modifications comply with the file format specification). ## Export package content The Export package is a ZIP archive containing XML or SISL file streams, embedded images and corresponding JSON files with metadata (for PDF images). Internal names of XML tags/attributes and SISL types/parameters are shortened to minimize the size of exported file streams. ### XML tags and SISL types: | **Tag / Type (Shortened)** | **Tag / Type (Full)** | **Description** | | :------------------------: | :--------------------:| ----------------------------------------------------------------------------------------- | | "S" | STRUCT | Represents a structure node from our tree. | | "SA" | STRUCTARRAY | The array of STRUCT objects. | | "I" | ITEM  | A property within a STRUCT object (e.g., whitespace indicators, end-of-file markers etc.) | | "V" | VALUE  | Represents the stored integer value that was read from a file. | | "VA" | VALUEARRAY | Represents a data block read from a file. |         ### XML attributes and SISL parameters | **Attribute / Parameter (Shortened)** | **Attribute / Parameter (Full)** | **Data Type** | **Description** | | :-----------------------------------: | :-------------------------------: | :---------------------------------: | :------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | "o" | offset | \  | The attribute contains the offset of the current item in the buffer. The buffer can represent things such as the file, a file within an archive, an amalgamation of streams from a CFB object, and more. | | "s" | size | \  | The total length of the current structure in bytes. | | "i" | itemEnum | \ | The internal numerical representation of the current ITEM. | | "n" | name  | \ | The internal name of the current structure. | | "t" | isText | \ \[ "true \| "false" \] | Indicates if the element contains text or not. Only applicable to items which are marked as text within internal schemas. |    | "se" | structEnum | \ | The internal numerical representation of the current STRUCT.       | | "sn" | streamName | \ | The current stream name. | | "c" | cameraName | \ | The current camera (parser/validator/writer) name. | | "st" | isStructuralText | \ \[ "true \| "false" \] | The attribute to distinguish between structural information and the file's visible text content. Only applicable to items which are marked as text within internal schemas. | | "e" | encoding | \ \[ "UTF 8 \| "Base64"\] | The attribute specifies the encoding of data within the current element. Only applicable to items which are marked as text within internal schemas. | ### SISL specific parameters | **Parameter (Shortened)** | **Parameter (Full)** | **Data Type** | **Description** | | :-----------------------------------: | :-------------------------------: | :---------------------------------: | :------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | "__s" | struct | \ | General SISL structure of type: \[ S \| SA \| I \| V \| VA \] | | "__m" | meta | \ | Dictionary of the current SISL structure parameters  | | "__d" | data | \   | The stored data of ITEM, VALUE or VALUEARRAY | | "__l" | length | \   | The original size of data stored in __d before non-printable characters were escaped. | ## Export text dump (alpha) The Export Text Dump feature introduces the option to produce a file containing all the text within the input file being exported. The file is produced and stored in the same directory as the output ZIP file. Text dumps can be exported alongside or without content export zips. | File Format | Supported | | -------------- | --------- | | Office 2003 | ✓ | | Office 1997 | ✓ | | PDF | ✓ * | | Binary formats | ✗ | | Audio formats | ✗ | | Image formats | ✗ | | MPEG formats | ✗ | > **Note:** Text Dump for PDF is currently unavailable when the sysConfig switch `export_embedded_images` set to `true`. It is also only limited to a subset of PDF text encodings. # Glasswall exported image metadata When operated in export mode, Glasswall outputs images embedded in PDF files in the form of a pair of files: - A file containing the raw image data - A file containing metadata describing the format of the image The metadata file takes the form of a JSON object containing the following Name/Value attribute pairs. ## Mandatory attributes These values will be present for every image regardless of its format, usage, or encoding and will always be present regardless of the encoding scheme or colour space in use. ### Width An unsigned integer indicating the images width in pixels. ### Height An unsigned integer indicating the images height in pixels. ### Components An unsigned integer indicating the number of intensity values used to represent each pixel, for example a simple monochrome image would have a Components value of 1, an RGB encoded image would have a component value of 3, and a CMYK encoded image would have a component value of 4. ### BitsPerComponent An unsigned integer indicating the number of bits per colour component, for example a simple monochrome image would have a BitsPerComponent value of 1, a greyscale image encoded with 256 intensity levels would have a BitsPerComponent value of 8 as would an RGB image encoded with 24 bit colour resolution. ### Length An unsigned integer indicating the number of bytes of data expected in the image data stream. ## Optional attributes In addition, the metadata file may also contain the following optional Name/Value attribute pairs where additional data is required to describe the usage of and/or decode to accurately render the image. ### ImageMask A Boolean value indicating that the image is intended to be used as a mask to reveal or hide parts of another underlying image. ### Filter A string value indicating the encoding algorithm used. Available encoding algorithms are described in the discussion of stream filters in the PDF Specification (ISO32000-1, Section 7.4) and common values include CCITTFaxDecode, LZWDecode (Lempel-Zif-Welch compression, as used in for example the GIF format), DCTDecode (discrete cosine transformation compression, as used in JPEG), or FlateDecode (zlib deflate compression). ### DecodeParms A string value containing any additional parameters required by the decode algorithm to correctly decode the image data. ### JBIG2 Global data segments to be used when processing JBIG2 encoded image data.type of colourspace ### Matte A boolean value indicating that the image is intended for use as a matte to form part of a composite image e.g. by colour keying or alpha blending. ## Colour space attributes ### ColorSpaceFamily A string value indicating the type of colour space to be used if the image is to be rendered as intended as described in the PDF specification (ISO32000-1, Section 8.6). #### Device colour spaces. Simple colour spaces (DeviceGray, DeviceRGB, and DeviceCMYK) require no further description and indicate that image sample (pixel) values can be used directly. #### CIE-based colour spaces CIE-based colour spaces shall be based on an international standard for colour specification created by the Commission Internationale de l’Éclairage (International Commission on Illumination). These spaces specify colours in a way that is independent of the characteristics of any particular output device. Colour space families in this category include CalGray, CalRGB, Lab, and ICCBased. Currently only ICC based colour spaces (ColorSpaceFamily ICCBased) are supported, and, in this case the following attributes will be present. ##### ColorSpaceIccProfileData A string value containing the ICC cross platform colour profile in ICC standard format. Because this is an arbitrary byte sequence, it is presented in Base64 encoding in order to restrict it to be represented within a JSON string value. ##### ColorSpaceAlternate A string value indicating an alternative colorspace which may be used to render the image if ICC based output is not supported. #### Indexed colour spaces Indexed colour spaces (ColorSpaceFamily value of Indexed) represent each pixel (or sample) value using a single value which is as an index into a table containing the actual output value, or set of values, the number of values in the lookup table will correspond to a base color space. For example, using a base colour space of DeviceRGB, each entry in the lookup table will contain 3 values. Lookup table entries will be presented in a single array with colour components occurring sequentially. For an index colour space the following attributes will be present. ##### ColorSpaceBaseFamily A string value indicating the colour space used for indexed lookup values. ##### ColorSpaceBaseComponents An unsigned integer indicating the number of components in the base colour space and hence the number of values in each entry in the indexed colour space lookup table. ##### ColorSpaceIccProfileData ICC based colour space profile data for an ICC based base colour space - only present if ColorSpaceBaseFamily has the value ICCBased. ##### ColorSpaceLookupData An array of unsigned integers containing the lookup table for the indexed colour space. The Glasswall Embedded Engine provides a pattern matching capability in the following file formats: - Microsoft Binary Office - Office Open XML - ASCII and UTF-8 plain text *(when `enable_text_support` is specified `true` under `sysConfig`)* The search strings are configured via a policy file, where they can be specified as either a `text` item or a `regex` item: - **Text** - Match only distinct words or numbers. Words and numbers are considered distinct if the character immediately preceding or succeeding the match is not a letter or digit respectively, meaning `or` will not produce a match when found in "ore", "word" or "door". - **Regex** - Match anywhere the regular expression pattern is found. This includes matches within distinct words or numbers, e.g. a regular expression of `r[aeiou]+` will match the "re" in "**re**gular", "exp**re**ssion" and "anywhe**re**". - Word Search does not support regular expression assertions. Regular expression containing `^` or `$` will return matches found anywhere in the file and regular expressions containing a lookaround will not return any matches. For every pattern matched, the following actions (textSetting) can be taken: - **Allow** - Produce an XML analysis report specifying the number of matching strings within the file and their location - **Disallow** - Report all matches and do not regenerate the input file if any are found - **Redact** - Report matches and regenerate the input file with all instances replaced with a character specified in the policy file with `replacementChar`. *This action is only available for Microsoft Binary Office and Office Open XML files.* - **Require** - Report all matches and do not regenerate the input file unless at least one match is found. *This action is only available for plain text files, and at least one must be specified.* The APIs for Word Search support **string**, **character based**, and **regular expression** matching. See [Word Search Library](/embedded-engine/embedded-engine-word-search-library) for Word Search API documentation. Automatic corrections back to the file specification are performed upon file regeneration. The purpose of this is to enable the Glasswall Embedded Engine to remove threats that are hidden within the file structure, as well preventing the possibility of activating exploits via the misuse of structural components in the file. In Analysis Mode, all automatic corrections made to a file are reported as **Remedy items**. ### Remedy application There are various types of remedies that are performed across all supported file types. - **Unrecognised objects** that are hidden within the file structure and are not defined in the official specification are removed automatically. - Where possible, **components of a file that deviate from the manufacturers specification** are automatically corrected back to the standards set in the specification, otherwise an **issue item** is reported. - For file **formats that do not have content management policy options**, Glasswall will automatically remove content deemed risky as a remedy. Glasswall provides the ability to quickly determine the file type for a given file **without fully processing the file.** The relevant API functions for this can be found [here](/embedded-engine/embedded-engine-api-functions-non-session). ### Identification of supported filetypes For file formats supported by the Glasswall Embedded Engine, the engine can attempt to determine the file type regardless of the presence of a file extension or where the file extension has been modified. One or more of the following methods are used to provide a reliable best guess of the file type: - Parsing parts of the file structure. - Magic number and header signature checks. - Class ID (GUID) lookups. This approach can be combined with alternative filetype detection methods (e.g., the `GW2GetFileTypeID` API call which identifies the filetype from the extension) to detect potential cases of file masquerading. ### Character encoding detection The Glasswall Embedded Engine can also detect the encoding of different text file formats not normally supported by the Glasswall engine. This is performed after the input file has been checked against "best guess" identification for all supported filetypes and will discern whether the input file uses ASCII or UTF-8 encoding. File Type ID Enumeration The following table details the possible enumerator that can be returned from the Embedded Engine file determination APIs. | Value | Enumerator | |-|-| | 0 | ft_unknown | | 1 | ft_fileIssues | | 2 | ft_bufferIssues | | 3 | ft_internalIssues | | 4 | ft_licenseExpired | | 5 | Password protected OPC file | | 6 | ft_nullPointerArgument | | 7 | ft_unsupportedFileType | | 8 | ft_fileTooLarge | | 16 | ft_pdf | | 17 | ft_doc | | 18 | ft_docx | | 19 | ft_ppt | | 20 | ft_pptx | | 21 | ft_xls | | 22 | ft_xlsx | | 23 | ft_png | | 24 | ft_jpg | | 25 | ft_gif | | 26 | ft_emf | | 27 | ft_wmf | | 28 | ft_rtf | | 29 | ft_bmp | | 30 | ft_tiff | | 31 | ft_pe | | 32 | ft_macho | | 33 | ft_elf | | 34 | ft_mp4 | | 35 | ft_mp3 | | 37 | ft_wav | | 38 | ft_mpg | | 39 | ft_coff | | 42 | ft_svg | | 43 | ft_webp | | 44 | ft_machouniversal | | 45 | ft_utf8 | | 46 | ft_ascii | | 256 | ft_zip | | 257 | ft_gzip | | 258 | ft_bzip2 | | 259 | ft_7zip | | 260 | ft_rar | | 261 | ft_tar | | 262 | ft_xz | # File type specifications ## PDF **Specification Validated Against:** - PDF Reference (third edition) Adobe Portable Document Format Version 1.7. **Note:** Glasswall supports PDF Reference (third edition) Adobe Portable Document Format from Version 1.3. ## Images There are no content management switches for any of the image file formats. Images embedded in a document that do not conform to the specification will result in the original document being quarantined. ### JPEG **Specification Validated Against:** - JPEG File Interchange Format, Version 1.02 September 1, 1992 - JP2 for ISO/IEC 15444-1 (JPEG 2000) - JPX for ISO/IEC 15444-2 (JPEG 2000) ### GIF **Specification Validated Against:** - Graphics Interchange Format, Version GIF 87a - Graphics Interchange Format, Version GIF 89a ### PNG **Specification Validated Against:** - Portable Network Graphics (PNG): Functional specification. ISO/IEC 15948:2003 (E) ### EMF **Specification Validated Against:** - [MS-EMF]: Enhanced Metafile Format: Open Specifications Documentation. v20151016 - [MS-EMFPLUS]: Enhanced Metafile Format Plus Extensions: Open Specifications Documentation. V20121017 ### WMF **Specification Validated Against:** - [MS-WMF]: Windows Metafile Format: Open Specifications Documentation. v20121017 ### TIFF and GeoTIFF **Specification Validated Against:** - TIFF 6.0 Specification: Final – June 3, 1992 - GeoTIFF Format Specification - Specification Version: 1.8.1 ### BMP **Specification Validated Against:** - BMP is covered by the WMF file specification. ## Binary Office ### Processing specific to Binary Office in Protect mode - The content management switch `GWFILE_MANAGE_REMOVE_MACROS` when set to **sanitise** in Protect mode removes the VBA script from the regenerated file but copies over the macro container. This can give the impression that the macros have not been deleted. - The content management switch `GWFILE_MANAGE_REMOVE_EMBEDDED_FILES` when set to **sanitise** in Protect mode follows these rules: 1. If the embedded file is supported by Glasswall, it will be processed as a standalone file, including its own configuration management policies. If it can be regenerated, it will be re-embedded; otherwise, the parent file will be marked as non-conforming. 2. If the embedded file is **not** supported (e.g., .mdb or .dat files), it will be removed. - Embedded files are validated to five levels deep. ## DOC **Specifications Validated Against:** - [MS-DOC]: Word (.doc) Binary File Format - Microsoft Corporation, 2011, v20121003 to [MS-DOC]: Word (.doc) Binary File Format - Corporation, 2017, v20170112 - Office Common Data Types and Objects Structure Specification - Microsoft Corporation, 2011, v20110608 - Office Document Cryptography Structure Specification - Microsoft Corporation, 2011, v20110608 - Office VBA File Format Structure Specification - Microsoft Corporation, 2011, v20110608 **Non-Supported Specifications:** - Word 95 and earlier versions (Word 1 to Word 6) are not supported. Processing these file types will result in an issue being reported in the XML report in **Analysis mode** and the file **not being regenerated** in **Protect mode**. - Word files containing Apple Mac or LibreOffice-specific file features will be marked as non-conformant. ## XLS **Specifications Validated Against:** - [MS-XLS]: Excel Binary File Format (.xls) Structure Specification - Microsoft Corporation, v20141018 to v20161017. - Spreadsheet Data Model File Format - Office Graph Binary File Format v20141019 **Non-Supported Specifications:** - Excel 95 and earlier versions are not supported. Processing these file types will result in an issue being reported in the XML report in **Analysis mode** and the file **not being regenerated** in **Protect mode**. - Excel files containing Apple Mac or LibreOffice-specific file features will be marked as non-conformant. ## PPT **Specifications Validated Against:** - [MS-PPT]: PowerPoint (.ppt) Binary File Format - Microsoft Corporation, v20121003 to v20160922. **Non-Supported Specifications:** - PowerPoint 95 and earlier versions are not supported. Processing these file types will result in an issue being reported in the XML report in **Analysis mode** and the file **not being regenerated** in **Protect mode**. - PowerPoint files containing Apple Mac or LibreOffice-specific file features will be marked as non-conformant. ## XML Office ### Processing specific to XML Office in Protect mode - If an XML Office file has another XML Office file embedded inside it and the `GWFILE_MANAGE_REMOVE_EMBEDDED_FILES` is set to **sanitise**, the embedded file will be **removed** when a new file is regenerated. - Office XML files are stored as **zipped archives**. If the zipped file is less than 10MB, when extracted, an individual stream exceeding 10MB: - **32-bit DLL:** The file is marked as **non-conforming** and will not be processed further. - **64-bit DLL:** Streams up to **50MB** are processed. ## Office 2010 **Specifications Validated Against:** - [ECMA-376 5th edition Part 1](https://ecma-international.org/wp-content/uploads/ECMA-376-1_5th_edition_december_2016.zip) - [ECMA-376 5th edition Part 2](https://ecma-international.org/wp-content/uploads/ECMA-376-2_5th_edition_december_2021.zip) - [ECMA-376 5th edition Part 3](https://ecma-international.org/wp-content/uploads/ECMA-376-3_5th_edition_december_2015.zip) - [ECMA-376 5th edition Part 4](https://ecma-international.org/wp-content/uploads/ECMA-376-4_5th_edition_december_2016.zip) ## Office 2013 **Specifications Validated Against:** - Word (.docx) Extensions to the Office Open XML SpreadsheetML File Format - Excel (.xlsx) Extensions to the Office Open XML SpreadsheetML File Format - PowerPoint (.pptx) Extensions to the Office Open XML SpreadsheetML File Format - Office Drawing Extensions to the Office Open XML Structure **Non-Supported Specifications:** - Office 2010 or Office 2013 files containing Apple Mac or LibreOffice-specific file features may be marked as **non-conformant** or the components may be **removed** from the file. ## Other ### WAV **Specifications Validated Against:** - Multimedia Programming Interface and Data Specifications 1.0 August 1991. ### MPG **Specifications Validated Against:** - ISO/IEC 13818-1:2000(E). Second edition 2000-12-01 ### MP3 **Specifications Validated Against:** - MPEG Layer III audio encoding is defined in two ISO/IEC specification families: - **MPEG-1:** ISO/IEC 11172-3 - **MPEG-2:** ISO/IEC 13818-3 ### MP4 **Specifications Validated Against:** - ISO/IEC 14496-12:2005(E). Part 12: Corrected version 2005-10-01. ### PE and COFF **Specifications Validated Against:** - Microsoft Portable Executable and Common Object File Format Specification Revision 6.0 – February 1999. ### ELF **Specifications Validated Against:** - Tool Interface Standard (TIS) Executable and Linking Format (ELF) Specification Version 1.2. May 1995 - ELF-64 Object File Format Version 1.2 Draft 2. May 27, 1998. ### Mach-O **Specifications Validated Against:** - OS X ABI Mach-O File Format Reference. 2009-02-04. ## Embedded engine features overview The table below lists the Embedded Engine features and SDK components that are currently available across the Glasswall supported platforms and architectures. | Feature / Component | x86-64 - Debian | x86-64 - Windows | x86-64 - Stop OS | ARM64 - Debian | | ------------------------------ | :-------------: | :--------------: | :--------------------: | :--------------------: | | Protect / Analysis | ✔ | ✔ | ✔ | ✔ | | File Identification | ✔ | ✔ | ✔ | ✔ | | Automatic File Corrections | ✔ | ✔ | ✔ | ✔ | | Configuration Management | ✔ | ✔ | ✔ | ✔ | | Content Export / Import | ✔ | ✔ | ✘ | ✔ | | Glasswall Conform | ✔ | ✔ | ✘ | ✘ | | Image Resizer | ✔ | ✔ | ✘ | ✘ | | .NET Language Wrapper | ✔ | ✔ | ✘ | ✔ | | Python Language Wrapper | ✔ | ✔ | ✘ | ✔ | | Java Language Wrapper | ✔ | ✔ | ✘ | ✔ | | Word Search & Redaction (beta) | ✔ | ✔ | ✘ | ✘ | ## What is Glasswall Foresight? Glasswall Foresight, now available in Glasswall Meteor, is an AI feature that combines threat prediction with CDR (Content Disarm and Reconstruction) analysis to enhance endpoint file security. It uses machine learning models alongside CDR-based structural analysis to assess files and generate a probabilistic threat label indicating the likelihood of malicious behavior, including unknown or zero-day exploits. This label is applied with existing CDR policies to determine the protection status of each file, helping users make informed decisions when handling potentially risky files. By integrating predictive intelligence with CDR, Foresight supports early detection of sophisticated malware and provides actionable information for file management. ## Supported file types * PDF * DOCX * XLSX ## Risk labels * *Malicious* - High-probability prediction that the file is weaponized; may warrant automatic quarantine. * *Suspicious* - Some indicators of harm but insufficient evidence to call it malicious; caution is advised when interacting with this file. * *NoThreatsDetected* - No indicators of harm were found; file can be delivered. ## Installing Glasswall Foresight with Meteor Glasswall Foresight is included automatically when you install or update Glasswall Meteor to version 5.0.0 or later. No additional installation steps are required. ## Enabling Glasswall Foresight If your license does not include a Glasswall Foresight entitlement, Foresight is disabled by default. To enable Foresight, upload a license that includes the Foresight entitlement. Once the license is applied, Foresight activates automatically. You can use the toggle on the **Settings** page to disable it at any time. ## Using Glasswall Foresight Glasswall Foresight runs inline any time you process a file, whether the file is processed from **Clean a file** or **Sync and clean**. After processing, the analysis report includes a Foresight section under the original file details. The results fall into one of three categories: No threats detected, Malicious or Suspicious. ### No threats detected When Foresight returns "No threats detected", the file is clean and safe to use. Regardless of the policy used, the file will be considered safe and available to use. ### Malicious If a file has been detected as malicious, Meteor displays a warning that the original file is dangerous and caution should be used when handling. If the file was processed with a policy that fully sanitizes the file, Meteor displays a warning that the content could still be risky. However, if the file was processed with a policy allowing some content, Meteor blocks the file as it cannot be guaranteed that the sanitized file is safe to use. ### Suspicious If a file is identified as suspicious, Meteor displays a warning. The rebuilt file remains available for download. A similar behavior occurs when risks are allowed. The rebuilt file is available to be used. Our range of APIs deliver our Zero Trust CDR file processing capabilities wherever a file is in motion, or at rest. They ensure our customers files are protected against complex and zero-day file-based threats by our patented four-step CDR process at critical stages within document workflows. ## Embedded Engine [Learn more about the Embedded Engine APIs](/embedded-engine/embedded-engine-api-overview) ## Glasswall Halo The Glasswall cloud-native Halo provides security teams with out-of-the-box file protection capabilities. Built on a Kubernetes architecture, it provides organizations with protection that can scale infinitely to meet demand. ### [Synchronous API](/rest-api/about-synchronous-rest-api) Our Synchronous REST API enables security teams to present file analysis and rebuild requests to Glasswall Halo – typically securing files in under a second. In addition, the API facilitates policy management actions – enabling security teams to match the processing of a files components with their organizations risk appetite. ### [Asynchronous API](/rest-api/about-asynchronous-api) The Asynchronous API enables security teams to present and respond to multiple file analysis requests for Glasswall Halo. The API's ability to handle concurrent operations and non-blocking I/O allows Glasswall Halo to manage resources and respond to multiple requests concurrently – enabling the user to continue at their own convenience, eliminating the requirement to wait for file processing. ### [Prompt injection detection API](/rest-api/prompt-injection-detection-api) The Prompt injection detection API detects prompt injection in a text string before it reaches a large language model. Security teams submit the text in one request and Glasswall Halo returns a verdict in the response. Glasswall licenses this capability separately. ### [Policy management API](/rest-api/about-policy-management-api) The Policy Management API gives security teams the capability to configure content management policies to match their organizations risk appetite, allowing them to govern the treatment of files and their contents during the CDR process. Users are able to create, update, delete or reset content management policies for Glasswall Halo. ### [License management API](/rest-api/about-license-management-api) This API is used to manage active licenses and can be accessed via the main entry point into the cluster. The license management service focuses solely on license storage and does not impact the application of licenses within the cluster. ### [ICAP profile management API](/rest-api/icap-profile-management-api) The Glasswall ICAP integration provides users with a robust tool for customizing file processing by an ICAP server and managing Content Disarm and Reconstruction (CDR). The ICAP Profile Management API allows for the creation and modification of these profiles, streamlining the management and optimization of file processing workflows. Additionally, the API supports the simultaneous activation of multiple ICAP profiles, offering flexible and precise control over file security measures. ### [XML validation API](https://api.docs.glasswall.com/xml-validation-v1#XML-validation-API) The XML Validation API is a tool that streamlines XML validation, ensuring your XML markup is secure and well-formed according to the specified schema. ### [Storage monitoring API](https://api.docs.glasswall.com/storage-monitoring-V1) The Storage Monitoring API provides a secure way to automatically sanitize files stored in cloud platforms such as SharePoint and OneDrive. It allows users to identify and manage storage locations, set up and adjust monitoring rules, and view active monitors, all through authenticated connections that ensure data safety and consistency. ## Overview The Glasswall cloud-native Glasswall Halo provides security teams with out-of-the-box file protection capabilities. Built on a Kubernetes architecture, it provides organizations with protection that can scale infinitely to meet demand. Glasswall Halo's OpenAPI specification compliant design provides developer-centric web services, while open source connectors provide easy on-premises and cloud integrations. Whether you're a developer aiming to enhance your application's security or an organization dedicated to safeguarding sensitive information, this guide will enable you to quickly access and use Glasswall Halo. ## Step 1 - Access Glasswall APIs Glasswall APIs can be accessed via: [https://api.glasswall.com/](https://api.glasswall.com/) ## Step 2 - Authentication To access [Glasswall APIs](https://api.glasswall.com), you'll need to contact us at **sales@glasswall.com** for a designated username and password. You will be sent an email containing a safe link granting access to our software distribution platform. 1. Once you have your authentication credentials, navigate to [Glasswall APIs Swagger](https://api.glasswall.com). 2. Click **Authorize**. 3. In the popup, enter your authorisation credentials. 4. Click **Authorize** to gain access to the Glasswall Halo API. ## Step 3 - Process a file You can now start processing files using the various API calls available via Swagger. 1. Select the **`api/v3/cdr-file`** endpoint and the row expands to display a new section. 2. Click **Try it out** in the top right of this new section. ### Step 3.1 - Set up parameters Under the *Parameter* section for each API call there are several parameters you can alter depending on what you want to achieve. Most of these parameters are described on the page itself but you can learn more about them via [Glasswall API capabilities](/rest-api/clean-files-to-remove-threats). ### Step 3.2 - Adjust request body The next section is for the request body. This is where you can provide instructions on how you want the file to be managed by adjusting the inputs as required. ### Step 3.3 - Select file Next, you can choose the specific file you want to work with. 1. Click **Choose file**, and select the file you want to process via the popup. 2. Once you're ready, click **Execute**. You have now initiated a request to Glasswall Halo which starts processing your chosen file based on your settings. ### Step 3.4 - View response If your request has been successful, you are presented with a status code and the output file. * If you see a **201** status code it means your request was successful. * You can download the binary output by clicking **Download file**. However, if your request wasn't successful, you may see the following: * A **500** status code indicates there is a problem with your file or Glasswall Halo and you may want to try another file. * A **400** status code indicates there is an issue with your request and you may want to double-check your inputs and ensure they are correct. * A **401** status code indicates there is an issue with your authorization credentials and you may want to double-check them. ## Summary Now that you have successfully performed CDR on your first file, you can either continue to use the Swagger UI to process more files or you can begin to integrate your application POC's programmatically using this endpoint and the basic authorisation credentials. If you would like to trial Glasswall Halo by running it locally, you can follow our [Evaluation via AWS Marketplace](/halo/evaluation-via-aws-marketplace) guide. For more information please contact our sales team at **sales@glasswall.com**. Glasswall Halo can be used to process files, ensuring that any potentially malicious content is removed or neutralised. Our CDR technology instantly cleans and rebuilds files to match their ‘known good’ industry specifications, removing potential malware. The ultimate goal is to produce a sanitised version of the file, making it safe to be opened and accessed by other users or information systems without any security risks. ## What is CDR? Content Disarm and Reconstruction (CDR) is a security measure taken to protect against potential threats and cyber-attacks that may be embedded in files, particularly in documents, images, and other attachments. There are several reasons why you would want to use CDR to sanitise files before allowing other users or information systems to access them: ### Malware protection Files received from external sources, such as emails or file downloads, can contain hidden malware, viruses, or other malicious code. CDR helps ensure that any potential threats are neutralised, preventing malware infections and data breaches. ### Phishing prevention Some files may include hidden phishing links or malicious scripts, aiming to steal sensitive information or login credentials. CDR removes such dangerous elements, mitigating the risk of falling victim to phishing attacks. ### Compliance and regulation Many industries and organisations have strict data protection regulations they must adhere to. By implementing CDR, businesses can demonstrate their commitment to security and compliance with data protection standards. ### Zero-day threat mitigation CDR can be effective against new, previously unknown threats (zero-day threats) because it focuses on the file's structure and content rather than relying solely on signature-based detection. ### Safe collaboration By using CDR, organisations can safely share files internally and externally, promoting efficient collaboration while maintaining security. ### System integrity Opening a malicious file can harm not only the user's device but also the entire it system. CDR helps preserve the integrity of the system by preventing the spread of malware. Overall, employing CDR as part of a comprehensive cybersecurity strategy helps reduce the attack surface and enhances the overall safety and trustworthiness of the shared files, making it safer for users and information systems to interact with the data. ## API documentation You can use the following APIs to create a sanitised file: ``` POST api/v3/cdr-file POST api/v3/cdr ``` For more information please refer to our [API Documentation](https://api.docs.glasswall.com) ## API authentication [Learn how to authenticate Glasswall Halo](/halo/glasswall-halo-authentication-overview) ## Glasswall Halo events When you make a request to Glasswall Halo the following events take place: 1. You send a file to the Synchronous API for processing. 2. The file is stored in Glasswall Halo whilst processing occurs. 3. The Glasswall Embedded Engine is notified to process the file. 4. The Glasswall Embedded Engine retrieves the file and begins its CDR process. 5. The clean file is then returned back to the user via the API response. ## Request construction Glasswall Halo provides a wide range of file processing capabilities, accommodating both binary and Base64 encoded files. Additionally, you have the option to leverage policies to guide the file processing according to your needs. For a comprehensive understanding and practical implementation of the Policy management feature, [learn more](/rest-api/define-content-management-policies). When utilising Glasswall Halo, you can submit files in either binary or base64 format, offering the flexibility to choose the most appropriate file representation for your specific use case and application requirements. Moreover, if you solely require a clean file without the analysis report, you can make this specific request using the `response-content` query parameter with the value set to `noAnalysisReport`. This versatile functionality of Glasswall Halo empowers you to tailor the file processing process precisely to your preferences and efficiently achieve your objectives with ease. ### Binary file processing ```json POST {baseUrl}/api/v3/cdr-file?response-content=noAnalysisReport ``` ### Base64 encoded file processing Submit the Base64 encoded string in the request body to the following endpoint: ```json POST {baseUrl}/api/v3/cdr?response-content=noAnalysisReport ``` #### Request body format The body of the request should be in JSON format and include the Base64 field containing the Base64 encoded string of the file, and the `fileName` field specifying the original filename (including the appropriate file extension). ```json { "Base64": "string", "fileName": "filename.pdf", } ``` #### Variations A variation of this request is to not pass in the `response-content` query parameter. This means that you will get both the rebuilt file and the analysis report back for the file you requested. ### Binary file processing ```json POST {baseUrl}/api/v3/cdr-file ``` ### Base64 encoded file processing Submit the Base64 encoded string in the request body to the following endpoint: ```json POST {baseUrl}/api/v3/cdr ``` **Note**: - Replace \{baseurl\} with the actual base URL of the Glasswall Halo API. - Correct auth header should be provided with each request - For binary file processing, use a multipart form post, while for Base64 encoded file processing, provide the file content in the JSON request body with the appropriate filename. ## Response handling When a file is successfully processed by Glasswall Halo, you will receive a 201 HTTP status code, indicating that a new file has been created, and this file is returned in the response. The format of the response depends on whether you used the binary or Base64 endpoint. For the binary endpoint, the file will be returned with the content type `application/octet-stream`. You can read all the bytes from the response body, forming the CDR'd file. The response will also include the `content-disposition` header, which contains the filename supplied in the multipart form, if available. If the filename was not provided, a generated GUID (Globally Unique Identifier) will be returned as the filename. This feature eliminates the need for you to maintain the filename while processing occurs. If you requested a Base64 encoded file to be processed, the response will be in JSON format with a status code of 201. The JSON body will contain the Base64 encoded string representing the clean file produced by Glasswall Halo. The response will look like this: ```json { "errorReason": null, "processingId": "d3bdc536-44ba-41ad-b40e-330ce6f169ce", "processingStatus": "rebuilt", "fileType": "jpg", "analysisReport": null, "rebuiltFile": { "content": "/9j/2wBDAAMCAgM...", "contentType": "application/octet-stream", "contentEncoding": "Base64" } } ``` To access the Base64 encoded clean file, you can retrieve it from the `rebuiltFile.content` location within the response. #### Variations If you omit the `response-content` query parameter and utilize the default settings, both the clean file and the analysis report will be generated and included in the response. In the context of the binary endpoint, the response format will be a compressed`.zip` file. Upon extracting this file, you will discover two subfolders: `clean` and `report`. The `clean` folder will house the clean file, which you can freely utilize according to your needs. similarly, the `report` folder will contain the analysis report. [Learn more about analysis reports](/rest-api/gain-insights-through-analysis-reports) If you requested a Base64 encoded file to be processed, the response will be in JSON format with a status code of 201. The JSON body will contain the Base64 encoded string representing the clean file produced by Glasswall Halo. The response will look like this: ```json { "errorReason": null, "processingId": "de30c22d-fcef-467c-9ed9-16296318615b", "processingStatus": "rebuilt", "fileType": "archive", "analysisReport": { "content": "UEsDBBQAAAgIACU/EFeWuyNVWwAAAL0AAAARAAAAbWFua...", "contentType": "application/octet-stream", "contentEncoding": "Base64" } "rebuiltFile": { "content": "/9j/2wBDAAMCAgM...", "contentType": "application/octet-stream", "contentEncoding": "Base64" } } ``` To access the Base64 encoded clean archives, you can retrieve it from the `rebuiltFile.content` location within the response, if you wish to gain access to the analysis files they can be found at `analysisReport.content`. ## Summary - You have successfully protected your organisation against potential malware threats by utilising Glasswall Halo to sanitise files before sharing or processing them. - You have prevented phishing attacks and safeguarded your users by neutralising hidden malicious scripts and phishing links in files processed through Glasswall Halo. - You have ensured compliance with data protection regulations by securely handling files and eliminating sensitive information through Glasswall Halo. - You have proactively mitigated zero-day threats by using Glasswall Halo, which focuses on file structure rather than relying on known signatures. - You have established a secure and trusted file-sharing environment, both internally and externally, by leveraging Glasswall Halo capabilities. - You have maintained the integrity of your information systems by preventing malware propagation through CDR-processed files. - You have demonstrated a commitment to cybersecurity, building trust among clients, partners, and stakeholders through the secure file-handling practices achieved with Glasswall Halo. - You have streamlined your file processing workflows and reduced manual intervention by automating the process through the Glasswall Halo API integration. - You have efficiently responded to potential security incidents by incorporating CDR as an additional layer of defence in your incident response plan. - You have minimised downtime and reduced remediation costs by preventing malware infections through Glasswall Halo. - You have protected your organisation against supply chain attacks by sanitising files received from external sources using Glasswall Halo. Overall, through CDR, you have significantly enhanced your organisation's cybersecurity posture, ensuring the safety of sensitive data and mitigating file-based threats effectively. ## Quick start To try Glasswall Halo yourself, please refer to our [Quick Start Guide](/rest-api/glasswall-apis-quick-start-guide). You can use Glasswall Halo to achieve seamless process flow optimisation, empowering you to efficiently manage content management policies. ## Why use content management? By defining content management policies, or policies for short, you can use a powerful set of configuration flags that control the treatment of files and their contents during the CDR process. In diverse system scenarios, the need for varying content management flags can arise. Glasswall Halo's flexibility allows you to apply different sets of flags for each situation, accommodating multiple use cases within a single system and efficiently handling edge case scenarios without manual intervention. You can also manage policies programmatically, eliminating the overhead of storing and maintaining changes between versions. With a unified integration point, you can easily manage policies by name and effortlessly request file rebuilds, streamlining your processes and reducing the amount of code needed for executing requests against Glasswall Halo. ## API documentation To use a defined policy in a request, you have two options. * You can create a named policy using the provided API calls. * Or if you prefer a more straightforward approach, you can directly use the default policy that comes pre-installed with all Glasswall Halo installations. * Managing the default policy is easy through the Glasswall Halo portal, which offers a user-friendly interface to customize it according to your needs. * By setting the default policy through the portal, it will automatically be utilised in any API call for file rebuilding, provided that a dynamic or named policy is not explicitly specified. For more information please refer to our [API Documentation](https://api.docs.glasswall.com/policy#policy-management-policymanagement). ## API authentication [Learn how to authenticate Glasswall Halo](/halo/glasswall-halo-authentication-overview) ## Glasswall Halo events 1. To create a new policy, use the following API call in the Policy management API, with your desired policy settings in the request body. ``` POST api/v1/policies/{policyname} ``` ```json { "status": "enabled", "policySettings": { "contentManagementFlags": { "pdfContentManagement": { "acroform": "sanitise", "digitalSignatures": "sanitise", "embeddedFiles": "sanitise", "embeddedImages": "sanitise", "externalHyperlinks": "sanitise", "internalHyperlinks": "sanitise", "javascript": "sanitise", "metadata": "sanitise", "actionsAll": "sanitise", "retainExportedStreams": 1, "valueOutsideReasonableLimits": 1 }, "wordContentManagement": { "dynamicDataExchange": "sanitise", "embeddedFiles": "sanitise", "embeddedImages": "sanitise", "externalHyperlinks": "sanitise", "hiddenData": "allow", "internalHyperlinks": "sanitise", "macros": "sanitise", "metadata": "sanitise", "reviewComments": "sanitise", "trackedChanges": "sanitise" }, "excelContentManagement": { "connections": "sanitise", "dynamicDataExchange": "sanitise", "embeddedFiles": "sanitise", "embeddedImages": "sanitise", "externalHyperlinks": "sanitise", "hiddenData": "allow", "internalHyperlinks": "sanitise", "macros": "sanitise", "metadata": "sanitise", "reviewComments": "sanitise", "trackedChanges": "sanitise" }, "powerPointContentManagement": { "embeddedFiles": "sanitise", "embeddedImages": "sanitise", "externalHyperlinks": "sanitise", "hiddenData": "allow", "internalHyperlinks": "sanitise", "macros": "sanitise", "metadata": "sanitise", "reviewComments": "sanitise", "slideNotes": "sanitise" }, "svgConfig": { "hyperlinks": "sanitise", "foreignObjects": "sanitise", "scripts": "sanitise" }, "webpConfig": { "metadata": "sanitise" }, "tiffConfig": { "geoTiff": "sanitise" }, "archiveConfig": { "bmp": 1, "doc": 1, "docx": 1, "emf": 1, "gif": 1, "jpg": 1, "wav": 1, "elf": 1, "pe": 1, "mp4": 1, "mpg": 1, "pdf": 1, "png": 1, "ppt": 1, "pptx": 1, "tif": 1, "wmf": 1, "xls": 1, "xlsx": 1, "mp3": 1, "rtf": 1, "coff": 1, "macho": 1, "svg": 1, "webp": 1, "unknown": 1 } }, "extendedFormatSettings": { "version": 1, "policy": { "cdr": { "globals": { "compliance": ["spec", "isg"], "allow-changes": "visual" }, "overrides": { "nitf": { "actions": { "originator-information": { "action-preference": ["report"] } } }, "jpeg2000": { "actions": { "uuid-box": { "check-type": "present", "action-preference": ["remove", "report"] } } } } } } } } } ``` 2. If you prefer not to create a new policy but instead update the existing default policy, you can do this by sending the new policy settings in the request body to the following endpoint: ``` PUT api/v1/policies/default ``` 3. Once you have successfully created or edited your policy, you can use it in your requests within Glasswall Halo. Refer to the next section to learn how to apply a specific policy to a rebuild request. By following these steps, you can easily manage policies and tailor them to suit your requirements in Glasswall Halo. ## Request construction Glasswall Halo offers three ways to use policies: * Default * Named * Dynamic Each approach can be applied to specific CDR endpoints: #### Default policy The default policy is the pre-installed policy that comes with Glasswall Halo. While it can be edited, if no other policy is specified in the request, this default policy is used during file processing. If the Policy management API is not available, a "Sanitise all" policy is applied as a fallback – applying the most restrictive policy possible. Sending a request to the following endpoints without content management flags in the request body will apply the default policy to the processing: ``` POST api/v3/cdr-file POST api/v3/cdr ``` #### Named policy When using a named policy, you submit a request with the policy name as a query parameter in the URL. The platform will then look up the named policy, which should have been previously created, and apply it accordingly. To achieve this, send requests to the following urls: ``` POST api/v3/cdr-file?policyName={policyname} POST api/v3/cdr?policyName={policyname} ``` As the default policy is essentially a named policy, you can pass 'default' as the query parameter value to apply the default policy during processing. #### Dynamic policy The dynamic policy refers to passing the policy settings directly at the time of the request. You can make a request to the following URLs and include the content management flags as part of the request body to apply those specific settings during file processing. A dynamic policy takes precedence over any other policy if a policy name is also provided. ``` POST api/v3/cdr-file POST api/v3/cdr ``` ```json { "ContentManagementFlags": { "PdfContentManagement": { "Acroform": 1, "ActionsAll": 1, "EmbeddedFiles": 1, "EmbeddedImages": 1, "ExternalHyperlinks": 1, "InternalHyperlinks": 1, "Javascript": 1, "Metadata": 1, "DigitalSignatures": 1, "ValueOutsideReasonableLimits": 1, "RetainExportedStreams": 1 }, "WordContentManagement": { "DynamicDataExchange": 1, "EmbeddedFiles": 1, "EmbeddedImages": 1, "ExternalHyperlinks": 1, "HiddenData": 0, "InternalHyperlinks": 1, "Macros": 1, "Metadata": 1, "ReviewComments": 1, "TrackedChanges": 1 }, "ExcelContentManagement": { "DynamicDataExchange": 1, "EmbeddedFiles": 1, "EmbeddedImages": 1, "ExternalHyperlinks": 1, "HiddenData": 1, "InternalHyperlinks": 1, "Macros": 1, "Metadata": 1, "ReviewComments": 1, "TrackedChanges": 1, "Connections": 1 }, "PowerPointContentManagement": { "EmbeddedFiles": 1, "EmbeddedImages": 1, "ExternalHyperlinks": 1, "HiddenData": 1, "InternalHyperlinks": 1, "Macros": 1, "Metadata": 1, "ReviewComments": 1, "SlideNotes": 1 }, "ArchiveConfig": { "bmp": 1, "doc": 1, "docx": 1, "emf": 1, "gif": 1, "jpg": 1, "wav": 1, "elf": 1, "pe": 1, "mp4": 1, "mpg": 1, "pdf": 1, "png": 1, "ppt": 1, "pptx": 1, "tif": 1, "wmf": 1, "xls": 1, "xlsx": 1, "mp3": 1, "rtf": 1, "coff": 1, "macho": 1, "svg": 1, "webp": 1, "unknown": 1 }, "SvgConfig": { "ForeignObjects": 1, "Hyperlinks": 1, "Scripts": 1 }, "WebpConfig": { "Metadata": 1 }, "TiffConfig": { "GeoTiff": 1 } }, "ExtendedFormatSettings": { "version": 1, "policy": { "cdr": { "globals": { "compliance": ["spec", "isg"], "allow-changes": "visual" }, "overrides": { "nitf": { "actions": { "originator-information": { "action-preference": ["report"] } } }, "jpeg2000": { "actions": { "uuid-box": { "check-type": "present", "action-preference": ["remove", "report"] } } } } } } } } ``` By understanding and utilising these three policy approaches, you can effectively tailor Glasswall Halo's behaviour to meet your specific requirements during file processing. #### NITF and NSIF protection settings Configure NITF (National Imagery Transmission Format) and NSIF (NATO Secondary Imagery Format) file protection through the optional `extendedFormatSettings` section of the policy, as shown in the examples above. The values shown are illustrative; set them to match your organization's risk appetite. | Setting | Where | Values | Effect | |---------|-------|--------|--------| | `globals` | `policy.cdr` | – | The baseline every format inherits. Required whenever the `cdr` section is supplied. | | `overrides` | `policy.cdr` | `nitf`, `jpeg`, `jpeg2000`, `emf`, `wmf`, `h264`, `h265`, `sidd` | Adjusts one format: `nitf` for the file structure itself, the others for the content types embedded within it. | | `actions` | `globals` or an override | keyed by content type | How each content type is handled, using the two settings below. | | `check-type` | an `actions` entry | `all`, `invalid`, `present` | When to act. | | `action-preference` | an `actions` entry | `report`, `reveal`, `replace`, `remove`, `reject`, `omit` | The preferred outcomes in order; Halo applies the first one the matching rule supports. | A scope can also set `compliance`, `allow-changes` and per-rule `rules` overrides. See the [API Documentation](https://api.docs.glasswall.com) for the full structure. The whole section is optional. When it is absent, or when `policy` is empty (which is how it appears in the default policy), default protection rules are applied. A request or policy carrying an invalid `extendedFormatSettings` section is rejected with an error naming the setting at fault. Because the policy is read for each processing request, a change takes effect for subsequent submissions without redeploying or restarting Glasswall Halo. In this release these settings are managed through the API only—they do not appear in the Glasswall Halo portal. ## Response handling As part of the response, you will receive not only the processed file as an output but also a set of headers. Among these headers, one key header is of particular interest: *x-applied-policy*. Viewing this header allows you to determine which policy was applied during the processing. The x-applied-policy header can take on the following values: - **Name** the named policy that was applied - **Default** the configured default policy was applied - **Service-dynamic** a dynamic policy was submitted with the request and applied - **Glasswall-fallback** the fallback policy was applied since the specified policy was not available By checking the value in the *x-applied-policy* header, you can determine which policy was used to process the file. This information is helpful for tracking and understanding the policy application in Glasswall Halo. ## Summary With the capability to apply various policies in different situations, Glasswall Halo empowers you to define policies tailored to your specific needs. By assigning names to these policies, you can seamlessly process different files with distinct rules, based on the business situation or the flow of your application. Some potential scenarios where this functionality proves valuable include: #### Email flow In an email flow scenario, you could gain the flexibility to process files differently based on the sender and receiver. By employing named policies, you can define unique rules for files originating from specific senders or destined for particular recipients. This way, you can apply appropriate policies for different email sources and destinations. #### File upload By using named policies, you can create rules tailored to specific IP addresses or ranges, ensuring that files from different sources undergo distinct processing based on their origins. ### User roles and permissions In a multi-user environment, you may have different user roles with varying levels of access and permissions. Based on the user's role, you might want to apply different policies to files uploaded or processed by them. For example, administrators could have more permissive policies for handling files, while regular users might have stricter policies in place. #### Geographical locations Depending on the geographical location of the user or the server handling file uploads, you may wish to enforce specific policies to comply with local regulations or data protection laws. Different regions may have distinct data privacy requirements, and applying location-based policies ensures compliance with regional laws. With these versatile policy management capabilities, you can efficiently handle varying file processing requirements within your application, adapting to different contexts and scenarios. The ability to apply named policies empowers you to optimize file handling and security measures according to the specific needs of your business and the unique circumstances your application encounters. ## Quick start To try Glasswall Halo yourself, please refer to our [Quick Start Guide](/rest-api/glasswall-apis-quick-start-guide). File hashing is a technique used in applications for many reasons, mainly to ensure data integrity, security, and efficiency. There are a number of reasons why you may want to use file hashing such as: * **Data integrity verification** >When transferring or storing files, there is always a risk of data corruption or tampering. By generating a hash value based on the file's contents using a hashing algorithm (e.g., MD5, SHA-1, SHA-256), the application can later recompute the hash and compare it to the original hash to verify if the file has been altered or corrupted during transmission or storage. * **Deduplication** >In certain applications like cloud storage or backup systems, file deduplication is used to save storage space. Hashing enables you to identify duplicate files efficiently. Instead of storing multiple copies of the same file, the application can use a single hash to represent the duplicated content. * **Cache management** >In caching systems, file hashing can be used to quickly check if a particular file has been accessed before and is available in the cache. The hash serves as a unique identifier for the file, allowing for faster retrieval and reducing unnecessary data transfer. * **Data indexing and searching** >In large-scale applications where files need to be indexed and searched quickly, file hashing can be used to build efficient data structures like hash tables, enabling fast lookup and retrieval based on file content. * **Malware detection** >Antivirus and security applications use file hashing to identify known malware and viruses. By comparing the hash of a file against a database of known malicious hashes, these applications can quickly determine if a file is potentially harmful. ## Glasswall Halo events 1. The Glasswall Halo API allows you to interact programmatically by initiating requests to specific endpoints based on the file type: * For binary files, use the `api/v3/cdr-file` endpoint. * For Base64-encoded files, use the `api/v3/cdr` endpoint. 2. With each request, you have the option to request the generation of file hashes in any combination of three different formats: SHA-256, SHA-1, and MD5. 3. Once the request is received, the input hash is generated and then Glasswall Halo processes it using the Glasswall Embedded Engine to rebuild and analyse the input file. 4. Upon a successfully rebuilt file, the platform generates the output file hashes and both the input and output hashes are returned in the response headers. 5. If a request fails, the platform has still generated the input hash which is returned in the response headers. 6. These generated input and output file hashes can be utilised in various ways within your application or system, such as performing security checks, further processing, or any other necessary actions. ## API authentication Glasswall Halo supports two types of authentication: **Basic **and **Bearer** authentication. Before making any request, you must authenticate using the appropriate scheme based on your configuration. ### Basic If your system is configured with **Basic** authentication, you need to obtain an organisation ID and token from the system administrator. Combine these values in the format `:` and then Base64 encode them. The resulting encoded value should be included in the request header: ```json Authorization: Basic ZGVtbzpwQDU1dzByZA== ``` ### Bearer For **Bearer** authentication, you will require a bearer token obtained from your identity provider. Once you have obtained the token, include it in the request header: ```json Authorization: Bearer ZGVtbzpwQDU1dzByZA== ``` **Note**: ensure that you use the appropriate authentication method as per your system's configuration. Using the correct authentication ensures successful access to the Glasswall Halo API and prevents any authorization issues. [Learn more about authenticating Glasswall Halo](/halo/glasswall-halo-authentication-overview) ## Request construction When leveraging Glasswall Halo for processing binary or Base64 encoded files, you have the flexibility to request the generation of multiple hash types, including SHA-256, SHA-1, and MD5. When making an API request, you can specify any combination of these hash types, and in response, you will receive the generated hash values for both the input and output files in the response headers. This enables you to ensure data integrity and authenticity, as well as facilitate efficient handling of files within your application or system. ### Binary file processing ```json POST {baseUrl}/api/v3/cdr-file?generate-hash-types=SHA256,SHA1,MD5 ``` ### Base64 encoded file processing Submit the Base64 encoded string in the request body to the following endpoint: ```json POST {baseUrl}/api/v3/cdr?generate-hash-types=SHA256,SHA1,MD5 ``` #### Request body format The body of the request should be in JSON format and include the base64 field containing the Base64 encoded string of the file, and the filename field specifying the original filename (including the appropriate file extension). ```json { "Base64": "string", "fileName": "filename.pdf", } ``` **Note** - Replace \{baseurl\} with the actual base URL of the Glasswall Halo API. - Correct auth header should be provided with each request. - Setting `generate-hash-types` to a comma separated list of hash values in the URL ensures that the correct hashes are generated and returned in the response. - For binary file processing, use a multipart form post, while for Base64 encoded file processing, provide the file content in the JSON request body with the appropriate filename. ## Response handling If you are successful in making a request to the Glasswall Halo API, a 201 status code is returned. If you have requested any hashes to be generated and returned, both input and output hashes will be available in the response header. If the response is unsuccessful, only the input hash values will be returned. You can identify the hash values with the following header keys: ### For SHA-256: - Input file hash: `x-hash-sha256-input` - Output file hash: `x-hash-sha256-output` ### For SHA-1: - Input file hash: `x-hash-sha1-input` - Output file hash: `x-hash-sha1-output` ### For MD5: - Input file hash: `x-hash-md5-input` - Output file hash: `x-hash-md5-output` ### Example ```json access-control-allow-origin: * access-control-expose-headers: * content-disposition: attachment; filename=18402777-7826-457f-bc02-6446611495e6.zip; filename*=UTF-8''18402777-7826-457f-bc02-6446611495e6.zip content-length: 288843 content-type: application/octet-stream date: Fri,21 Jul 2023 12:42:15 GMT strict-transport-security: max-age=31536000; includeSubDomains x-applied-policy: service-dynamic x-file-size: 301189 x-filetype: pdf x-hash-md5-input: AF5EB421F8AFC69AC4178ACF747E618C x-hash-md5-output: 8018FC4F37AB858442F354430203CEB4 x-hash-sha1-input: 9C714CAC1F82A450728F3C046CEFEFAF0BBD2C07 x-hash-sha1-output: 058991F3A252F0A4FD0E7F8B3178D3F15494099B x-hash-sha256-input: 5F83AB533252C00AD1C60A0EFC016BF273497F3179FD6CE642AD8A4857148B8D x-hash-sha256-output: 9D5D88F073D74160828F92E9E8DADAC5712706C65D49448D9ABA91D8FDC6FC28 x-processing-id: 18402777-7826-457f-bc02-6446611495e6 x-processing-status: rebuilt ``` ## Summary By using generated hashes in your file processing workflow via Glasswall Halo, you can streamline operations, enhance security, and improve the reliability of your application. These collectively contribute to a more robust and resilient system that ensures the integrity and safety of your files. ## Quick start To try Glasswall Halo yourself, please refer to our [Quick Start Guide](/rest-api/glasswall-apis-quick-start-guide). Glasswall Halo offers advanced protection for archive files. This powerful feature utilises the state-of-the-art Embedded Engine to safeguard every single file within an archive. It not only shields the files but also intelligently recompresses the entire archive into a supported format. With this enhanced functionality, your archive files are effectively secured and optimized for seamless use. ## Why CDR? Content Disarm and Reconstruction (CDR) is a security measure taken to protect against potential threats and cyber-attacks that may be embedded in files, particularly in documents, images, and other attachments. There are several reasons why you would want to use CDR to sanitise files before allowing other users or information systems to access them. [Learn more about using CDR to remove threats from files.](/rest-api/clean-files-to-remove-threats) Scanning an archive, which is a file format that contains multiple files and directories, can offer several advantages over scanning individual files one by one. Archives are specifically designed to bundle multiple files together, often with the intention of reducing storage space or organising related content. However, it's important to note that not all compressed file types are archives, as some compression formats focus solely on reducing file sizes without necessarily grouping multiple files. The following reasons are why you might want to scan an archive rather than a single file: 1. **Efficiency**: scanning a single archive file is more efficient than scanning multiple individual files, especially when dealing with a large number of files. It saves time and reduces the manual effort required to scan each file separately. 2. **Comprehensive protection**: archives often contain multiple files that are interconnected or dependent on each other. Scanning the entire archive ensures that all files within it are checked for potential threats, ensuring comprehensive protection against malware or other security risks. 3. **Streamlined management**: when dealing with a collection of related files, such as documents, images, or code, keeping them in an archive maintains organisational structure. Scanning the archive helps to keep this structure intact, making it easier to manage and share the entire collection. 4. **Reduced false positives**: some security software may trigger false positive detections for individual files due to the way they are packed or encrypted. Scanning an archive can help reduce the likelihood of such false positives by analysing the files within their intended context. 5. **Ease of distribution**: when sharing a collection of files, packaging them in an archive can simplify distribution. Scanning the archive before sharing ensures that the recipient gets a clean and safe bundle of files. 6. **Simplified user experience**: from a user's perspective, scanning a single archive is more straightforward and convenient than initiating scans for each individual file. It simplifies the scanning process and reduces user interaction. In summary, scanning an archive offers efficiency, comprehensive protection, and streamlined management when dealing with multiple files. It can help users maintain security, organisational structure, and an overall smoother experience when working with collections of files. ## API documentation By utilising Glasswall Halo, you can safely process archives and ensure they are free from hidden threats and malicious content. You can use the following API's to create an archive of sanitised files: ``` POST api/v3/cdr-file POST api/v3/cdr ``` For more information please refer to our [API documentation](https://api.docs.glasswall.com). ## API authentication [Learn how to authenticate Glasswall Halo](/halo/glasswall-halo-authentication-overview) ## Glasswall Halo events When you make a request to Glasswall Halo the following events take place: 1. You send an archive to the Synchronous API for processing. 2. The archive is decompressed inline with [Glasswall's Archive Support](/halo/glasswall-halo-archive-support) rules. 3. Each file within the archives is stored in Glasswall Halo whilst processing occurs. 4. The Glasswall Embedded Engine is notified to process all the files found inside the archive. 5. The Glasswall Embedded Engine retrieves each of the files and begins its CDR process. 6. Glasswall Halo then re-builds the archive with all the sanitised files in the original structure. 7. The clean archive is then returned back to the user via the API response. ## Request construction Glasswall Halo provides a wide range of file processing capabilities, accommodating both binary and Base64 encoded archives, and both of these endpoints also support password encrypted zips. Please refer to our [supported file types](/halo/glasswall-halo-supported-file-types) for a complete list of supported archive file types. Additionally, you are able to use our [policy management API](/rest-api/about-policy-management-api) to guide how each file within an archive is processed. When utilising Glasswall Halo, you can submit archives in either binary or base64 format, offering the flexibility to choose the most appropriate file representation for your specific use case and application requirements. If you only require archives with just clean files and no analysis reports, you can make this specific request using the `response-content` query parameter with the value set to `noAnalysisReport`. In this instance a request will be made for both the clean file and analysis reports to be produced for the submitted archive. This versatile functionality of Glasswall Halo empowers you to tailor the file processing process precisely to your preferences and efficiently achieve your objectives. ### Binary file processing ```json POST {baseUrl}/api/v3/cdr-file ``` ### Base64 encoded file processing Submit the Base64 encoded string in the request body to the following endpoint: ```json POST {baseUrl}/api/v3/cdr ``` #### Request body format The body of the request should be in JSON format and include the Base64 field containing the Base64 encoded string of the file, and the `fileName` field specifying the original filename (including the appropriate file extension). ```json { "Base64": "string", "fileName": "filename.zip", } ``` **Note** - Replace \{baseurl\} with the actual base URL of the Glasswall Halo API. - Correct auth header should be provided with each request. - For binary file processing, use a multipart form post, while for base64 encoded file processing, provide the file content in the JSON request body with the appropriate filename. - If you wish to provide a password for an encrypted zip file you can do this by providing it in the header of the binary request with the key `password` or it can be supplied in the body of the Base64 request. ## Response handling When an archive is successfully processed by Glasswall Halo, you will receive a 201 http status code, indicating that a new archive has been created, and this file is returned in the response. The format of the response depends on whether you used the binary or Base64 endpoint. For the binary endpoint, the archives will be returned with the content type `application/octet-stream`. You can read all the bytes from the response body, forming the CDR'd archive. The response will also include the `content-disposition` header, which contains the filename supplied in the multipart form, if available. If the filename was not provided, a generated GUID will be returned as the filename. This feature eliminates the need for you to maintain the filename while processing occurs. If you requested a Base64 encoded file to be processed, the response will be in JSON format with a status code of 201. The JSON body will contain the Base64 encoded string representing the clean file produced by Glasswall Halo. The response will look like this: ```json { "errorReason": null, "processingId": "de30c22d-fcef-467c-9ed9-16296318615b", "processingStatus": "rebuilt", "fileType": "archive", "analysisReport": { "content": "UEsDBBQAAAgIACU/EFeWuyNVWwAAAL0AAAARAAAAbWFua...", "contentType": "application/octet-stream", "contentEncoding": "Base64" } "rebuiltFile": { "content": "/9j/2wBDAAMCAgM...", "contentType": "application/octet-stream", "contentEncoding": "Base64" } } ``` To access the Base64 encoded clean archive, you can retrieve it from the `rebuiltFile.content` location within the response, if you wish to gain access to the analysis files they can be found at `analysisReport.content`. Now that you have the output, you can leverage it in a variety of ways. If your preference is for clean files only, the resulting archive will mirror the structure of the initial input archive. By downloading and unpacking this archive, you can meticulously review each file. In the event that any individual file encounters processing issues, it will be substituted with a `.txt` file, detailing the reasons for the processing failure. When an analysis report is requested to be generated the report folder within the returned archive will be a file called `manifest.cdr-json`, this is a JSON output detailing the result of each file, this can be used to quickly understand and make decisions for each file within an archive. On the other hand, if your aim is to analyse all files within an archive, you'll receive an archive that upholds the original structure. However, in this instance, the previously clean files will be substituted with `.json` or `.xml` files. These files embody the analysis reports corresponding to each file's analysis. If your objective is to obtain both the clean file and the analysis report from the binary endpoint, the result will be a zip archive encompassing two distinct folders: `clean` and `report`. This output zip archive will bear a globally unique identifier (GUID) as its file name, directly linked to the transaction id within Glasswall Halo. Contained within each of these folders will be an archive mirroring the name of the initial input archive. the `clean` folder comprises the clean archive, while the `report` folder holds the archive containing all the analysis reports alongside the `manifest.cdr-json` file. The`manifest.cdr-json`File has the following structure: ```json { "rebuilt":[ { "filename":"/Sample.docx" } ], "failed":[ { "filename":"/Sample.pdf", "reason": "This is a reason" } ], "errored":[ { "filename":"/Sample.png", "reason": "This is a reason" } ], "allowed":[ { "filename":"/Sample.xlsx", "reason": "This is a reason" } ] } ``` Within the 'JSON' file, every individual section serves to signify the overarching result. Each result category is accompanied by a list of file names, thereby clarifying which specific files within a given archive correspond to that particular outcome. In scenarios where the result isn't categorised as 'rebuilt', a detailed explanation will be furnished to clarify the underlying cause of that particular outcome. ## Summary - You have achieved streamlined scanning, saving time when dealing with numerous files. - You have achieved thorough checking of all files within an archive, ensuring security against threats. - You have achieved the maintenance of file organization while simplifying management and sharing. - You have achieved a reduction in false positive detections caused by packed or encrypted files. - You have achieved simplified distribution of files while ensuring their safety through packaging in an archive. - You have achieved a user-friendly scanning experience by minimizing interactions and complexities. By using CDR you have significantly enhanced your organisation's cybersecurity posture, ensuring the safety of sensitive data and mitigating file-based threats effectively. ## Quick start To try Glasswall Halo yourself, please refer to our [Quick Start Guide](/rest-api/glasswall-apis-quick-start-guide). You can use Glasswall Halo to analyse the contents of a given file, gaining insights and extracting relevant information about the files structure, and potentially hidden risky content, with the aim of extracting meaningful and actionable information. ## File analysis reports Performing in-depth content analysis of a file is useful across various domains and industries: 1. **Information extraction:** by analysing the binary structure of files, you can extract valuable information that might be obscured within the intricate arrangement of bytes. This information can include insights, trends, statistics, and patterns that are crucial for making informed decisions. 2. **Decision-making:** content analysis of the binary structure helps individuals and organisations make data-driven decisions based on the flow and arrangement of data within a file. Understanding the file's structure aids in grasping the context and relationships between different parts of the data. 3. **Trend identification:** analysing the binary structure allows you to identify emerging trends early on, especially in fields like machine learning where data patterns and structures play a key role in training models and recognising trends. 4. **Quality assurance:** analysing the binary content of files ensures the quality and accuracy of documents. By examining structural components and verifying expected patterns, you can identify discrepancies that might indicate errors or inconsistencies. 5. **Risk assessment:** analysing the binary structure of files lets you identify potential structures that could harbour security threats or vulnerabilities. Recognising unusual patterns or headers can facilitate proactive risk mitigation. 6. **Research and academia:** understanding the contents and structural patterns within documents is essential for identifying trends and drawing insights in research and academia. This extends to security research, where analysing the binary structure of files can reveal crucial evidence. 7. **Data recovery:** analysing the binary structure becomes invaluable when attempting data recovery from corrupted or damaged files. By recognising familiar patterns, you can piece together fragments of the file even when parts of it are corrupted. 8. **File validation:** analysing the binary structure helps validate the integrity of files. verifying headers, checksums, and other structural components ensures that files haven't been tampered with or modified without authorization. 9. **Media and multimedia files:** when dealing with audio, image, and video files, analysing their binary structure is essential for identifying headers, metadata, and encoding schemes. This information ensures proper rendering and playback of media content. 10. **Forensics and investigations:** in the realm of digital forensics, analysing the binary structure of files is a cornerstone for reconstructing events, tracking digital evidence, and understanding the sequence of actions taken within a system. Understanding the binary structure of files is a fundamental skill in many technical fields, offering insights that range from extracting valuable information to ensuring security and integrity. It's a versatile tool that has implications across disciplines where data and information play a critical role. ## API documentation You can use the following APIs to create an analysis report for any given file: ``` POST api/v3/cdr-file?response-content=noRebuiltFile&format=JSON POST api/v3/cdr?response-content=noRebuiltFile&format=JSON ``` For more information please refer to our [API documentation](https://api.docs.glasswall.com) ## API authentication [Learn how to authenticate Glasswall Halo](/halo/glasswall-halo-authentication-overview) ## Glasswall Halo events When you make a request to Glasswall Halo the following events take place: 1. You send a file to the Synchronous API for processing. 2. The file is stored in Glasswall Halo whilst processing occurs. 3. The Glasswall Embedded Engine is notified to process the file. 4. The Glasswall Embedded Engine retrieves the file and performs CDR. 5. The analysis report is then returned to the user via the API response. ## Request construction Glasswall Halo provides a wide range of file processing capabilities, accommodating both binary and Base64 encoded files. Additionally, you have the option to leverage policies to guide the file processing according to your needs. [Learn more about defining content management policies](/rest-api/define-content-management-policies) When utilising Glasswall Halo, you can submit files in either binary or Base64 format, offering the flexibility to choose the most appropriate file representation for your specific use case and application requirements. Moreover, if you solely require an analysis report without the clean file, you can make this specific request using the `response-content` query parameter with the value set to `noRebuiltFile`. You are also able to request the analysis report in either XML of JSON format by using the `format` query param and setting the value to either `XML` or `JSON`. This versatile functionality of Glasswall Halo empowers you to tailor the file processing process precisely to your preferences and efficiently achieve your objectives with ease. ### Binary file processing ```json POST {baseUrl}/api/v3/cdr-file?response-content=noRebuiltFile&format=JSON ``` ### Base64 encoded file processing Submit the Base64 encoded string in the request body to the following endpoint: ```json POST {baseUrl}/api/v3/cdr?response-content=noRebuiltFile&format=JSON ``` #### Request body format The body of the request should be in JSON format and include the Base64 field containing the base64 encoded string of the file, and the `fileName` field specifying the original filename (including the appropriate file extension). ```json { "Base64": "string", "fileName": "filename.pdf", } ``` #### Variations A variation of this request is to not pass in the `response-content` query parameter. This means that you will get both the rebuilt file and the analysis report back for the file you requested. You can also pass in `format=XML` which will generate an XML analysis report instead of JSON. This can be done regardless of the desired output. If you omit this parameter the default of XML will be applied. **Note:** Analysis reports for NITF (National Imagery Transmission Format) and NSIF (NATO Secondary Imagery Format) files are available in JSON format only. Requesting an XML report for a standalone NITF or NSIF file is rejected with a `400 Bad Request` response containing the following error, and no processed file is returned: ```json { "Errors": [ { "ErrorCode": 4033, "ErrorDescription": "XML report format not supported for NITF files" } ] } ``` Because omitting the `format` parameter applies the default of XML, always set `format=JSON` when submitting NITF or NSIF files. The analysis report for a NITF or NSIF file has a different structure and format from the analysis report described above, with its own fields. An integration that parses reports should check the file type before reading one. ### Binary file processing ```json POST {baseUrl}/api/v3/cdr-file?response-content=noRebuiltFile&format=XML ``` ```json POST {baseUrl}/api/v3/cdr-file?format=XML ``` ### Base64 encoded file processing Submit the Base64 encoded string in the request body to the following endpoint: ```json POST {baseUrl}/api/v3/cdr?response-content=noRebuiltFile&format=XML ``` ```json POST {baseUrl}/api/v3/cdr?format=XML ``` **Note**: - Replace \{baseurl\} with the actual base URL of the Glasswall Halo API. - Correct auth header should be provided with each request - For binary file processing, use a multipart form post, while for Base64 encoded file processing, provide the file content in the JSON request body with the appropriate filename. ## Response handling When you submit a request to the binary endpoint for an analysis report, the response will provide you with a binary version of the analysis report. This report can be downloaded and opened in the format you specified in your request. Alternatively, if your request is directed to the base64 endpoint, the response will consist of a `json` object. Within this object, the analysis report is presented as a base64 encoded string, accessible through the `analysisReport.content` field. By decoding this Base64 string, you can then access the analysis report in the desired format you've indicated. ```json { "errorReason": null, "processingId": "de30c22d-fcef-467c-9ed9-16296318615b", "processingStatus": "rebuilt", "fileType": "archive", "analysisReport": { "content": "UEsDBBQAAAgIACU/EFeWuyNVWwAAAL0AAAARAAAAbWFua...", "contentType": "application/octet-stream", "contentEncoding": "Base64" } "rebuiltFile": { } } ``` When you open the report there will be a number of information points which you may want to pay attention to. [Learn more about Glasswall Analysis Reports](https://docs.glasswall.com/rest-api/gain-insights-through-analysis-reports) ## Summary You have now been able to conduct a comprehensive analysis of the provided file's contents, successfully extracting valuable insights, and relevant information regarding the file's structure, and potentially embedded media. This achievement facilitated the extraction of meaningful and actionable data, contributing to informed decision-making and enhancing overall understanding. ## Quick start To try Glasswall Halo yourself, please refer to our [Quick Start Guide](/rest-api/glasswall-apis-quick-start-guide). When handling files electronically, it's important to identify their file type correctly. This eliminates the threat of file type masquerading, where files may be misrepresented with false extensions or MIME types. Our API correctly identifies the true file type, so that the system can take appropriate actions to prevent security breaches, data corruption, or unintended processing of files. Once the correct file type is identified, you can make informed decisions within your application based on the following scenarios: ### Supported file type If the detected file type is included in the list of supported file types, you can confidently proceed with regular CDR processing of the file within your system. ### Unsupported file type If the detected file type falls under the list of unsupported file types, it is essential to take immediate action to prevent any further progression of that file within your system. In such cases, it is recommended that you implement a "fail fast" mechanism to halt any further processing of the file, ensuring that your system is protected from handling potentially harmful or incompatible files. ### File type masquerading If the detected file type differs from the file's declared extension or MIME type, it may indicate file masquerading, where a malicious actor attempts to mislead the system. In such a scenario, consider taking more evasive actions to handle the file within your system. these actions may include isolating the file, running additional security checks, or notifying administrators for further investigation. ## Glasswall Halo events How to make a request to Glasswall Halo: 1. Begin by initiating a request to the Glasswall Halo API, choosing the appropriate endpoint based on the file type: * For **Binary** files, use the `api/v3/cdr-file` endpoint. * For **Base64** encoded files, use the `api/v3/cdr` endpoint. 2. Glasswall Halo processes the request and conducts file type detection on the original file. 3. To ensure that only file type detection is performed without any further actions, include the `response-content` parameter in the URL with the value set to `noRebuiltFile,noAnalysisReport`. This activates the file type detection only mode. 4. Glasswall Halo analyses the file using the Glasswall Embedded Engine to determine the true file type. 5. Upon successful detection, Glasswall Halo includes the identified true file type in the response header with the header key`x-filetype`. 6. You can now use the obtained true file type information for further processing, security checks, or any other necessary actions within your application or system. ## API authentication Glasswall Halo supports two types of authentication: *Basic* and *Bearer*. Before making any request, you must authenticate using the appropriate scheme based on your configuration. ### Basic If your system is configured with *Basic* authentication, you need to obtain an organisation ID and token from the System Administrator. Combine these values in the format`:`And then Base64 encode them. The resulting encoded value should be included in the request header as follows: ```json Authorization: Basic ZGVtbzpwQDU1dzByZA== ``` ### Bearer For bearer authentication, you will require a bearer token obtained from your identity provider. Once you have obtained the token, include it in the request header as follows: ```json Authorization: Bearer ZGVtbzpwQDU1dzByZA== ``` **Note**: ensure that you use the appropriate authentication method as per your system's configuration. Using the correct authentication ensures successful access to the Glasswall Halo API and prevents any authorization issues. [Learn more about authenticating Glasswall Halo](/halo/glasswall-halo-authentication-overview) ## Request construction To process a binary file or a base64 encoded file using Glasswall Halo, follow the instructions below: ### Binary file processing Submit the binary string in the request body to the following endpoint: ```json POST {baseUrl}/api/v3/cdr-file?response-content=noRebuiltFile,noAnalysisReport ``` ### Base64 encoded file processing Submit the Base64 encoded string in the request body to the following endpoint: ```json POST {baseUrl}/api/v3/cdr?response-content=noRebuiltFile,noAnalysisReport ``` #### Request body format The body of the request should be in JSON format and include the Base64 field containing the Base64 encoded string of the file, and the filename field specifying the original filename (including the appropriate file extension). ```json { "base64": "string", "fileName": "filename.pdf", } ``` **Note** - Replace \{baseurl\} with the actual base URL of the Glasswall Halo API. - Correct `auth header` should be provided with each request. - Setting `response-content` to `noRebuiltFile,noAnalysisReport` in the URL ensures that only file type detection is performed without rebuilding the file or generating an analysis report. - For binary file processing, use a multipart form post, while for Base64 encoded file processing, provide the file content in the JSON request body with the appropriate filename. ## Response handling After making a successful request, you will receive a 201 status code. However, when running the API in file type detect mode, there will be no file output in the response. To identify the true file type, inspect the response header and locate the `x-filetype` key. The value of this header will indicate the true file type. - If the `x-filetype` value is unknown, it means that Glasswall was unable to identify the true file type for the given file. - If the `x-filetype` value is 'txt', it indicates that the file contains purely UTF-8 or ASCII text. ```json Status: 201 Created Headers: access-control-allow-origin: * access-control-expose-headers: * content-length: 0 content-type: text/plain date: Thu,20 Jul 2023 10:55:46 GMT strict-transport-security: max-age=31536000; includeSubDomains x-applied-policy: service-dynamic x-file-size: 301189 x-filetype: pdf x-processing-id: 779b6828-1e1b-4148-9dc0-a3f1e38cb119 x-processing-status: rebuilt ``` In the example above, the `x-filetype` value is PDF, indicating that the true file type is a PDF document. For further details about possible error codes that can be returned by the API, refer to our [API Documentation](https://api.docs.glasswall.com). There you can also find comprehensive information for handling various error scenarios. #### Expected values The output file types are associated with file type groups rather than individual extensions. With Glasswall's Embedded Engine multiple extensions are identified by a single value. To better comprehend the output values of the file type header, please refer to the table below: | File type value | Extensions | |-------------------------------|-----------------------------------------------| | Doc | .Doc, .dot | | Xls | .Xls, .xlt | | Ppt | .Ppt, .pps, .pot | | PDF | .PDF | | Docx | .Docx, .docm, .dotx, .dotm | | Xlsx | .Xlsx, .xltx, .xltm, .xlsm | | Pptx | .Pptx, .pptm, .ppsx, .ppam, .potm, .ppsm, .potx | | Tiff | .Tif, .tiff, .geotiff | | Svg | .Svg | | Emf | .Emf | | Wmf | .Wmf | | Jpeg | .Jpeg, .jpg, .jpe, .jif, .jfif, .jfi | | Png | .Png | | Gif | .Gif | | Bmp | .Bmp, .dib | ## Summary By implementing true file type detection in your application, you can ensure that your system processes files safely and efficiently, mitigating risks associated with unsupported or potentially malicious file types. Properly handling file type detection and reacting appropriately to different scenarios will help maintain the security and reliability of your infrastructure. ## Quick start To try Glasswall Halo yourself, please refer to our [Quick Start Guide](/rest-api/glasswall-apis-quick-start-guide). Glasswall's APIs can be accessed via endpoints; a specific location within an API that accepts requests and sends back responses. It's a way for different systems and applications to communicate with each other, by sending and receiving information and instructions via the endpoint. You're able to input a file directly into each endpoint to perform their designated function, by following these steps. These steps are virtually identical for all the endpoint functions. You can demo Glasswall's APIs via our [Swagger](https://api.glasswall.com/swagger/index.html). 1. To run a file through a specific endpoint, expand the desired endpoint and click **Try it out**. 2. Select your file. 3. Click **Execute**. The endpoint will perform its function and create its corresponding output for your reference. Glasswall's Synchronous Rest API endpoints enable security teams to present file analysis and rebuild requests to Glasswall Halo – securing files in under a second. In addition, the Rest API facilitates policy management actions – enabling security teams to match the processing of file components with their organizations risk appetite. Instead of looking for malicious content, our advanced CDR (Content Disarm and Reconstruction) treats all files as untrusted, validating, rebuilding and cleaning each file to a safe and compliant standard – automatically removing potential threats. Glasswall Halo’s flexible Kubernetes-based architecture allows standalone machine instances to act together in concert, with a load-balancer configured to allow auto-scaling and self-healing. This is characterized as a disposable Kubernetes cluster configuration. ## Why Glasswall Synchronous API? * Our Synchronous API performs complete file analysis - giving you transparency into file non-conformance with industry specifications. * We provide content management options to shape your organization’s security policy based on your personal risk appetite. * We offer complete file protection - threats are removed and files returned to known-good specifications. * Our software performs true file type detection, going beyond just the file extension or magic number. Integrating our advanced CDR technology into your company's infrastructure is easy using our APIs. ## Accessing Synchronous API The Glasswall Synchronous API can be accessed via: [https://api.glasswall.com/](https://api.glasswall.com/) ## Endpoints The Glasswall Synchronous API consists of several endpoints which carry out specific functions. The endpoints and their descriptions are listed below. ### Analyse and rebuild This endpoint analyses and rebuilds binary or Base64 encoded files to determine if they contain threats, by comparing them to their known manufacturer's specification. Once the process is complete, the rebuilt file along with either a XML or JSON report, is produced and available as a zip download in the responses section. If an archive file is selected, the response is a zip file which includes the rebuilt archive along with a report file detailing the outcome of each processed file within the archive. NITF (National Imagery Transmission Format) and NSIF (NATO Secondary Imagery Format) files use these same endpoints with no request changes. Analysis reports for these files are available in JSON format only: requesting an XML report returns a `400 Bad Request` (error code 4033), so set `format=JSON` when submitting them. See [Gain insights through analysis reports](/rest-api/gain-insights-through-analysis-reports) for the full error response. ### Import and export This endpoint exports binary or Base64 encoded files as a zip containing SISL/XML file(s), or imports a zip containing SISL/XML file(s) from the export endpoint and returns a binary or Base64 encoded file. For more information please refer to our [API Documentation](https://api.docs.glasswall.com/v3#CDR-platform-sync-rest-api). While the Synchronous API effectively meets the need for handling substantial volumes of data, certain client architecture patterns will require the incorporation of an asynchronous mode. With Glasswall Halo's Asynchronous API, requests can be handled via a two-stage process. Initially, a UUID is generated for a request, which can subsequently be used to retrieve the processing outcome. This enables the client to continue at its own convenience, eliminating the requirement to wait for file processing. ## Accessing Asynchronous API The Glasswall Asynchronous API can be accessed via: [https://api.glasswall.com/](https://api.glasswall.com/swagger/index.html?urls.primaryname=async%20api%20v1) ## Endpoints The Glasswall Asynchronous API consists of several endpoints which carry out specific functions. The endpoints and their descriptions are listed below. **CDR Async** This endpoint asynchronously analyses and rebuilds binary or Base64 encoded files to determine if they contain threats, by comparing them to their known manufacturer's specification. Once the process is complete, the rebuilt file along with either a XML or JSON report, is produced and available as a zip download in the responses section. NITF (National Imagery Transmission Format) and NSIF (NATO Secondary Imagery Format) files use this same endpoint with no request changes. Analysis reports for these files are available in JSON format only: an XML report request is rejected at submission with a `400 Bad Request` (error code 4033) rather than returning a transaction id, so set `format=JSON` when submitting them. See [Gain insights through analysis reports](/rest-api/gain-insights-through-analysis-reports) for the full error response. For more information please refer to our [API Documentation](https://api.docs.glasswall.com/async-v1#asynchronous-API). The Policy management API gives security teams the capability to configure content management policies to match their organizations risk appetite, allowing them to govern the treatment of files and their contents during the CDR process. Users are able to create, update, delete or reset content management policies for Glasswall Halo. ## Accessing Policy management API The Glasswall Policy Management API can be accessed via: [https://api.glasswall.com/](https://api.glasswall.com/swagger/index.html?urls.primaryname=policy%20api%20v1) ## Endpoints The Glasswall Policy management API consists of several endpoints which carry out specific functions. The endpoints and their descriptions are listed below. * **Create policy**: this endpoint request creates a new policy with the provided values. * **Update policy**: this endpoint request updates an existing policy with the provided values. * **Get policy**: this endpoint request retrieves details about an existing policy. * **Delete policy**: this endpoint request deletes the specified policy. * **Get all policies**: this endpoint request retrieves a complete list of all configured policies. * **Reset policy**: this endpoint request resets a specified policy to its default schema values. ## NITF and NSIF protection settings NITF (National Imagery Transmission Format) and NSIF (NATO Secondary Imagery Format) protection is configured through the optional `extendedFormatSettings` section of a content management policy. See [Define content management policies](/rest-api/define-content-management-policies). For more information please refer to our [API Documentation](https://api.docs.glasswall.com/policy#policy-management). The License management API manages active licenses and can be accessed via the main entry point into the cluster. The license management service focuses solely on license storage and does not impact the application of licenses within the cluster. **Note:** only one license can be stored in Glasswall Halo, but it is possible to update or overwrite it at any time. To obtain your new license, please [contact us](https://www.glasswall.com/contact). ## Endpoints The Glasswall License management API consists of several endpoints which carry out specific functions. The endpoints and their descriptions are listed below. * **Insert new license**: this endpoint request stores a new active license. * **Description of current license**: this endpoint request returns a description of the currently installed license. * **Delete current license**: this endpoint request enables the currently installed license to be deleted. For more information please refer to our [API Documentation](https://api.docs.glasswall.com/license#license-management-API). The Glasswall ICAP integration provides users with a robust tool for customizing file processing by an ICAP server and managing Content Disarm and Reconstruction (CDR). The ICAP Profile Management API allows for the creation and modification of these profiles, streamlining the management and optimization of file processing workflows. Additionally, the API supports the simultaneous activation of multiple ICAP profiles, offering flexible and precise control over file security measures. ## Endpoints The ICAP profile management API consists of several endpoints which carry out specific functions. The endpoints and their descriptions are listed below. * **Retrieve and edit ICAP profile settings**: this endpoint request supports the retrieval and editing of current ICAP profile settings. * **Retrieve a specific ICAP profile setting**: this endpoint request retrieves a specific ICAP profile setting. * **Creates a new ICAP profile**: this endpoint request creates a new ICAP profile. * **Update an ICAP profile**: this endpoint request updates an existing ICAP profile. * **Delete an ICAP profile**: this endpoint request deletes an existing ICAP profile. * **Resets profiles to schema default values**: this endpoint request resets the settings for a specified profile to the default values in the profile schema. this also applies to the ‘default’ profile. For more information please refer to our [API Documentation](https://api.docs.glasswall.com/profile-v1#ICAP-profile-management-API). Prompt injection is an attack in which instructions concealed in ordinary text hijack a large language model, causing it to disregard its own instructions, disclose its system prompt, or act on the attacker's behalf. Text arriving from outside an organization, such as a customer message, an extracted document, or a web page, cannot be assumed to be safe to pass to a model. The Glasswall Prompt injection detection API gives security teams the capability to check text for prompt injection before it reaches a large language model. Text is submitted in a single request and a verdict is returned in the response, allowing the text to be blocked, escalated, or released according to the organization's risk appetite. Detection runs entirely within your Glasswall Halo deployment. Submitted text is not logged, written to disk, or sent to a third-party model. Prompt injection detection is licensed separately and is an optional component of a Glasswall Halo deployment. To enable it, please [contact us](https://www.glasswall.com/contact). ## Accessing Prompt injection detection API The Glasswall Prompt injection detection API can be accessed via: https://api.glasswall.com/ ## Endpoint The Prompt injection detection API consists of a single endpoint, **Detect prompt injection**, which analyses a submitted text string to determine whether it contains a prompt injection attack, and returns a verdict indicating either that no threat was detected or that the text is malicious. ## Supported input Detection is scoped to English prose of between 10 and 512 tokens. Text outside that scope is rejected rather than scored, and the reason is returned in the response. This covers text in another language, and text that is runnable code such as a script, a shell command, or a tagged code block. For more information please refer to our [API Documentation](https://api.docs.glasswall.com/profile-v1#ICAP-profile-management-API). #### Author: Connor Morley (Principal Malware Security Researcher) [Download PDF](https://app.box.com/s/h4z9wr13ezcn50bv3gub563upmbh9st3) ## Purpose outline This document provides a summary of Glasswall’s research into the threats posed by polyglot files including how they are created and how they can be effectively mitigated. It is intended to offer readers: * The threat posed by polyglots, their malicious application, and reasoning for use in operations. * Limitations in mitigation via existing/traditional security systems which depend on file type determination. * Information on how polyglots are used in real-world scenarios as well as hypothetical deployments. * An explanation of the various types of polyglot creation methods as well as their strengths and weaknesses. * Technical description on how polyglots can be created using various file formats as the container. * An understanding of the Glasswall CDR approach to neutralise polyglot threats in processed documents. * An overview of the progressive research conducted at Glasswall to address both polyglots and other future threats. ## Summary This research explores polyglot file structures and evaluates how Glasswall's CDR technology can mitigate their risks during file processing. Our initial focus is on image formats due to their simpler structures, allowing for rapid advancements in this research area. The primary goal of this project is to develop a universal approach to mitigating polyglots based on their core file type, detecting file type variations, and ultimately extracting polyglot elements - prioritized in that order. Here we examine different polyglot variants, their deployment methods, and additional obfuscation techniques, and also provide insights into how polyglots are actively created, offering valuable knowledge for threat mitigation. ## What are polyglots? The term polyglot in common terminology refers to someone who speaks/reads multiple languages. Therefore, someone can be a polyglot if they can integrate themselves into various cultures or social circles. In a technological sense, polyglots are files which can be interpreted as multiple file types whilst all residing in the same file system entry/memory space. This results in a file which, when opened in an image viewer, displays a picture, when run through Adobe Acrobat, displays a PDF document and, when run through a decompression tool, unpacks as a compressed archive. These files are possible due to quirks in the specification of various file types as well as the potential for interpreters of various scripts being exception-tolerant. A prime example of this is the PDF specification, which permits the header identifier (or magic bytes) of “%PDF-“ or in hex “25 50 44 46 2d” to be located anywhere within the first 1KB of the file’s memory space. Due to this tolerance in the header location, it opens the door to other file headers being introduced within the first 1KB or even stored entirely within the 1KB limit. Depending on the specification of the inserted file type, the PDF file can either be encapsulated within legitimate components of the new file type or simply appended to the end of the inserted file type, the result being that the file is now both a PDF and an inserted file type legitimately. ## Why are polyglots an issue for security? Most security systems, including Glasswall CDR, rely on interpreting the file type being processed correctly to know how to parse and inspect the file for specific or anomalous content. As polyglots rely on their structures being legitimate and therefore within specification of the host document, they are not considered corrupt or malformed, and so file identification systems will typically identify the file as the most “obvious” file type present. This is done by either examining the file extension, MIME type or magic byte values. However, as a polyglot can contain multiple legitimate magic byte entries/headers and a profiling system only looks for the most “obvious” file identifier, it can ignore potentially suspicious content. Let’s think of an example where a file looks like a PNG image file but is also a legitimate PDF file. The PNG is harmless; however, the PDF file contains auto-open commands and a highly malicious JavaScript payload. The security system determines the most "obvious" file type to be a PNG and so analyses the file as a PNG, finding no suspicious content, malicious functionality, or malformations. As the PDF code is contained within a comment text or iTXt block of the PNG, it is processed as a metadata text block which cannot be acted upon directly by an interpreter. As such, the comment content poses no active risk, and the content is deemed to not be a threat. However, upon delivery, the file has no extension and is opened in a web browser by default, which interprets the file as a PDF, triggering the internal payload. The above is a basic example of the risks associated with polyglots used for malware deployment, but the risks also extend to both infiltration and exfiltration of data. From a CDR perspective, the same applies, if we are not interpreting the file types integrated into a file we are not validating and applying content management schemas to all the present files. In the example above, the installed and legitimate PDF file will not be verified by CDR, and its existence in the PNG will not necessarily be sanitized depending on the current content management profile. ## Our objectives Our primary internal objectives for this project, in descending order of priority, are to: * Ensure Glasswall CDR mitigates/removes polyglots * Images * Documents * Archives * Executables * Detect present polyglot file types * Extract polyglot file sections into separate files for CDR processing As we are focusing on zero-trust architecture and there are limited legitimate situations where polyglots should be used, a default control to mitigate polyglots is the priority. This mitigation should aim to remove all elements in a file which can or (when scanned) are potentially being used to support a secondary polyglot file type whilst ensuring the “obvious” or primary file type is not corrupted as a result. This methodology is to be applied to image, office, archives, and potentially executable file types. As image file specifications are much more simplistic, they provide less scope for polyglot file injection methods relative to more complex structures. Therefore, image types were addressed first to prioritize security capability enhancement deployment into product systems. As the project progresses, more complex file types will be addressed, which will result in the resources required subsequently increasing and feature enhancements rates decreasing as a natural result. Detection of the installed polyglot sub-types is a secondary objective relative to the core zero-trust functionality our CDR prioritizes. Detection is a commonly addressed issue with several projects focused on this area, with mixed results highlighting the complexity of the problem. However, when detection is implemented, this information can be used as threat intelligence to help analysts determine what potential risks were being introduced and from where which can be invaluable. Extraction of polyglot sub-files in an advanced and purely optional third objective which would aim to leverage the results of the secondary detection objective to isolate the sub-type files and their locations. From this, using the identified header’s locations, we can determine the memory location within the polyglot hosting the relative file sections and extract them to a new memory location, resulting in the full export of the sub-file for CDR processing or secondary security analysis. ## Types of polyglots Polyglots come in a variety of types, reflecting the way the sub-type files are inserted into the primary/host file. These range from simplistic to extremely complex, with some variants only being supported by a very limited subset of file types. As such, not all polyglot variants are supported by all file types and are typically limited by factors such as header/magic byte offset location limitation, internal component capabilities and non-parsed segment availability. ### Stack Stacks are the most simplistic of polyglot structures and only require the file types to be “stacked” on top of one another. In practice, this is achieved by appending the data which makes up the sub-type file onto the primary file with the two files occupying the same file entry. This type of polyglot is limited to sub-types which do not have a header offset limitation or who’s interpretation is inverted (read from the bottom up), such as zip files. In zip files, the “header” or primary segment that is used to identify the file is located at the end of the file rather than the beginning. As such appending a zip file to most other file types will achieve a supported legitimate polyglot which is both a standard file and an archive. *Image 1 – Polyglot stack configuration* The primary limitation of these types of polyglots is that they tend to only have one sub-type file injected, unless paired with one of the other polyglot insertion techniques. Additionally, the appended sub-type file is typically added post the End Of File (EOF) identifier or final entry of the primary file. Although interpreters will allow the two file types to be handled legitimately, analysis of the file for the location of EOF specifiers can allow for quick resolution of this polyglot variant by truncating all data post the EOF. As this is the simplest polyglot variant, it also has the simplest solution. ### Parasite Parasites are more complex and involve the embedding of the sub-type file into legitimate components of the primary file type. This can be achieved in several ways such as taking advantage of embedded file capabilities or even injecting in metadata components which are not parsed/processed by most systems for active rendering. Regarding the latter, PNG or GIF are great examples where comment or text segments are outlined in the specification for the addition of metadata but are not actively used by almost any system. Using the example previously of a PDF embedded in a png, this is achieved via the parasite technique. The PNG has a tEXt segment specified near the start of the file, which is populated with the PDF file data. If the tEXt segment size is correct and the contents within the specified format of ASCII, the image file is not corrupted, and both the PDF and PNG can be interpreted from the same memory location. *Image 2 – Graphic, parasite hiding as part of an organism, avoiding detection* A more advanced example of this would be using legitimate components of a file to create an active and custom unused space in a file, such as creating an XObject in PDF. In a PDF, you can specify/create an XObject, which is not referenced anywhere in the document and, as such, is not rendered or processed on opening. The XObject can be populated with a stream, which can be made up of the contents of a sub-type file. One concerning implementation of this is to embed encrypted file containers into PDF[[9](https://glasswallsolutionsltd-my.Sharepoint.com/personal/cmorley_glasswall_com/documents/9.%09https:/mcfp.felk.cvut.cz/publicdatasets/pocorgtfo/contents/articles/04-11.PDF)]; by adapting the container to use the PDF header block as the salt and ensuring the verification block is at offset 0x40, the container can be embedded in the XObject stream at the start of the document without corrupting the PDF. As the XObject is not referenced it is not rendered/processed but is a legitimate component of the PDF specification and structure. As such, the file can be opened as both a PDF and decrypted to reveal a hidden encrypted directory/archive. ### Zipper Zippers are a much more complex version of parasites where the data of both the primary and sub-type files are embedded in the comment sections of each other. Rather than there being a primary file type and a sub-type, zippers work by using differing comment specifier values to nest the data blocks of each file within the comment blocks of the other (hence zipper). This is typically reserved for script files which can be interpreted by multiple engines and where the multi-line comment blocks can encapsulate the other script versions without needing to handle header locations or requires structural elements. *Image 3 – Representation of two items being enfolded into one another* ### Cavities Cavity-based polyglots work on embedding sub-type files into unprocessed memory space within a file structure. As an example, in executable files, null padded space can be referred to as a cavity or code cave which can house arbitrary data. However, the same principal can apply to other file types. Any space within a file’s memory space which can be populated with arbitrary data and is overlooked by an interpreter of the primary file type can be considered a cavity for polyglot injection purposes. *Image 4 – Representation of cavity, commonly called “code caves” which can be used* ## Sample collection Several databases of polyglot samples are available for free online and can be a good basis for initial sample collection. However, the samples will need to be analyzed to determine their polyglot type variation as well as assessing whether they are legitimate (accessible in all formats without extraction methods required for sub-type access) polyglots. The sources used for initial sample collection are: * [Source 1 - Tweetable polyglot png](https://github.com/davidbuchanan314/tweetable-polyglot-png) * [Source 2 - Polyglots database](https://github.com/polydet/polyglot-database) * [Source 3 - Polydet](https://github.com/polydet/polydet/tree/master/tests/samples) * [Source 4 - Polyshell (Zipper sample script)](https://github.com/llamasoft/polyshell) * [Source 5 - WEBP with MBR bootable segments](https://github.com/fandauchytil/webp-polyglot/tree/master) ## Sample generation - automated The samples which were within the sample sets and databases freely available were not exhaustive or terribly reliable in being true polyglots. Therefore, we are required to turn to sample generation via available polyglot creation tools or manual adaptation/creation of samples. ### Mitra [Source](https://github.com/corkami/mitra) This is one of the primary tools used to generate polyglots which are both true polyglots as well as mock or “near” polyglots. “Near” polyglots allow for overwriting of components of the primary file with the original content being encoded in other sections when the primary and sub-type begin at the same offset and, as such, are not considered true polyglots for what we are after. Mock polyglots are designed to insert the identifying markers of various file types (as many as can be supported by the primary) to help create detectable polyglots. However, mock files do not consider the practicality of the sub-type file data embedding and only account for the header identifier, so are also not useful for our purposes. This tool is good for handling some of the binary formats and will attempt to embed them into parasites and zips where possible but will fall back to stacks by default. If a file type cannot be determined by the tool’s parser, you can force polyglot the files which essentially treats the sub-type as a data blob and appends/stacks it to the file. *Image 5 – Mitra tools combination table for polyglot creation* ### PowerGlot [Source](https://github.com/mindcrypt/powerglot/tree/master) PowerGlot is a tool which can be used to a limited degree to embed script payloads into PDF and JPEG images. This has a limited use case but highlights how this can be achieved automatically. ### Metasploit bmp-polyglot generator [Source](https://github.com/rapid7/metasploit-framework/blob/master/modules/encoders/x86/bmp_polyglot.rb) Metasploit has an encoder module to embed shellcode payloads into BMP files. This can be used to generate, transfer and then deploy malicious shellcode to targeted endpoints. The generator script is in ruby; however, it appears to require a metasploit agent on the endpoint to interpret and extract the shellcode for local memory injection. This may not technically constitute a functional polyglot such as those we are looking for; however, the embedding technique and location may be used for other file types (depending on interpreter readability). ## Manual sample creation and vulnerable segment identification This section will outline how to generate polyglots for various formats in a manual way, this will focus on parasite and where possible zipper and cavity methods. Stack polyglots only require the appending of the sub-type file to the primary file type and so will not be covered in detail. For each primary file type I will, over the course of this research, identify where sub-types can be supported and provide sample structures (images) to demonstrate their utility. ### Images #### Universal technique – Pixel data injection The simplest method an attacker has when dealing with image types is to inject the sub-type file data directly into the image data itself. This can be achieved extremely easily but results in corruption of the image data with clear evidence of suspicious content to a human user or even potentially statistical analysis. As the sub-type file is not injected into ancillary or non-functional elements, it is arguable if this is a polyglot method. However, if both file types can be opened, it may be moot from an offensive perspective depending on the attack vector. If the image is intended to be viewed by end users, pixel corruption may be a constraining factor. However, if the polyglot primary type being an image is to bypass type limited restrictions to deploy a sub-type file to a restricted area, the image data corruption is irrelevant. For example, a web portal which allows public file uploading of a specific image file type, which are then hosted on the server, is a prime target for this. By creating a polyglot to match the primary type of the image format required, an attacker could upload sub-type script files such as PHP or a PDF file to be opened on the server via we- based attacks which alter the type determination of the hosted file (or bypass the extension of the file) to achieve malicious payload execution. If this is the case, detection of such sub-types would require the data block would need to be checked for indicators of secondary file structures such as PDF, script blocks and even things like ISO. *Image 6 – BMP with PDF injected in image data, resulting in pixel corruption along bottom of image* Mitigation of this type of sub-type injection is problematic to resolve due to the key CDR requirements to retain semantic/visual integrity of the file post-processing. Traditional parasites are confined to specification-defined elements with data types and sizes, which can be leveraged for injection, but injection into the pixel data itself means we cannot simply remove the stored data as we would with other segments. Instead, we must either scan the contents of the pixel data for identifying markers or employ image-wide pixel alterations to achieve destruction of the sub-type document. In regards to the latter method, Glasswall has developed a steganography mitigation system which is tailored to achieve this result[[13](https://docs.glasswall.com/docs/steganography-smudging-the-invisible-ink)]. The images below demonstrate the results of this system on the bmp image above in achieving sub-type PDF file destruction. *Image 7 – BMP image data with PDF injected into pixel data stream* *Image 8 – BMP image with PDF pixel data embedding, cleaned after steganography mitigation* ### Universal technique – Palette data injection In simple images, such as logos, the image may be stored as a palette image instead of a TrueColour image. In TrueColour images such as image 6, each pixel has typically 3 bytes assigned to it which represent the RGB channels which determine its ultimate colour. In palette images each pixel instead only contains 1 byte which is an index value of the colour palette. Standard palettes have a maximum size of 255 colours which aligns to the 1 byte index value, meaning the image only has a colour range of 255 colours. There are images which support larger palette sizes but the concept remains the same, by having a pre-et range of colours rather than a dynamic per-pixel value the required size of the image file is reduced but equally so is the colour depth of the image. When creating polyglots, akin to direct pixel data injection, malicious authors can inject content directly into the palettes of images to store sub-type content. Depending on the size of the palette, this can support script content injection or even potentially complex sub-types also such as PDF. By altering the palette data, the pixels which are indexed to the altered entries will be affected making visual detection of corruption by a human user extremely likely. However, as with direct pixel injection if the attacker does not intend for the file to be viewed by a human this visual corruption may not be an issue. As an example lets use a PNG file using a 255 byte palette, from section 11.2.3 of the PNG specification this requires a PLTE chunk addition made up of 3 byte entries for RGB. By creating a cover PNG image we can find the PLTE entry in the files structure. *Image 9 – PNG file using colour palette PLTE(Red) entry values(Green)* If we change the palette data contents to contain a malicious php script, the rendered image is affected but not corrupted. As the palette contents are accepted as long as the palette entries are the correct size (number of entries x 3), manipulation of the file in this way does not invalidate the file or the visual components. *Image 10 – PNG with PHP(Purple) injected into PLTE(Red) palette(Green)* This example has injected a PHP script which will attempt to read a file and dump its contents. At the beginning of the script we use a multi-line comment end specifier to link with a previously injected multi-line comment start injected earlier in the file to ensure the script runs as intended. Mitigating this type of injection has the same problems highlighted for pixel data injection, but it also has the same solution. Glasswall’s steganography mitigation tool can destroy these types of polyglots in palettes by employing palette shuffling which randomises the palette colour entries. By relocating each palette defined colour to a new index, this corrupts the required UTF-8 character sequence for the php script in this example but would also corrupt the file structure of any sub-type file in the images palette. Below, image 11 demonstrates this effect when the malicious file has been run through the Glasswall CDR process including the steganography mitigation via palette shuffling. *Image 11 – Palette PLTE(Red) chunk in PNG image after CDR and palette shuffling(green)* #### PNG ##### **tEXt or iTXt** Sections 11.3.4.3 and 11.3.4.5 of the PNG specification outlines the uncompressed ancillary text chunks which can be used to store arbitrary textual information. Depending on the keyword, the text can be used to achieve ancillary processing based on specific rendering software. Regarding keyword “comment” for example, the textual information is limited to the maximum chunk size of the file and must be ISO-8859-1 character set compatible for text and ISO 10646-1 for iTXt but otherwise permits any content. The tEXt or iTXt chunk is not actively interpreted by most image rendering systems and is ignored, with the focus instead being on IDAT chunks. As these chunks can be embedded anywhere within the file, although recommended outside the IDAT block, they can be used to embedded at the start or end of a document a sub-type file based on the sub-type’s requirements (header offset etc.). *Image 12 – tEXt chunk definition at start and end of document, PHP payload with image data encapsulated in multi-line comment sequence* From the example above, we have embedded a PHP script in the PNG file under two different tEXt chunks. The first chunk is located just after the IHDR section and ends with the multi-line comment start symbol “/”. The second comment is located just before the IDAT block and consists of the multi-line comment end symbol “/” and the closing of the PHP block. Essentially, this achieves the commenting out of all the other binary and metadata of the primary image file, which can potentially cause PHP interpreter issues. Running this file results in the correct echo printout using php.exe with only minor errors after the execution of the block we specified. It is worth noting that in most cases with script injection the interpreter will throw exceptions, however if the primary action/purpose is achieved, the script deployment can be deemed a success. ##### **Custom chunk (cOMM)** From section 5.4 we are provided a comprehensive outline of the primary and ancillary chunk specifications. From this, we can see that ancillary chunks are essentially ignored as standard by PNG rendering systems and are instead processed only by specific rendering systems, essentially making them unrestricted metadata chunks. Due to this, we can add any ancillary chunk we like with any name if it adheres to the requirements outlined in section 5.4 for chunk naming. As such, we now have the capability to add any ancillary chunk, under any name, to any section of the PNG file we like with custom content. Section 14 of the specification outlines the limitations of custom ancillary chunk locations which are, in summary, not to be placed in a location corrupting ihdr data or between IDAT chunks. *Image 13 – Custom chunk with keyword “cOMM”* In the example above, we can see the addition of the “cOMM” object at the start of the PNG document before the IHDR, which is populated with 508 bytes which makes up the sub-type PDF document. As the ancillary block is ignored by the general interpreter, the PNG still renders correctly and when opened in adobe the sub-type PDF document is displayed correctly achieving a true parasite polyglot file. ##### **IDAT compress data padding** Research by Dabid Buchanan[[1](https://github.com/davidbuchanan314/tweetable-polyglot-png)] revealed a quirk in the way that PNG interpreters handle the compressed stream (deflate compression) within the IDAT chunks of PNG images. This is because, due to the use of Huffman deflate coding, the size of the decoded blocks is not determinable by the external chunk specifiers and can only be determined during decoding. As the decoding operation processes the IDAT chunk contents until the decode operations reaches a block with flag “last block” (BFINAL – RFC-1951) set and its subsequent End Of Block (EOB) marker (256 – RFC-1951), without decoding, it is not possible to find these flags. As such, IDAT blocks containing deflate encoded blocks can be appended to without corrupting the image allowing injection of sub-type file content into its legitimate structures with the only change needed to be the chunk size specifier to incorporate the new data. The image will still render correctly, and the injected data can only be identified as such during an analytical decompression process, which would check if the whole chunk is decoded or not. *Image 14 – IDAT chunk with zlib header 0x78DA (Red), followed by decode block header 0xED (Yellow)* *Image 15 – zlib decode block header 0xED binary value, highlighting set of BFINAL in first block* #### GIF ##### **Comment section** GIF supports a comment segment as outlined in GIF specification section 24. The requirement for such a segment is that it has the extension introducer field 0x21, the comment label (tag) value 0xFE, the comment section size specifier and finally the comment data. This looks like: *"21 FE <size in hex, 2 bytes> <comment data in sub-blocks of 255 bytes encoded as ascii 7-bit>"* *Image 16 – GIF comment block “21 FE” with PS1 script file sub-type* The data must be in 7-bit ASCII and is not to be used for controller information for custom processing. Of the specification, there are two primary points of interest for use for introducing polyglots which are: *“This block is OPTIONAL; any number of them may appear in the Data Stream.”* This indicates that there can be multiple comment sections in one GIF file. This can be important for handling script file injections where binary data may need omitting (see Image 9). The second important part is: *“Position - This block may appear at any point in the Data Stream at which a block can begin; however, it is recommended that Comment Extensions do not interfere with Control or Data blocks; they should be located at the beginning or at the end of the Data Stream to the extent possible.”* This outlines that these blocks, which are ignored by decoders/interpreters, can appear at any point in the document. It recommended that they appear outside the block of the data stream but not a requirement. This means that they can be injected at any point in the files structure making them harder to find and capable of being used for more advanced parasite polyglot injection (PDF XObject scattering e.g.). *Image 17 – GIF with comment segment “21 FE” with PDF header content sub-type* ##### **Logical screen descriptor manipulation** The Logical Screen Descriptor (LSD) of a GIF file specifies information about the image pixel size, colour tables and other such information. As with other formats, malicious actors can manipulate this to inject character sequences such as script block comment segments. In the example below, the logical screen descriptor width is altered to “0x2F2A” which in UTF-8 is “/*”, this is the multi-line comment operator for JavaScript. *Image 18 – GIF LSD width size alteration to achieve command block char injection for JS* In this case, the image files data/pixel information was modified so that the LSD aligned with the stored image information. As such, the LSD is legitimate relative to the image data and is not considered corrupt or malformed. Using this method, an actor would be able to run this image as a JS script file and comment out the data content of the image file leaving only the JS content to be executed whilst retaining a legitimate GIF image file structure. *Image 19 – GIF executing as a JavaScript in fiddle* #### JPG ##### **JFIF segment (metadata)** JFIF is the JPEG File Interchange Format which can be used to store metadata about JPEG files as part of the APP segment structure. This data can include thumbnail data as well as, potentially, a JFIF extension (JFXX) segment or extension data which can be used to hide arbitrary data. As these elements are used to store secondary information, which is not a core requirement for the image itself but instead normally for vendor or platform specific handling, as it allows for manipulation/injection whilst retaining the images integrity. Due to the arbitrary nature of this embedded metadata element in most rendering systems at the max size of 0xFFFF, removal should be considered a standard operation toc mitigate risks here for polyglot injection. *Image 20 – JFIF content in APP0 segment* ##### **COM segment (comment)** Within the JPEG specification ISO-IEC-10918-1 under section B.2.4.5, the comment syntax outlines the configuration of the comment segment, which can exist within a JPEG file structure. *Image 21 – JPEG specification B.2.4.5 for comment segment configuration* Specification adherent comment sections start with segment identifier “0xFFFE”, which is specified in the marker code assignment table B.1. *Image 22– JPEG specification table B.1 outlining segment marker values* The specification does not outline how many comments the document type can support/allow or where they are located within the document. This means that, in practice, these comment segments can be injected anywhere between data/essential segments of the file permitting their use for polyglot parasite creation. As these can be injected at the bottom, in the middle or even very close to the beginning of the document, this supports most file types which have non-zero offset header requirements such as PDF, script payloads blocks, compressed file types and even ISO/executables. The example below demonstrates a PHP script injection within JPEG which allows the file to be opened as both a JPEG image and executed through a PHP interpreter (such as php.exe): *Image 23 – JPEG COM segment with sub-type PHP script file polyglot injected (Red = COM identifier, Yellow = COM segment size, Purple = COM data)* ##### **No APP0 JFIF comment** In some examples, JPEG files can be created without APP segments, which are typically reserved for hosting the JFIF or EXIF structures. This is highlighted by the research done by Ange Albertini[[2](https://github.com/angea/pocorgtfo/blob/master/releases/pocorgtfo03.pdf)] this is useful as some interpreter for other file types, such as PDF, specifically check the byte sequence of the headers to determine if the file is a PDF or a JPG. In these use cases, the systems check for the JPEG SOI “0XFFD8” followed by a segment identifier “0XFF”. If this pattern is encountered, the interpreter will open it as a JPEG and not a PDF, regardless of whether a PDF header is present in the file. Due to robustness built into several JPEG interpreters, however, it’s possible to include the JFIF segment without including an APP0 segment specifier. In the example below, the APP0 segment specifier is removed from a file whilst retaining the JFIF content and a comment segment added immediately after the segment containing PDF data. The PDF data specifies a stream start, which effectively means that the contents of the JPEG will be ignored by the PDF interpreter as part of a stream content if rendered as a PDF until it hits the “endstream” keyword. As the JPEG pattern has been avoided and the PDF data is included in a comment segment, the creator avoids the PDF interpreter forcibly interpreting the file as a JPEG. *Image 24 – JPEG file with APP0 segment identifier removed and PDF data contained in COM segment. (Red = Missing APP segment identifier, Yellow = Segment size, Green = JFIF identifier)* It would be expected in this use case, due to the expected nature of the APP segment from its specification, that this JPEG would fail to open as an image. However, as interpreters are built to be robust, they in most cases instead simply assume an APP segment will be present and check for the segment identifier and continue without issue. This allows for a JPEG file to subsequently host a PDF document without running afoul of the PDF interpreter safeguards to prevent this. #### SVG ##### **Ancillary elements (custom, metadata, description)** SVG files are essentially just XML files which are interpreted as images to achieve a functional polyglot from a parasite method we can employ arbitrary content element injection. Elements which can achieve this include expected file type specific elements such as "<metadata>" or “<desc>”. Additionally, data can be injected using custom field elements (in alignment with SVG specification section 5.10 “Foreign namespaces and private data”) or via appropriate commenting in-line with standard XML structure requirements (section 2.5 of XML specification 5th edition). All these methods are legitimate structure components which are permitted by interpreters. This means that parasite polyglots for SVG are extremely easy to implement as the header for SVG’s is relatively small and simplistic when compared to other file format types. This is demonstrated below with the creation of a custom element “<customField>”, which hosts a PS1 command line. When run as a PowerShell script file it executes successfully: *Image 25 – Custom element injection into SVG image block of HTML file structure* Research by Mauro Gentile at Minded Security conducted in 2015[[3](https://blog.mindedsecurity.com/2015/08/pdf-based-polyglots-through-svg-images.html)] investigated specific use cases where SVG is used to polyglot malicious PDF files to achieve misinterpretation and execute malicious payloads. This was achieved via SVG uploads without CDR or structural verification allowing hosting of PDF files which could subsequently achieve JS execution or file system access via PDF forms. This was duplicated in our test set creation, both using a standalone custom element and mixed and split across the file using custom elements and comment blocks. *Image 26 – SVG with PDF sub-type injection across both custom field and comment block* #### BMP From a polyglot construction perspective, BMP is a limiting choice of base file type due to its extremely curated structure and disregard and exclusion of all comment and vendor-specific fields. BMP was conceived and developed by Microsoft, who kept the original design limited to only structural definitions of image data without scope for custom renderer handling using ancillary metadata. Later versions added some metadata support, but large or textual support is not included in their specification. Custom versions of BMP can exist, but they are not covered here. The specification for BMP is bundled under the WMF specification in the Windows documents. [[14](https://winprotocoldocs-bhdugrdyduf5h2e4.b02.azurefd.net/MS-WMF/%5bMS-WMF%5d.pdf)][[15](http://www.dragonwins.com/domains/getteched/bmp/bmpfileformat.htm)] From this, we can ascertain that polyglots using a BMP base typically are only viable via the stack creation method due to the strict nature of the specification. However, some complex injection methods can be included with limited viability. ##### **Segment padding** One of the few ways to achieve parasite polyglots with BMP is to abuse some of the file definitions to create spaces (cavities) in the file, which can potentially be leveraged to store arbitrary content. The interpreter will ultimately ignore the created space as it exists outside the required data size & location and so can contain any data. This is a much more involved process and can rarely be automated. To do this, the header information is adapted so that gaps are created between the required sections, which most interpreters will ignore. These gaps can be any size relative to other required elements (such as colour tables) and unused space between the original end of the document and the maximum 4-byte value of 4,294,967.295 bytes. This can allow the injection of complex file types such as a PDF. *Image 27 – PDF injected into BMP segment padding (Red = BMP type specifier, Yellow = File size, Green = Offset to image data location, Purple = Image Header, Blue = Padding data between Image Header and data, PDF injected)* ##### **Header injection** Another curious methodology, outlined by researcher “NuclearFarmboy”[[4](https://malicious.pro/post/python_polyglots/)], leverages how as the header information can be any value we desire, we can manipulate this to include ASCII values of our choosing. This is valuable as it can allow us to inject comment ASCII character sequences to embed script blocks without encountering errors during execution. In their work, they adapt the BMP file size specifier at offset 0x04 for 4 bytes to a size which corresponds in ASCII to variable definition and the comment characters for python scripts “=4;#”. The python script itself is then appended to the bottom of the file achieving a parasitic stack, where the injected comment sequence allows a script interpreter to successfully ignore the entire image data block by first assigned the value of 4 to variable BM, delimits the line then adds the comment initiator “#”. This results in full execution of the subsequent script at appended to the file whilst retraining the file as a legitimate image file. *Image 28 – Source “NuclearFarmboy” [[4](https://malicious.pro/post/python_polyglots/)] research demonstrating injection of python comment block code “=4;#”* ##### **Size overload** This section will focus on injection into BMP version 2 or lower via data segment size overloading and injection. To do this, we must manipulate the image data section at the beginning of the file to later the data segment sizes, which inevitably affects the generated image. In a simple monochrome image, this can be achieved by altering the width and height specifiers in the BITMAPCOREHEADER, which is specified in section 2.2.2.2 of the MS-WMF specification. An identifier that a BMP is version 2 is that it uses the BITMAPCOREHEADER without a BITMAPINFOHEADER object being present. *Image 29 – BMP header with core header pixel width (Yellow) and height (green) highlighted* By understanding the equation used to calculate the image data size, outlined under section 2.2.2.9, we can manipulate the image data size to the one we desire. With monochrome images this is simple as bits per pixel is 1. Equation 1 = Size in bytes when compression value is BI_RGB. BI_BITFIELDS or BI_CMYK By doing this, we can extend the data segment to hide our sub-type data. This does amount to image data tampering and will affect the original image. In the example below, we alter the width value from 0x0D00 to 0xE000 and using the equation changes the image data size from 28 Bytes to 196 Bytes. This creates additional space in the image pixel data which we can then populate with our sub-type file, in this case a ZIP file. This corrupts the original image content but creates a specificationadherent BMP file with embedded ZIP, which works as a parasitic polyglot. *Image 30 – Original BMP image* *Image 31 – Image data expanded with ZIP data BMP* *Image 32 – BMP image data width (Yellow) value altered from 0x0D to 0xE0 to store zip file in image data padding* #### TIFF ##### **UNDEFINED content** TIFF poses a particular challenge when dealing with parasitic polyglots due to its support post TIFF 6.0 for arbitrary field contents with type “UNDEFINED” in the Image File Directory (IFD) (page 16 of TIFF specification). In practice, this allows for an IFD to specify segments of the file which can have arbitrary content which fully supports all sub-type files. Additionally, due to the way TIFF works, these segments can be located anywhere in the document and their associated size and offsets specified only in the IFD. Due to these factors, it is relatively easy to add sub-type file data to an existing TIFF file either at the end or the document or between existing segments by modifying the IFD. The latter method requires manipulation of the existing IFD entries that have offsets after the injected entry, as their offsets will need to be recalculated based on the injected data size, but this can be achieved to result in a fully integrated polyglot. In the examples below, we have inserted a PDF to the start of the document and a ZIP file to the end of the document. In the case of the PDF, the IFD entry specifies the undefined data segment to start at byte offset 0x8 and therefore well within the strictures of the PDF header allowance of 1KB. *Image 33 – PDF data stored within TIFF structure at offset “0x08”* *Image 34 – IFD entry showing tag 0xDDDD (Red) which is custom, 0x07 type (Yellow) “undefined”, size 0x8BC9 (Green) segment at offset “0x08” (Purple)* In the case of the ZIP, whereas this would typically result in a stack polyglot, we have added an IFD entry to the end of the document to encapsulate the ZIP file segment, effectively changing a stack polyglot into a parasite by adding an entry to the IFD. As the EOCD of ZIP files can be within the last 65,547 bytes of the file to account for the potential comment size in the EOCD, there is considerable scope for injection. As such, by adding an undefined segment to the end of the file or anywhere within this memory range, we can insert the EOCD within the defined segment with TIFF EOF padding and still achieve integration of the archive structure. *Image 35 – IDF entry showing 07 type “undefined” segment at offset “0x5850” in Little Endian, PK header indicating ZIP sub directory data beginning at offset “0x5850” immediately after IFD in orange* In both cases, as the sub-type file data has been injected within a component of the TIFF IFD structure, they are considered legitimate components. Undefined segments are within specification and are typically used for vendor specific image handling software. ##### **Padding data** As TIFF relies on the IFD to specify image file segments including information on their location (offset) and size, it creates an easy mechanism to create gaps/cavities in the file. As an example, if we change the offset of the last entry in the IFD to increase by 100 bytes and move the contents to this new offset, the 100 bytes introduced between the final and penultimate entry is now a cavity. As a TIFF interpreter only processes elements within defined IFD segments, the gap is ignored and is also not in breach of the specification. This means that we can create any cavity size we want which can be populated with arbitrary data within a TIFF file by manipulating the IFD to facilitate sub-type file injection. *Image 36 – IFD of TIFF file, offset to image data value with tag 0x111 (Cream) is 2 entries from the end. Indicates image data begins at offset 0x20E, creates cavity between header IFD offset and data start of 0x206 bytes.* *Image 37 – PDF data inserted immediately after the TIFF header specifying the initial IFD at offset “0x02CC”, in padding space between header and strip data at “0x020E” in orange* #### *WEBP* ##### **Chunk addition** WEBP image files are created in a Resource Interchange File Format (RIFF) which is like a TIFF file construct with chunks/segments of data which refer to each other. Instead of using IFD’s, RIFF’s chunks specify their individual sizes only and the interpreter reads them in sequence using the size specification and chunk identifiers to determine what to do with each chunk and when they end. In the case of WEBP, if we are using an Extended File Format (VP8X) which accounts for the inclusion of fields such as EXIF or vendor specific metadata, we can add “unknown chunk” types. This means that FourCC, or chunk identifiers, outside the core requirements of the WEBP format are permitted when using VP8X. Effectively, this allows the specification of RIFF chunks of vendor-specific ancillary data to be added which are ignored by interpreters and can contain arbitrary data. As with all other formats, ancillary data segments allow the injection of any sub-type file we want if that sub-type files header locations requirements are met, and the encoding is within the constraints of the ancillary segment. The use of RIFF chunks in this case allows us to add the ancillary data at any location in the file desired if the defining RIFF chunk (header) is located at offset 0. This means we can use these ancillary chunks to injects both top offset and bottom offset sub-type files such as PDF and archive formats (ZIP). In the example below, as PDF is injected into a WEBP image by adding a “HACK” chunk to the WEBP RIFF structure immediately after the header, succeeding in creating a functional WEBP-PDF polyglot. *Image 38 – WEBP RIFF chunk “HACK” containing PDF sub-type file* ##### **OP code and script code injection** WEBP header identifier “RIFF” has the benign opcode translation of: * R = 52 = push dx * I = 49 = dec cx * F = 46 = inc si * F = 46 = inc si Researcher Fanda Uchytil[[5](https://research.h4x.cz/html/2023/2023-08-08--webp_polyglot_i-bootable_picture.html)] going by the handle “H4X” leveraged this benign capability and the customisable nature of the RIFF container size (following 4 bytes) to create a bootable WEBP file or WEBP-MBR polyglot. This is done by adding a “call”, “jump” or “loop” opcode in the RIFF size (opcode = “E8”, “E9” or “E2” respective) and a location specifier immediately following to a RIFF chunk containing the Master Boot Record (MBR) of the hidden payload. In their documented use case, they decide to leverage the “dec cx” opcode in the header which decreases the cx register (which is a loop counter) by 1. The opcode “E2” specifies “LOOP”[[16](http://ref.x86asm.net/coder64.html)], which checks the cx register, and if the value is not 0 is jumps to following specifier location. As the cx register will likely be reset in most systems to 0 on load, or in other systems loaded with a random value is not reset, the odds of cx register starting at count 1 are very low. In the example created for external evaluation, they used a “NOP” 0x90 code and the “JMP” 0x1F[[16](http://ref.x86asm.net/coder64.html)] operator to simply things. If the MBR signature “0x55AA” is located at byte 510, a boot loader will load the 512 bytes preceding it into memory at 0x7C00. The containing code is then executed, with the JMP code in place the MBR will skip over the WEBP and VP8X chunk and straight to the MBR data stored in the HACK chunk. *Image 39 – MBR inserted into chunk “HACK” within WEBP file. 0x90 “NOP” and 0xE9 “JMP” code (red) inserted as WEBP size to make MBR code block(purple) compatible with RIFF header placement and ignore header (grey)* The exploit is heavily involved but does provide an example of how offensive shellcode can be injected into such a file with actionable consequences such a boot capability. Another article[[6](https://research.h4x.cz/html/2023/2023-09-01--webp_polyglot_ii-script.html)] by the same researcher highlights how the RIFF size parameter can be abused to insert comment specifiers for various script block types akin to Image 27. In one example, they adapt the 4-byte size specifier to “ =’ “ to capture the WEBP and VP8X chunks as string values of variable “RIFF”. This allows the WEBP file to be run as several different script types including RUBY or even Python if the script block has the appropriate termination command within it. *Image 40 – WEBP RIFF size adapter to “ =; “ with RUBY script in chunk “HACK” containing string terminator and appropriate script terminator “exit(0)”* ### Metadata structures There are defined structures for supported metadata types, which can be installed within certain file types. These metadata structures allow for more complex metadata inclusion, which is typically important for files which may rely on this data for rendering or post processing image or video files. However, these metadata structures are also commonly supported by office document formats as well. As these structures are designed for metadata storage, they are tailored towards vendor specific or detail/text data storage, which commonly permits arbitrary data storage with the expectation that the rendering software will handle/validate or ignore the content as appropriate. This loose restriction on content can make these metadata structures a prime target for sub-type polyglot file injection. #### EXIF Exchangeable image file format (EXIF) is a commonly used metadata format across several file formats, which can be used to store arbitrary data in several formats. As EXIF is essentially a TIFF database, it is relatively easy to alter its contents in a live file without resulting in a corrupt file. As EXIF is largely ignored by interpreters during rendering or supported in a format to specifically store vendor-specific data, custom fields and data entries are expected and overlooked for compatibility reasons, making this a prime target for sub-type file injection. ##### **Ancillary fields** EXIF specification allows for specific data sections to be added to the EXIF structure to be used as metadata, each of which supports different data types. Three of the primary types are “Image Description” outlined in Table 37 of the EXIF specification DC-008 and “MakerNote” or “UserComment” outlined in Table 7. The specification for EXIF is extensive, and so this is not exhaustive. However, the field types outlined are valuable as they allow for arbitrary data content and can be populated to “ANY” size, typically not exceeding 64KB in theory and 1KB in practice. *Image 41 – Table 37 of EXIF specification outlining “ImageDescription” field key* *Image 42 – Table 7 of EXIF specification outlining “MakerNote” and “UserComment” field key, content type and size* By using tools such as “exiftool”, the injection of an EXIF into the APP1 data segment of a JPEG image file as an example is trivial and can be populated with any data type we want. As the EXIF segment does not have a size specifier, this means that we can in practice use the tool to create the EXIF segment, find the populated field and the copy and paste any malicious data we would like via tools such as a hex editor. In the example below I have inserted a PDF file into a JPEG via the “UserComment” segment of the EXIF by adapting the TIFF, achieving a parasite polyglot. *Image 43 – Example of EXIF “UserComment” field content being replaced with sub-type polyglot file* ##### **Segment injection** As EXIF is a TIFF structure, it introduces all the same risks that a TIFF file permits regarding polyglots such as TIFF IFD manipulation for segment injection or padding. As EXIF is treated as metadata and not actively processed, there is no detrimental effect to the host file if it is a legitimate TIFF structure. As an example, WEBP RIFF format with VP8X allows for EXIF to be added anywhere within the file after the header. As such, the ease of manipulating the EXIF structure creates a prime target for sub-type injection. The WEBP specification also outlines that interpreters should only process the first encountered EXIF chunk, all subsequent chunks are ignored and so have higher impunity. *Image 44 – EXIF chunk added to EOF to host ZIP file successfully* #### XMP injection Extensible Metadata Platform (XMP) is a metadata structure developed by Adobe which can be used to store arbitrary metadata within specific file types. Supported images are outlined under section 1.1 of the XMP specification part 3. Two specific things to note on the format of XMP is that Note 6 under section 6.1 of the XMP specification part 1 outlines that: “*Users of XMP are allowed to freely invent custom metadata*” And section 7.3.3 attests: “*For the x:xmpmeta element, XMP processors may write custom attributes. Unknown attributes shall be ignored when reading.*” From these two points we can assert that XMP allows for fully custom fields to be created within the structure of the XML and that additions to the “xmpmeta” elements in the XMP, which are not of known type, will be ignored but are not outside specification scope and so will not cause corruption/rejection. ##### **Content substitution** As XMP is for metadata storage and expects XML, anything that is not in this format or is not named correctly is ignored. Therefore, XMP can be populated with anything in UTF-8 format and if it is not XML the interpreter just ignores the content entirely. This opens the door to content injection of sub-type file contents effectively anywhere within the XMP structure. ##### **JPEG** JPEG supports XMP as an APP data segment. It is the only segment which can have a repeated ID of APP1, which is used for both XMP and EXIF. This configuration of XMP allows for a segment data size of up to 65.4KB which is more than sufficient for most sub-type files. In the example below, an XMP has been added at the start of the document which hosts a PDF file: *Image 45 – XMP segment hosting PDF sub-type supported file* ##### **GIF** GIF XMP injection is achieved via Application Extension additions with the XMP data added as application data sub-blocks. Each sub-block has a format of a size specifier byte and a following data block with a size limit of 255 bytes. As such, every 255 bytes, a new block is required which also required a size specifier. This necessity of a value specifier can cause issued depending on sub-type file injections, which may not tolerate such values. However, in practice, the only requirement is that the sub-block size specifier encountered after each block is not 0, which can be achieved with minor alterations or, in most cases, needs no alteration at all. Once the file has been injected, the author must find the final sub-blocks size specifier and pad the content to the end of the sub-block to ensure it is conforming to specification. The “magic trailer” required at the end of XMP an Application Extensions is a sub-block closing mechanism and will handle the rest by forcing alignment. In the example below, an XMP is created at the beginning of the file with a PowerShell script, to make sure this executes as expected, a comment chunk is added to the end of the document to achieve commenting of the data block. *Image 46 – XMP chunk injection into GIF hosting PS1 script block working polyglot* ##### **PNG** XMP in PNG is installed as a chunk with type “iTXt” using the keyword “XMP:com.adobe.xmp” for identification by parsers. As this follows the same rules as any other chunk, there is a 4-byte length specifier for the chunk (excluding the length, type, and CRC), meaning it can have a maximum size of 4.29MB. *Image 47 – XMP inserted into iTXt chunk containing functional PHP script polyglot* ##### **TIFF** TIFF adds XMP as a segment in the primary IFD of the file under the tag value “0x02BC” with a filed type of either undefined or byte, which allows for arbitrary data hosting. The only unique identification of the XMP in this case is the tag value used for the IFD segment entry, which can be searched for. Interpreters use this tag to understand that the content should be treated as XMP XML, if the interpreter does not handle XMP or the content is corrupt, it is ignored by the interpreter. *Image 48 – TIFF IFD entry with tag 0x02BC (in LE) pointing to begin at offset 0x0160 (in LE) containing XMP and embedded PDF sub-type document* ##### **WEBP** Adding XMP or EXIF to WEBP is achieved by altering the WEBP format to the extended format “VP8X” and altering the extended file format flags so that bit 6 is assigned a “1”, indicating XMP metadata presence. The metadata is then added as a chunk to the file with the chunk header or fourCC value being “XMP “. The chunk is then populated with the UTF-8 encoded XMP metadata, which is interpreted by appropriate systems or ignored by incompatible systems. *Image 49 – XMP chunk addition at top of file after VP8X chunk, contains active PHP script* ##### **SVG** XMP is added under the “metadata” field in the XML structure of SVG files which we know is already an arbitrary data location in this file type. As the XMP is just a standardized extension of the metadata handling using specified fields and having capacity to link out to default field values (title, description), there is no additional context XMP provides for exploitative purposes beyond that of metadata field handling, which is already overly broad. *Image 50 – SVG file with XMP metadata hosting live PHP script block* ### Document formats Polyglots using document type files as base formats are more complex due to the wider range of available features and internal structures, which can potentially be exploited for sub-type injection. #### PDF ##### **Metadata stream XML (default and custom)** Metadata can be added to a PDF document by creating a “Metadata” stream type XObject. This XObject has a type of “Metadata” and subtype of “XML” outlined under section 14.3.2 of the PDF specification. These streams can be attached to the document’s primary structure as additional information or to individual objects. When attached to objects, metadata can be referenced within the document structure making them actionable elements/data which make up a specific functionality in the rendering of the document. As the metadata content of the stream must only adhere to the XML grammar and XMP specifications, as with other uses of XMP, it allows for arbitrary content injection. There are some keywords which are associated with metadata attributes highlighted in section 14.3.3, which must be avoided, however XMP allows for custom attributes to also be added. This means attackers can add any content they want, if the character set is within the XML/XMP specification limits, and it will be perceived as acceptable metadata. *Image 51 – PDF metadata stream with custom “nonsense” attribute hosting ps1* *Image 52 – PDF metadata stream hosting PK zip file within “creator” attribute* ##### **Metadata document information dictionary** In older versions of PDF, following section 14.3.3 of the PDF specification, document level metadata could be defined in a document information dictionary. In later versions, this was deprecated for all metadata besides “mod date” and “creation date”. However, it can still be added and interpreted correctly by most rendering systems. Multiple fields in this dictionary take “text string” as the data type again allowing for arbitrary data injection. *Image 53 – PK zip injected into “Author” Information Dictionary* ##### **Textual embedding (body opacity 0, page colour match, overlapping)** One of the most logical methods to embed a secondary file in a textual document is to embed the hidden data as part of the content of the document itself. This can be achieved relatively easily in PDF as page content can be overlaid via page pixel reallocation for current printing and then changing the content such as with text via the Tj command operator, to be fully opaque (via saved graphic state) or matching text colour to the page background colour as some examples. Another method is to hide content behind other object via occlusion, such as overlaying, which works by overlapping objects via co-ordinate jump operators (such as Td). This works on the premise that page content information is context-based as is hard to spot as being malicious from an automated perspective. By occluding the content, the author aims to hide the visual page content from the human user, so they are not aware of the sub-type content and, as such, do not determine something to be amiss. As the content of the page is within the specified definitions of how page content, such as text, can be defined, even if they make no logical sense, they are permitted as being within specification and allow potentially malicious sub-type injection depending on where the page conte XObject resides in the file. *Image 54 – Powershell script block injected into page content text object (BT), aligning text to location (Td) and then altering text colour fill to white (rg)* *Image 55 – Zip injected into page content text object (BT), aligning text to location (Td) and then altering text colour fill to white (rg)* ##### **Unfiltered embedded file** Embedded files can allow files with structures which do not require 0 offset or script blocks of exception tolerant languages (or can be paired with comment block character injection) to be injected arbitrarily in the host PDF. The PDF specification section 7.11.4 outlines the requirements for defining embedded files which are included via “EF” or ”EmbeddedFiles” entries. Embedded files general information shows that although a “Length” for the embedded file must be defined as a standard stream parameter, “Filter” (which specifiers compression method), on the other hand, is optional, meaning the embedded file can be inserted without alteration/compression. This allows the embedded file to maintain all is key characteristics, such as Magic Byte identifiers, segment definitions and even clear text content in some formats. As two easy examples, we can embed both a ZIP file and a TXT file into the PDF host to create effective polyglots, which are both active archive files and executable scripts, respectively. If the ZIP-embedded object is located near the end of the document or within the ZIP archive offset tolerance, it will decompression properly and the PDF will have no observable difference. For the script embedding, as the embedded file content is not parsed by the PDF rendering system, it can contain any characters the author desires, allowing limitless script block injection possibilities. Accounting for embedded script block location and potential exceptions when being interpreter by language engine is a separate issue and may require comment block code injections. *Image 56 – ZIP embedded uncompressed in PDF, creates PDF-ZIP polyglot* *Image 57 – TXT embedded uncompressed in PDF; script content can be executed from embedded content clear code with comment block value added to beginning of file. Create PDF-PS1 polyglot* ##### **PDF version alteration – MBR injection** One of the more exotic polyglot injection methods previously outlined is that of MBR sub-type injection. This is also available in PDF formats as the PDF header “%PDF-1.“ corresponds to the benign 16-bit OP codes: *Image 58 – “%PDF-1.“ ASCII translated into 16-bit OP code* These are operationally safe op codes, which when executed, will not perform any action which will trigger an error. However, as with all MBR injection it is important to trigger a jump to the relevant section memory section and avoid interacting with the PDF XObject definitions. To do this, the PDF version information is altered to store the required CALL, JMP, LOOP etc. code to achieve the necessary execution pointer movement. *Image 59 – PDF header is present but is missing required version number immediately following* Most interpreters do not require the version information and instead render the document based on the contained features due to specification iterations being additive not regressive (although not always the case). Due to this, the removal of the PDF version does not impact usage of the file. ##### **Duplicate keyword intolerant** Not a polyglot injection method but something that should be made aware of, PDF files which contain multiple instances of keywords, even if they reside within XObject streams, can corrupt the file type determination or parsing process of files. In the example below, the tool “Mitra” has been used on the POC||GTFO version 23 document to inject multiple file signatures and keyword patterns, which can confuse file type determination systems. As a result, one of the injected patterns is a secondary PDF document, which includes its own “trailer keyword”. From this, we can determine that clear-text embedding some file types/contents can potentially hinder polyglot deployment operations or cause the file to be rejected due to error depending on the intended target of the polyglot. *Image 60 – PDF host with secondary PDF keywords and other identifiers injected into XObject stream* ##### **Pre-header data padding** The PDF reference 1.7[[7](https://web.archive.org/web/20080624230300/http:/www.adobe.com/devnet/acrobat/pdfs/pdf_reference_1-7.PDF)] and prior specifies that the header for PDF files must be located within the first 1024 bytes of the file to be interpreted correctly under section 3.4.1. This stipulation was later removed in ISO 32000-1:2008 specification for the PDF file. However, PDF interpreters still adhere to this allowance due to legacy compatibility. This means that between the official specification and PDF interpreter operations, there is a disparity of 1024 bytes in the header location. This is a prime location for file injections effectively converting the PDF into a sub-type. This allowance is what makes PDF files so capable as a sub-type in polyglots, as the offset tolerance allows for almost any base file types header and structure to be defined before subsequently injecting the PDF content data. With the addition of PDF XObjects being offset based rather than sequentially located, it allows for a complex deployment. In the example below, an MBR is injected into the 512B of the file by using this header buffer to inject OP codes to perform jumps on boot to an appropriate memory sector. The PDF header is located at offset 0x07. *Image 61 – PDF document with MBR opcode injected into header buffer space* *Image 62 – PDF header buffer contents translated into OP code x16 architecture* ##### **XObject injection with no link/reference** One of the simplest methods to inject into the PDF format is to create an XObject with a length specifier for the stream and then populate the stream with the sub-type file content. As the PDF structure allows as many XObjects as desired, and they do not need to be used/referenced, this permits abuse. As the XREF table can be altered to accommodate these additions post file creation, this is a relatively simple addition. However, the XREF itself can also be completely corrupt and the PDF will still be permitted by most rendering systems due to their robust capabilities. *Image 63 – XObject with ID 999, stream contains “PK” ZIP file contents* In the example above, we can see that the PDF file has an XObject with the reference ID 999. This contains a stream which in-turn contains a ZIP files data for the sub-type polyglot. However, the sample was considered corrupt due to the missing “length” XObject dictionary value of the stream. This shows that creation of such polyglots still requires care to avoid rendering system errors. To create a properly formatted XObject, we repaired the file by adding the required length value to its object dictionary. *Image 64 – XObject with corrected length parameter* ##### **VeraCrypt/TrueCrypt encrypted container injection** Encrypted containers, such as those created by TrueCrypt or its successor VeraCrypt, are designed to work as encrypted file systems that hide within the existing file system. The containers are fully encrypted blobs in the file system, which can be decrypted via the appropriate system to access the hidden and secured content. As the containers are themselves file systems, they can contain any type of file within the constraints of the decrypted containers size. Researcher Ange Albertini published research[[9](https://mcfp.felk.cvut.cz/publicDatasets/pocorgtfo/contents/articles/04-11.pdf)] into how these containers can be injected into PDF file structures. This does not rely on leveraging the header buffer space but instead assumes the PDF header will be located at offset 0. As a decryption tool will also read the file from offset 0, the data from the beginning of the file must be treated as part of the encrypted container and, for a polyglot, also a PDF file. This is achieved as encrypted containers use a SALT, which is a random sequence, and the user’s password to decrypt the header. Decryption works by the password being combined with the SALT, which makes up the first 64 bytes, to create a key to decrypt the header, which starts with the keyword “TRUE” at byte 0x40. If the keyword is found after supplying a password, the system knows the key is correct and the container is decrypted. By decrypting a container for which the password is known, editing the SALT, and then re-encrypting the container you can edit the SALT of the file to anything. As the only requirement of the SALT is that it is expected to be random, it can be anything. When a container is re-encrypted, it uses the SALT, and the key provided to re-encrypt the contents. I If this is altered, the keys are changed. To this end, the user can change the SALT to look like any file header we want, in this case, a PDF header. *Image 65 – PDF header which, highlighted, is the SALT of the following encrypted container stored in the SALT specified XObject stream* We can perform entropy calculation on the first objects stream of a PDF file to determine the presence of encrypted data. This works as true encryption generates data segments which are indistinguishable from noise/random data to avoid analysis. As such, they will always have high entropy. The container also has a minimum size of 292KB which can be used as a file shard for entropy calculation using either the chi-squared or monte-carlo pi test. Prior research into this type of analysis and encrypted file container detection has been conducted with promising results[[8](https://drive.google.com/file/d/1p2W0RYuNo1GthDc_53RPB1ZLBIufRINz/view?trk=public_profile_project-button)]. This is an extreme edge-case of how polyglots could be introduced into files. However, from the technology, it is possible. The prevalence of such techniques is almost impossible to determine due to the nature of the container and the seeming inconsistency with the requirements (SALT not being random and being embedded in another file). Detection of such polyglots could be implemented relatively simply via entropy checking on the first 292KB of files (if they reach this size limit). ##### Line comment injection in object dictionary PDF specification under section 7.2.4 of the 1.7 ISO outlines that in-line comments are permitted by using the control character “%” or “0x25”. When used, the content of the file up until the following end-of-line control character (0x0A) is ignored by the PDF rendering software. This allows for arbitrary injection of content into the file at any point, regardless of component or structure, as the line is completely ignored, to hide sub-type document. *Image 66 – PDF files with comments inserted into XObject structures and as singular entries* ##### **Inter object padding** XObjects in PDF documents are specified within the xref table as to their offsets within the document and as such, they can appear anywhere within the file’s memory space. Due to this factor, spaces between the XObjects can be made bigger or smaller at will by appropriately adapting the XREF table to reflect the new offsets of the objects. This means that, by moving XObjects to new locations, we can create unused space/cavities between the existing objects which the rendering system does not process. This unused space does have limitations which authors must adhere to, this requires the omission of specific key characters or words. These are things such as “xref” or “trailer”, which a rendering system may detect in the injected space and attempt to process as a key element of the document. However, if these key names/characters are avoided any other content is free to be injected. *Image 67 – PDF with PK ZIP(red) injected into space between header(yellow) and XObject 1(green)* ### Currently unsupported file formats #### PCAP-NG ##### **Custom block injection** PCAP-NG files support custom block[[10](https://www.ietf.org/archive/id/draft-tuexen-opsawg-pcapng-03.html)] insertion, which can contain arbitrary data akin to comment blocks in other formats. The block has a type definition of 0x40000BAD or 0xAD0B0040 in LE, followed by its total block length (4 bytes), an identifier number (4bytes) and then the data content. These custom blocks can be inserted anywhere within the file, allowing for easy creation of parasite polyglots as the PCAP interpreter reads the file in a linear fashion with no termination block allow for both top-down and bottom-up file insertion. In the example below, a custom block is defined immediate after the Section Header Block (the file header), which contains a PDF XObject and stream definition. At the end of the PCAP-NG file another custom block is added, which extends to the end of the file memory which defined the end of the initial XObject and stream and contains the rest of the PDF structure, which permits its successful rendering. *Image 68 – Custom block header (yellow) with 0x40 content size (green) containing PDF header and XObject parameters (purple)* #### ILDA ##### **Palette injection** ILDA supports colour palettes, which are a set of 3 bytes to make up each entry. By translating a non 0 offset file header or script code into bytes you can inject this into the colour palette if its length is a factor of 3. If the image does not use indexed colours in its other entries, then the colour table has no effect on the resulting output. In the example below, a PDF header and XObject stream is added to the colour palette of the ILDA first entry. *Image 69 – First ILDA entry shows the format code as 0x02 (red), which is a colour palette with the number of records as 0x0014 (yellow) which is 20. Each entry is 3 bytes which allows 60 bytes, which encapsulates the PDF header and XObject (green)* #### iNes ##### **PRG-ROM injection** The iNes header allows the specification of the PRG-ROM segment in KB blocks. However, you can inject data into this segment which may only be detected as corrupted if loaded into an emulator. As the segment is data content, we can add arbitrary data if this is not attempted to be run. In the case below, a PDF header and XObject stream start are added in the PRG-ROM segment, with the end of the stream added after the end of the following CHR-ROM segment. This means that the entire game data is stored in the XObject stream, and the rest of the PDF is added to the iNes padding data section which is within specification. As the data is added to one of the game binary segments, it is unknown if the file will load into an appropriate emulator correctly. However, from a file determination standpoint, it is a legitimate ines file structure. *Image 70 – iNes file with PDF added to the start of PRG-ROM segment* #### WavPack ##### **WavPack metadata injection** WavPack header permits metadata sub-blocks as part of the header section. These metadata sections allow for arbitrary data insertion to support vendor-specific media handling. The max size of the metadata section is 3 bytes at 0xFFFFFF, which is 16,777,215, however this value is the number of words the metadata occupies, which is 2 bytes meaning the actual limit is 33,554,420 bytes (33.5 MB). This is more than sufficient to host most of the script or non 0 offset header supporting files such as PDF. *Image 71 – WavPack header with metadata sub-block with size 0x1C (green) of words (56 bytes - purple) containing PDF header and XObject stream specifier* #### TAR ##### **TAR header injection** TAR headers allow for arbitrary names of up to 100 bytes as part of their specification, followed by required structural information such as file mode, size, checksums, content type and others. The header makes up the first 512 bytes of a file, and if a header is below this threshold, it is padded to reach this size. Regarding polyglots, this “file name” field of 100 bytes effectively allows for any local variant ASCII to be injected at the start of the file contents. As TAR is an offset 0 header location file, this would typically eliminate other top-heavy documents. However, due to the “field name” allowance, this is not the case. If the TAR section contents are appropriately contained within sub-type ancillary blocks (such as comments) top heavy files and TAR files can be converted into a polyglot. The example below is a TAR file, which adds a PDF header at offset 0 as part of the “file name” section of the TAR header. The “file name” field also allows for the creator to add an XObject specifier, length parameter and stream start keyword, which effectively encapsulates the rest of the TAR content as part of an XObject stream. This XObject is not acted upon/referenced by the rest of the document but is structurally sound according to the PDF specification if an “endstream” keyword is included. As such, the file is both a legitimate base PDF file starting at offset 0 and a legitimate TAR archive file. The same mechanism could be applied to file types such as GIF, which have comment segments by starting a comment block at offset 99 and chaining comments to encapsulate the TAR content blocks. *Image - 72 – TAR file with “field name” space (red) populated with PDF header and XObject specifier* ### Other type spoofing technologies #### Spoofing learning models by small alterations Magika[[11](https://github.com/google/magika/tree/main)] is a deep learning model application developed to detect file types based on learned file type layouts. Analysis of the tool shows that it takes a 1KB chunk from the start and end of the document, with capacity for a middle section which is currently unused and feeds these chunks to the deep leaning model created. The models created were done by training against “25M files across more than 100 content types”. However, the files being trained against are likely in expected/specification adherent structures. During experimentation, it was found that small modifications to the files at key points could fool the detection system. An easy example is padding PDF files with a header of null characters, resulting in a classification of “ISO” type. This is likely, although not certain at this point, to be caused by standard ISO files being populated with padding data before the identification bytes at offset 0x8001 (minimum), which is not in the catchment of the chunk. As the model must infer the file type from the chunk it has, the model expects padding data of null to correspond to ISO and so misclassifies the file. Such alterations are the premise of polyglots and so model based systems like this which expect well-structured files will have poor accuracy and will be easy to fool. #### Hash collisions This is a method of file spoofing which can allow the reconstruction of files with different prefixes (headers) to have the same MD5 or even SHA1 hash. POC||GTFO 18[[12](https://github.com/angea/pocorgtfo/blob/master/releases/pocorgtfo18.pdf)] has great documentation on how this works, effectively by calculating the collision blocks on the prefix content, as long as the rest of the document remains constant (such as parasite polyglot) the prefix can be adapted with the appropriate collision block to result in the same has across all variants of the document. This means that the prefix can be adapted to legitimately open any document embedded in a polyglot whilst, for all instances, maintaining the same hash despite the alterations. ## Conclusion From our current research we have identified the various methods which can be used across our supported image formats and PDF file types to create polyglot files. As can be seen, the techniques span from the trivial to the complex depending on the skill, target or operation the original author intends the polyglot to be used for. However, by understanding the method in which polyglots are created we are provided with the basis for developing effective mitigation capabilities. As our research has shown, polyglots either take advantage of format specific quirks such as the PDF header space, arbitrary data storage segment types such as GIF comments, unused/padding space injection such as BMP data stream injection or direct image data manipulation. Regarding all these methods, detection is an extremely difficult task due to the various sub-type options and their own format quirks or robust interpreter tolerances (as is the case with PDF). This is where Glasswall’s zero-trust CDR provides a more robust security posture regarding polyglots, instead of scanning for polyglots it is instead only focused on removing areas or malformations of documents which can contain sub-type files. This allows the primary document, which the file is transmitted as, to retain its information integrity whilst removing all locations and format quirks which can support the creation of a polyglot. Almost all file-type determination systems, even those designed to detect polyglot files, routinely fail to identify polyglot files accurately. Although they may be more accurate regarding basic polyglot structures such as stacks or parasites with determinable locations, more complex types can elude them. Glasswall CDR avoids the limitations of traditional technologies which typically require the detection of an abnormality before action is taken, instead performing the same security-based operations on all files regardless of their origins. This paper outlines our current research progress into the polyglot file format issue; however, it is not exhaustive as we continue our work on other base-type formats and injection techniques. As an ultimate objective, Glasswall aims to provide a comprehensive CDR mechanism for removing all elements with a risk of polyglot support in supported document types including Microsoft Office document types and other media-based file types. This will allow documents to be processed across domain boundaries or between users with confidence that any sub-type documents and their associated risks have been removed during processing, with the base document being visually indistinguishable from the original. #### Sources 1. [https://github.com/davidbuchanan314/tweetable-polyglot-png](https://github.com/davidbuchanan314/tweetable-polyglot-png) 2. [https://github.com/angea/pocorgtfo/blob/master/releases/pocorgtfo03.pdf](https://github.com/angea/pocorgtfo/blob/master/releases/pocorgtfo03.pdf) 3. [https://blog.mindedsecurity.com/2015/08/pdf-based-polyglots-through-svg-images.html](https://blog.mindedsecurity.com/2015/08/pdf-based-polyglots-through-svg-images.html) 4. [https://malicious.pro/post/python_polyglots/](https://malicious.pro/post/python_polyglots/) 5. [https://research.h4x.cz/html/2023/2023-08-08--webp_polyglot_i-bootable_picture.html](https://research.h4x.cz/html/2023/2023-08-08--webp_polyglot_i-bootable_picture.html) 6. [https://research.h4x.cz/html/2023/2023-09-01--webp_polyglot_ii-script.html](https://research.h4x.cz/html/2023/2023-09-01--webp_polyglot_ii-script.html) 7. [https://web.archive.org/web/20080624230300/http://www.adobe.com/devnet/acrobat/pdfs/pdf_reference_1-7.PDF](https://web.archive.org/web/20080624230300/http://www.adobe.com/devnet/acrobat/pdfs/pdf_reference_1-7.PDF) 8. [https://drive.google.com/file/d/1p2W0RYuNo1GthDc_53RPB1ZLBIufRINz/view?trk=public_profile_project-button](https://drive.google.com/file/d/1p2W0RYuNo1GthDc_53RPB1ZLBIufRINz/view?trk=public_profile_project-button) 9. [https://mcfp.felk.cvut.cz/publicDatasets/pocorgtfo/contents/articles/04-11.pdf](https://mcfp.felk.cvut.cz/publicDatasets/pocorgtfo/contents/articles/04-11.pdf) 10. [https://www.ietf.org/archive/id/draft-tuexen-opsawg-pcapng-03.html](https://www.ietf.org/archive/id/draft-tuexen-opsawg-pcapng-03.html) 11. [https://github.com/google/magika/tree/main](https://github.com/google/magika/tree/main) 12. [https://github.com/angea/pocorgtfo/blob/master/releases/pocorgtfo18.pdf](https://github.com/angea/pocorgtfo/blob/master/releases/pocorgtfo18.pdf) 13. [https://docs.glasswall.com/docs/steganography-smudging-the-invisible-ink](https://docs.glasswall.com/docs/steganography-smudging-the-invisible-ink) 14. [https://winprotocoldocs-bhdugrdyduf5h2e4.b02.azurefd.net/MS-WMF/%5bMS-WMF%5d.pdf](https://winprotocoldocs-bhdugrdyduf5h2e4.b02.azurefd.net/MS-WMF/%5bMS-WMF%5d.pdf) 15. [http://www.dragonwins.com/domains/getteched/bmp/bmpfileformat.htm](http://www.dragonwins.com/domains/getteched/bmp/bmpfileformat.htm) 16. [http://ref.x86asm.net/coder64.html](http://ref.x86asm.net/coder64.html) ## Smudging the invisible ink – image steganography disruption with a focus on cover image degradation minimisation #### Author: Connor Morley (Principal Malware Security Researcher) [Download PDF](https://app.box.com/s/mjipu3zbym0rdu6kq828jl8rq9d31kab) ### Summary Steganography, the art of hiding data within other data, is a long standing security issue which is constantly evolving in both academic circles and with malicious architects. Image Steganography (IS) is a branch of steganography which specifically targets formats commonly used in almost all platforms for image rendering and storage, making IS use a widespread problem and very difficult to defend against without adverse effects. Common methods for mitigating IS typically effect the original image in a way that degrades it significantly or potentially allows the recreation of the hidden channel. Digital IS is often employed for malicious communications to achieve objectives such as exfiltration of sensitive information, IP theft, espionage and even infiltration of malicious elements (TA558 – April 2024). IS is a concern which has been constantly addressed by the cybersecurity industry over the years with various mitigation and detection strategies devised. However, the balance between mitigation and usability is something which routinely leads to a compromise which typically does not satisfy either concern completely. Understanding the ways IS can operate as well as the mitigation strategies available and their disruptive impact can be important to developing any potential solution to be universally applied. This paper is split into three sections; the first will aim to explain what exactly IS is, how it operates and some of the variants of IS that are commonly used in current systems. The second section will explain the research we conducted to try and effect high level of mitigation with a core focus on minimizing any degradation to the original image with statistical analysis and targeted solutions explored in detail. The final section will elaborate on solutions to this issue, the statistical fidelity of its use and how we have paired this with CDR technology to affect a domain-wide mitigation strategy to IS. ## Image steganography’s inner workings ### How do images work in digital space? IS can target the image components of formats that support image data such as PNG, BMP and JPEG. Steganography can be applied to almost any data structure such as text, audio, video and even program code itself. With IS the focus is solely on the components which comprise the visual representation of the subject and disregards all other elements such as metadata.  First, it’s important that we understand how images are displayed to a human user on any display device. Displays are made up of pixels which are made up, in most modern monitors, of subpixels made of LED's which control the red, green, and blue light emitted. The most common colour depth is 1 byte (8 bits) meaning that each colour has an intensity range of 0 – 255 with 255 meaning full intensity. Via colour theory, with these three base colours we can alter the intensity of the subpixels individually to cause a blend of the light emitted to create any colour we desire.  If you have an image which has the dimensions 2,544 x 3,392, that image is comprised of 8,629,248 pixels. Let’s assume the image is using standard RGB channel encoding, this means that the image data takes up 8,629,248 \* 3 = 25,887,744 Bytes or 25.89MB in raw pixel data. It’s important to know that image formats do not store images in the raw pixel data typically, instead, they compress them in some manner to make them easier to transmit and store. The ways in which images are formatted can have a direct impact on the IS methods which are available to that format which is something we will cover later. #### What is the Human Visual System (HVS)? The HVS can refer to the amalgamation of components which make up the visual processing that occurs within humans. HVS can be determined by an individual’s capabilities in processing visual information. This considers factors such as light perception (cones and rods) in the eye, the optical nerve bandwidth, and the cognitive processing speed of an individual as well as several other factors. As such, HVS can refer to the capacity for a person to perceive an image and its details.  The HVS has some known limitations which are actively targeted when steganography methods are devised. Three primary flaws in the HVS are poor perception of colour differences; poor perception of image edge abnormalities and finally poor perception of fine detail degradation. HVS can depend on visual capacity and can be used to determine whether alterations in images are perceptible. Using HVS may, in some cases, result in different statistical conclusions and some limited deviation can occur. ### Least Significant Bit (LSB) steganography – A basic version of IS One of the simplest methods to achieve IS, LSB steganography operates by manipulating the LSBs of colour channels in an image to hide the data stream. Raw pixel data is made up of the red, green a blue channel which are each 1 byte corresponding to a value between 0 and 255. So, for the shade of green below the values are: Red = 50, Green = 168, Blue =52: _Image 1 – Colour RGB (50, 168, 52)_ This means that a pixel with this colour has the raw pixel channel values in binary of: | Color | Binary value | | --- | --- | | **Red** | 0011 0010 | | **Green** | 1010 1000 | | **Blue** | 0011 0100 | _Table 1 – RGB colour channel values in binary for green (Image 1)_ Now let’s say I want to hide the letter “A” into the image. “A” in ASCII has a hex value of 0x41 or in binary: | | | | --- | --- | | "A" | 0100 0001 | _Table 2 – ASCII letter “A” in binary format_ LSB steganography will take the binary value of “A” and encode this into LSBs in the colour channels of the pixels in the image. LSBs of a binary value are those on the right-hand side. If we do some data mathematics, we can determine the maximum capacity of an LSB steganography data stream in an RGB image in bytes with the following formula: _Equation 1 – LSB hidden data capacity in bytes_ Therefore, if an image had a pixel area of 42 x 42 pixel it would have the hidden data capacity of 661 bytes. So, if the letter “A” has a binary value which is 1 byte or 8 bits long, we will need to encode at least 3 pixels as each pixel has a capacity of 3 bits we can encode. So, what does that look like with two new colours added with RGB values “98, 78, 01” and “86, 12, 198”: _Image 2 – Colour RGB (50,168,52), (98, 78, 01), (86, 12, 198)_ Now we have a binary sequence of: | - | Pixel 1 | Pixel 2 | Pixel 3 | | --- | --- | --- | --- | | **Red** | 0011 0010 | 0110 0010 | 0101 0110 | | **Green** | 1010 1000 | 0100 1110 | 0000 1100 | | **Blue** | 0011 0100 | 0000 0001 | 1100 0110 | _Table 3 – RGB channel binary values making up pixels in image 2_ When we use LSB steganography to inject the letter “A” into the pixel data the result is a manipulation of the far right value of each byte for each channel in order. This results in the following changes: | - | Pixel 1 | Pixel 2 | Pixel 3 | | --- | --- | --- | --- | | **Red** | 0011 0010 | 0110 0010 | 0101 0110 | | **Green** | 1010 1001 | 0100 1110 | 0000 1101 | | **Blue** | 0011 0100 | 0000 0000 | 1100 0110 | _Table 4 – LSB encoding results of letter “A” into RGB channels of image 2_ This shows that the LSB values (highlighted in blue) between the two have changed: * Original =    000 001 000 * Stegged =   010 000 010 From this, we can see the first 8 LSB is now “0100 0001” which we confirmed earlier corresponds to ASCII character “A”. But as we have changed the colour intensity values the pixels will be different, this difference appears as: _Image 3 – Contrast between original and altered pixel colours._ From the perspective of the HVS the two colours are identical, this is because the minor variations are imperceptible to the human eye. A hidden message, data blob or even payload can be encoded by using the same process for each hidden character with each character being encoded into the LSB of each pixel’s channel in sequence. LSB steganography does not employ any complex choice in which pixels to embed data into, and instead does so in a linear sequence.  One way the capacity of this technique could be expanded is to manipulate values other than the singular LSB but rather by using LSB 1 and 2 as well. LSB 1 corresponds to the penultimate bit value, LSB 2 the proceeding bit and so on. However, modification of the colour channel values further away from LSB 0 has a much greater effect on the generated colour and incurs a greater risk of visual detection by the human user. As such modification of any bit other than LSB 0 is typically used only in more advanced techniques which account for where in the image the pixel being altered exists relative to the pixel values around it. ### Alpha Channel Steg – LSB + 1 If an image is using the alpha channel and as such using the colour space RGBA, pixels have an additional channel assigned to them. The alpha channel is used to determine the opacity level of the pixel, effectively meaning how transparent the pixel should be. This value is required for composite images where multiple images are overlaid, the alpha value determines how the two colour values of pixels from two different images should interact.  In relation to steganography, the addition of the alpha channel which is also given in most cases a 1 byte depth expands the LSB capacity per pixel from 3 to 4. This means that in the prior example instead of requiring a minimum of 3 pixels to encode a single ASCII character, we now only require 2. This changes the formula to the following: _Equation 2 – LSB + Alpha hidden data capacity in bytes_ If we use our prior example of an image with pixel area of 42 x 42 instead of having 611 bytes of hidden data capacity we now get 882 bytes. ### Palette based steganography – Colour Chart Covert Comms Palette based images are normally used for extremely simple images such as logos or swatches. This is because palette based images have a pre-set index of colours which are used to generate the image, we call this the palette. The majority of use cases for palette based images are formatted as palette images to reduce the memory capacity needed for an image to make it easier to embed and transmit. The colour palette size is typically 256 which allows for a single byte to be used as the index ID for each entry. Each pixel in the image then stores a single byte index reference which determines the stored colour in the palette which is to be used for that pixel. This looks something like this: _Image 4 – Palette based image pixel and index relationship representation._ Some formats can support palettes of reduced sizes such as 2, 4, or 16, however it is rare to see a palette size other than 256. Regardless of the palette size, the relation between pixel and palette remains the same. The only other factor to consider is the colour depth supported in the palette, again this is typically 8 bits however some can store 16, 24 or 32 bit colour values. The use of such high colour depth values in a palette can itself be considered anomalous in most cases, however, its presence does not alter the way in which the palette image is processed only the potential data size which can be used to hide data.  IS for palette based images typically does not rely upon the encoding of the colour channels as with other methods. This is due to the substantially limited range of colours that are present and the likely visual impact such modifications would have on the resulting image. To calculate the maximum capacity of a palette based image with a palette of the maximum size 256 using RGB we use the equation: _Equation 3 – Palette based hidden data capacity in bytes using LSB._ With the palette size at 256 the maximum capacity is an extremely limited 96 bytes using LSB. As the palette is limited, modification of LSB values other than LSB 0 are very likely to cause visual discrepancies which are noticeable to a human observer and as such are typically not used in such cases. However, hiding data in palette based images is still possible by instead manipulating the index order itself.  Let’s look at two examples of the ways palette based steganography can work. #### Palette Index Reordering (PIR) PIR is a primary version of the palette based steganography. There are several ways this IS method can work, all of which require the palette index to be ordered into a specific pattern where the pattern contains the Hidden Data Stream (HDS). One theoretical technique works by converting a message into binary and then converting that into an index sequence relative to the original index of an image. Each modification of the index, such as colour in index X now being in index Y, represents either a 0 or a 1. The new index sequence is then applied to the original image palette creating a new permutation of the image palette which when received can be converted back into the binary sequence via the appropriate schema which makes up the hidden message. As a basic example let’s use a palette of 16 colours, for this method of steganography we are not concerned with the colour values themselves just their index value. When running the message “hello” through our encoder it generates the permutation “7, 4, F, 8, D, 9, 2, A, 1, E, 3, B, 5, 6, C“. When applied to the original palette the following occurs: _Image 5 – PIR encoding representation._ In the encoded image demonstrated in the colours in the bottom row, the palette index 1 stores the colour data originally stored in index 7, 2 stores value 4 and so on. The alterations in the palette sequencing represent 1’s and 0’s relative to the encoding schema as such by decoding the image using the same schema you would generate a binary sequence representing the string “hello”.  One of the most effective methods of palette modifications rather than modification of the colour channels themselves is that there is no visual impact on the image in most cases. Using the example above, the pixel which was original pointed at palette index 1 will now be pointing at palette index 9, however in the encoded image the modified index contains the exact same colour channel information. As such, the rendered image pixels display the exact same pattern. Let’s think of a practical use case, imagine two people email each other and their emails contain a corporate logo as part of the formatting which is a palette image. The sender and recipient are in possession of the original image used for the logo as well as an encoding schema/tool to generate and decode permutations of a 256 range index. The users would be capable of making covert communications as a sub-channel in those emails without any observed visual deviation.   The capacity of the PIR techniques is directly related to the number of unique permutations possible and then finding the logarithm base 2 of that limit. The equation for a full index PIR is: _Equation 4 – PIR hidden data capacity for 256 palettes using entire available sequences_ This results in a theoretical maximum capacity relative to the maximum number of permutations translated into either 1 or 0 to be 1684 bits or 210 bytes. This in practice would be unlikely due to the complexity in the encoding schema required, it is more likely that the use of segment encoding would be used as this reduces the complexity of the encoder but also the capacity of the HDS. #### EZ steg One of the most popular methods of palette based IS and used as a basis for other techniques, EZ steg devised by Machado in 1997 works on the principle of first reordering the palette based on colour luminance values. Luminance is the “brightness” of a pixel relative to its colour based on the way humans perceive certain colours to be brighter than others. Colours with a close luminance value are normally also close to each other in the colour space. Luminance is typically calculated in digital images to the perceived luminance with the following formula: _Equation 5 – Perceived luminance_ The HDS is then broken down into its binary components and encoded into each pixel’s palette index reference. As the colours adjacent to each other in the palette should now be very similar, based on their luminance value, the LSB of the index reference is modified when required by increasing or decreasing its value by 1 to reflect the current bit of the data stream being encoded.  The luminance reordering in a tiny subset of palette entries results in the following changes: _Image 6 – EX colour luminance re-ordering_ From this we can see that many colours that we perceived to be very closely matches to each other are now grouped together. However, this is not infallible as is demonstrated between the index entries with luminance values 68.151 to 76.536 where shades of brown and grey are sequences next to each other. This is one of the primary limitations with EZ steganography which can lead to significant discrepancies between the original and altered image.  When the palette is re-ordered the pixel references are also updated to match the new locations of the original colour pointed to by the pixel. EZ steg then uses the index references LSB to switch between a 1 or 0 as required altering the index pointed to by the pixel by 1 as required either up or down the palette. In the EZ colour palette, the index ID of 8 has the binary value “0000 1000” with the LSB being 0. If the first pixel is pointed at index 8 but the first bit of the hidden message is a 1, the pixels index reference would be potentially increased by 1 to “0000 1001” resulting in the pixel now using the colour stored in palette index 9. In this case, the two colours are very visually similar and so the alteration is unlikely to be detected.  However, as mentioned the visual difference between example index 4 and 5 with luminance scores 68.151 and 70.494 respectively is significant. Although the luminance order is correct, switching the LSB of index 4 which is “0000 0100” to a 1 “0000 0101” would result in the pixel now pointing to 5 and displaying an easily detected discrepancy.  In this example, we are using a tiny sample set of 16 colours whereas in real cases there would be up to 256 colours ordered. Despite this consideration, it is not uncommon when using EZ steganography for these jumps to significantly different colours to occur accidentally as part of the encoding process. ### Discrete Cosine Transformation Steganography (DCT) This method of steganography only works from image formats which are stored in the frequency domain, such as JPEG. When a JPEG image is written to disk, it is stored as a JPEG File Interchange Format (JFIF) file, which contains the frequency information for the associated image. When opened/rendered, the frequency information in the JFIF is decompressed using the JPEG decompression method which translates those frequencies into pixel values moving the image from the frequency domain to the spatial domain. _Image 7 – JPEG image of woodland, AI generated. Pixel grid 8x8 from location (0,0)._ _Image 8 – Frequency cosine representation of 8x8 pixel grid at location (0, 0) from Image 7_ DCT steganography works by instead of altering the pixel data in the spatial domain, it encodes the HDS into the coefficients in the frequency domain. Changes to the coefficients result in multiple pixels being affected but in very subtle ways if the modifications made are minor (typically to the LSB value) and typically limited to the higher frequencies. The higher frequencies are selected as the HVS has a lower perception to alterations in these ranges relative to lower frequencies as they affect finer details. This means that DCT steganography allows for the same capacity, if not more, than that of LSB steganography, depending on the tolerance of alteration permitted. DCT steganography results in a distributed deviation in the resulting image rather than specific and significant alterations which occur when spatial modifications are made. As specific alterations may be more easily detected by the HVS when adjacent pixels have sudden significant deviations, this has a greater chance of going undetected by an end user. Assuming the sender and recipient use the same DCT steganography system, they may or may not use a key to determine which coefficients are modified or use a standard determination. As the JFIF format is already compressed it is less likely to be affected by transmission compression making it slightly more robust than other methods. ### Pixel Value Differencing Steganography (PVDS) PVDS is a method of IS which does not directly encode the HDS into the binary values of an image but instead applies a mathematical process to hide the data stream. Whereas in the prior examples the hidden data is stored typically in the LSB, PVDS aims to make modifications to the LSB of pixels but not so that they correspond to the hidden data sequence. Instead, PVDS will calculate the difference between the values of colour channels in pixels adjacent to each other and manipulate this difference to encode the data. Using a greyscale image with a colour range of 0 to 255 as an example we can easily see the calculated differences between the pixels. _Image 9 – Pixel value differences in greyscale_ Whereas with LSB steganography we would be attempting to encode the HDS into the value of the pixel, with PVDS we instead alter the difference by adjusting the difference between the two adjacent pixels. This is achieved by adding or subtracting to the pixels to achieve the desired distance. As an example, let’s assume we have the symbol “)” as a binary sequence “0010 1001” and we want to insert this into the sequence above in the LSB 0 position of the difference values between pixels. The difference between pixel 1 and pixel 2 is “1” which has an LSB of “1” whereas the first binary value of “)” is 0. To change this, we adjust the values of pixel 1 and 2 in order to increase or decrease the difference value so that its LSB is 0. In this case we can increase the value of pixel 2 by 1 which results in a difference of 2 which does have an LSB of 0. Subsequently, the difference between pixel 2 and 3 is altered as we have changed the value of pixel 2 which is changed from 45 to 44. The next binary value of “)” is another 0 which 45 has as an LSB, so by changing the value of pixel 2 we have effectively encoded 2 bits of the HDS “)” binary sequence already. The process is continued with the alterations typically being made to exacerbate the difference that already exists rather than reduce it. However, calculations are normally made within the encoder to determine the minimum alteration required to encode the desired bit sequence rather than strictly exacerbating or dealing with pixel values already at their threshold. _Image 10 – PVDS encoding example for symbol “)” first four bits as PVDS LSB 0_ This example of PVDS is extremely simple due to its limited bit choices and linear application, whereas in real use-cases far more complex strategies are likely to be employed. They aim to maximise the HDS whilst maintaining minimal visual disruptions by taking advantage of the HVS weakness to modification to pre-existing colour difference. It is important to note that PVDS typically will not work only on LSB 0 but may work up to LSB 3 if the tolerance is deemed high enough. To do this, three common strategies are employed, either a difference threshold for linear sequencing, a range assignment is determined for data segment embedding or a scan for edge pixels is used. A threshold, let’s call it K, is the definition of the minimum difference that must exist between two pixels before information is permitted to be embedded. The value of K is designed to identify pairs of pixels where the difference between them is significant enough that it can be modified without obvious visual effect and that the difference is large enough that it can contain a minimum data segment without obvious visual disruptions. If you had 2 pixels beside each other that had the same or very closely matching values, and the encoder required an LSB modification of LSB 0 to 3 which has a maximum effect of 15, the visual impact would be identifiable by a human viewer. However, if the difference between the two pixels was K or higher it is assumed that further modifications would be imperceptible to a human viewer. In the example below, comparing the pixels in vertical order, it is considered that the sample pixels on the left are easy to distinguish between the original and encoded values. However, the example pixels on the right are harder for the human eye to distinguish between. _Image 11 – Difference threshold example_ Range assignment works similarly to the difference threshold but instead breaks ranges down into capacity determinations. This is a specification that if a difference between two pixels is between X and Y, then it can only support the encoding of Z bits from the HDS. The ranges can look something like: | Difference | LSB range available | Binary | | --- | --- | --- | | 0-16 | N/A | 0000 | | 17-62 | 0 | 0001 | | 63-126 | 0-1 | 0011 | | 127-184 | 0-2 | 0111 | | 186-255 | 0-3 | 1111 | _Table 5 – PVDS ranges example_ This means that the encoding of the HDS is still conducted in a linear fashion against the pixels, but they are dynamically adjusted depending on their original distance. As from the previous example this ensures that only minor changes can be made to pixels with proximity in the colour space, whilst those with a great divide can be modified more significantly. As the more significant changes only occur within pixels with greater distance between them already, the effect is not easily detected by the human eye. Tables can also be modified so that specific difference ranges or specific LSB depths are avoided, such as LSB 0. This adds more complexity as isolated ranges may interact with LSB 1 and 3 and the next range only interacting with LSB 2. This can make HDS distribution and, as a result decoding and detection, more complex. The final method used is a scan of an image to determine edge pixels, this is the most complex method of PVDS but also the most effective. Whereas the prior two methods can be used in a linear fashion against all pixels in an image, edge scanning instead identifies pixels in an image which make up the edges between two distinct colour spaces before performing encoding. This is done because another weakness of the HVS is it distinction of variations in edges. Significant fluctuations in the edges of objects in images are “ignored” by the HVS which introduces a high capacity pixel modification space to be exploited. Alterations against edges can also be controlled using a threshold range table like the one above. This would be done in order to dynamically adjust the HDS embedding depth in the edges detected making it even harder to detect and decode without the original range table. Using a kernel/grid of 1x2 or 2x2, the encoding program will scan the image for pixels that exceed a relatively high threshold in a specific pattern which typical identifies an edge. In the image below you can see an example of what the scan would determine to be an edge, which in this case is the edge of a wall of a hut. The stark colour difference between the adjacent pixels allows for maximum encoding capacity with a very low probability of being observable by a human viewer. _Image 12 - Pixel edge example_ ### Spread Spectrum Image Steganography (SSIS) Spread Spectrum Image Steganography (SSIS) is a complex method and works by effectively leveraging expected noise in an image to encode the HDS. See US6557103B1.  SSIS works by modulating a narrow band signal, which is the HDS, and spreading it across the spectrum of the cover image. This is done so that the modification in the image replicates the noise expected in photoelectronic images such as background radiation, UV spikes, sun flares etc. In practice this means taking the binary sequence of the HDS and spreading and modulating this via the use of a Pseudo Random Noise Generator (PRNG) sequence into an image. This can be done in the frequency or spatial domain, for this example we will explain in the spatial domain.  A key is provided which the sender and receiver know which is used with the PRNG to generate a specific encoding and subsequent decoding sequence. This sequence determines the pixels which are to be modified (the spreading) and to what degree each individual pixel should be modified (the modulation). In the spatial domain this means altering the binary values of the colour channels of the pixels to encode to HDS, however, the modulation sequence means that each selected pixel will be altered in a different way. This means that unlike other methods moving up from LSB 0, SSIS allows modification of specific bits in the channel independently. Despite this, a range limit from the LSB is normally maintained to prevent significant and therefore detectable modifications.  The encoded image should be imperceptibly altered when compared to the original regarding the HVS as well as steganalysis. As the alterations aim to replicate expected noise, steganalysis should determine the changes (without access to the original image) as typical spectral fluctuations rather than patternable or detectable modifications. As both the sender and recipient have the PRNG key, the receiver can generate the required modulation and spreading sequence to identify which pixels have been altered and in what way. As an external observer it would be nearly impossible to identify this form of encoding without access to the original image or the PRNG key. ## Investigation into steganography mitigation After identifying commonly utilised steganography methods and determining how they operate we began our investigation into what mechanisms could be used to mitigate them. The two core objectives of this research were first to find methods which could effectively and consistently corrupt the HDS in stegged images and second to minimize the visual degradation such methods result in. As mentioned previously, this balance between these two objectives can result in a compromise as it is often difficult to mitigate steganography without altering the image in some way. In some aspects, a key can be to make the alterations as imperceptible to the HVS as the original steganographic encoding method. We considered adding steganalysis to our mitigation mechanism to only make alterations on images where steganography was likely to occur. However, steganalysis is resource intensive and is not infallible with the primary goal of some steganography techniques being to specifically evade such analysis and detection. Instead, we conducted our investigation under the assumption that the mitigation methods identified would be applied to all images processed without exception. This meant that we would have to assume that image modification and subsequent degradation would always occur if the mitigation method were appropriate to the image format being processed.  To measure the effect the proposed mitigation methods would have, we decided to employ multiple statistical values to accurately track deviations occurring between the original and processed image. These are: * **Mean Absolute Error (MAE)** – Used to quantify the difference between two images based on their pixel values. This measurement accounts for all deviations/errors encountered between the two images. _Equation 6 – MAE equation_ * **Mean Square Error (MSE)** – Used to quantify the difference between two images based on their pixel values. The deviation/errors are squared resulting in a greater sensitivity to larger deviations and a squashing of minor deviations. This measurement is tailored for significant alteration detection. _Equation 7 – MSE equation for greyscale image \[1\]_ * **Peak Signal to Noise Ratio (PSNR)** – Defines the ratio between the maximum possible power of signals in an image against introduced noise (errors) typically measures as MSE. _Equation 8 – PSNR equation_ * **Structural Similarity Index Measurement (SSIM) –** A model metric which perceived modification as structural changes in the image. Considers additional aspects other than absolute errors such as luminance. _Equation 9 – SSIM equation_ By using these statistics, we were able to measure between strict pixel deviations and automated perception deviations. Using HVS as a basis, we began by modifying images using various filters and effects with varying intensities. For each process we calculated the statistical deviation and from this established an HVS baseline for each metric, indicating a breach of this baseline was likely to be perceived by the HVS. This is subjective to the researcher’s visual acuity and individual HVS will vary considerably, however as with all experimentation the available resources introduce limitations. The baselines determined are as follows: * MAE = Below 0.01 * Palette based images = Below 0.015 * MSE = Below 0.0025 * PSNR = Above 0.4 * Palette based images = Above 0.35 * SSIM = Above 0.9 The first two metrics increase between 0 and 1 the more errors/deviations are detected between the two images with 0 meaning identical and 1 meaning complete corruption. The latter two metrics decrease from 1 to 0 as more deviations are detected with 1 meaning identical and 0 meaning complete corruption.  Palette based images introduced deviations in the statistics observed which needed to be accounted for. These deviations can be caused due to the limited data set/colour set which can be affected during processing. As the set was significantly smaller and the images typically simpler it was found that slightly higher statistical deviations were required to achieve HVS detection. This is reflected in the modified baselines for both MAE and PSNR.  When determining mitigation success encoding and decoding tools were required for each tested IS technique. During experimentation, each mitigation technique would be tested against compatible image formats from the test set. Each supported in the test set would have a key phrase encoded into them using the corresponding steganography tool. The original image was tested after encoding to ensure that decoding/extraction of the key phrase was successful. After mitigation experimentation, the generated image would have the same decode/extraction operation conducted against it to determine if the output generated the key phrase originally encoded. If the phrase was a match the mitigation was considered a failure, if the content was corrupted the mitigation was considered a success.  The test set used is made up of different image complexities, such as clear blue skies, scenery, and a human portrait, translated into multiple formats and colour profiles. For each mitigation method only the supported image formats were tested. In the following, image names are appended with either “P” or “TC” referencing whether they are palette or TrueColour image formats respectively. This resulted in a total test set of 42 images. All statistics documented here are averaged results on the statistical deviation observed across the test set for each specific format. This provides an overall determination of mitigations impact against varying file types with a range of image complexities considered. ### Palette shuffling – Mitigating palette based techniques all at once Palette based IS typically revolves around the technical art of reordering the palette to encode information. This, as specified in chapter 1, can be due to the colour depth available to such images being restricted creating an extremely low HDS bandwidth. As palette IS instead can require the palette to be sequenced to allow pixel index references to contain the HDS, the weak point of these methods can be the expected palette index sequence. Palette shuffling involves taking the colour stored in one index and randomly allocating it to another index in the palette. Imagine you have red stored in index Y and blue stored in index X. Palette shuffling will move the colour red to be hosted in index X and blue instead will be moved to index Z. All pixels pointing to index Y are altered to now point to index X where the colour has been moved to. This results in the palette being completely reordered but the rendered image being unaffected as the pixels still relate to the same colours. By randomising the colour locations, the index based encoding is corrupted and the HDS is broken. However, this mitigation can be reverted if the ordering follows a specific schema such as luminance values in EZ steg. As the recipient could reorder the palette themselves to achieve the same sequence the randomisation can be reverted. This is why although palette shuffling does disrupt index encoding in the transmitted image, it can be important that it is paired with another mitigation method which will affect the colour values themselves. Slight alterations in the colour values can severely disrupt the calculations used to achieve these specific sequences and as such is required to prevent the randomisation from being reverted. ### LSB stomping LSB stomping (which can also be referred to herein as “modified LSB” or “least significant bit modification”) can be used against the steganography method outlined earlier in this paper. LSB stomping can be a mechanism for mitigation, achieved by changing the LSB’s in channels in pixels to either a 0 or a 1. Considering many steganography methods rely on encoding data in some way into the LSB, least significant bit modification can be an effective method of mitigation. This mitigation method was tested against all 7 images in 6 formats consisting of a total of 42 images: | **LSB** | Average PSNR | Average MAE | Average MSE | Average SSIM | | --- | --- | --- | --- | --- | | JPG | 0.400437143 | 0.008164286 | 0.000292857 | 0.964072857 | | TIF | 0.481410714 | 0.003907143 | 0.00001 | 0.997832857 | | BMP-P | 0.461514286 | 0.00527 | 6.28571E-05 | 0.990738714 | | BMP-TC | 0.481408571 | 0.003907143 | 0.00001 | 0.997832857 | | PNG-P | 0.481489143 | 0.003901429 | 0.00001 | 0.997822857 | | PNG-TP | 0.4814 | 0.003907143 | 0.00001 | 0.997822857 | _Table 6 – LSB stomping statistics table_ _Image 13 – LSB stomping statistics graph_ Across LSB, PVDS, DCT and SSIS IS methods this achieved full mitigation in all test cases where LSB 0 was included as part of the encoding range. These results demonstrate that this mitigation method yields statistical deviations that do not breach the baselines established for the HVS. The JPG image can come close regarding the PSNR value but does not exceed it.  However, these results can be limited due to the limitation of LSB stomping on LSB 0. As LSB related steganography is known in the security industry and by malicious actors it is typically avoided wherever possible to prevent both steganalysis detection and ensure robustness. As LSB values can be likely to change during routine interchanges or maintenance operations, attackers can either avoid them or opt to encode into LSB 1 or 2 values instead accepting the higher chance of detection. Other more advanced IS methods outlined may use higher LSB values but can be very specific about where they are encoded which could sidestep this mitigation. When the experiment was reconducted when encoding methods specifically avoided LSB 0, the mitigation failed in all cases. As LSB stomping on values any higher than LSB 0 would result in considerable alteration in the colour space between the original and resulting image this is also not something we considered as viable. Therefore, despite these results being promising the mitigation method was considered too limited to be effective. ### DCT coefficient manipulation Thinking about DCT steganography we decided to try a targeted approach against JPEG images which can support this IS method. Manipulation of the images coefficients is possible and initial considerations were to perform coefficient stomping like the mechanism used in experiment 1. However, instead a factorial modification of the DCT coefficients seemed more effective to minimize the disruption to the image and replicate the distributed modification the original IS mechanism would have achieved.  The mitigation method was tested with varying degrees of modification against a set of JPEG images. The first two tests are magnitude 1.3x was conducted twice with two different libraries, FFT and DCT, respectively. The other tested magnitudes were conducted using the DCT library. The results: | **DCT** | Average PSNR | Average MAE | Average MSE | Average SSIM | | --- | --- | --- | --- | --- | | 1.3x (FFT) | 0.174518571 | 0.117555714 | 0.040518429 | 0.941655714 | | 1.3x (DCT) | 0.174499714 | 0.11759 | 0.034148143 | 0.941644714 | | 1.2x (DCT 16 block) | 0.20029 | 0.215055714 | 0.025751429 | 0.961205 | | 1.1 (DCT) | 0.254948571 | 0.176668429 | 0.00294 | 0.986185 | _Table 7 – DCT alteration statistics table_ _Image 14 – DCT alteration statistics graph_ The first thing to note here is the difference between the first two statistics when the same magnitude was applied whilst using two difference libraries. This is due to slightly different methods used to affect the coefficient changes. Between the two the DCT library was found to have less impact and thus could be better. In all tests, messages embedded using all IS methods were corrupted achieving full mitigation but also had significant deviations observable by the HVS. This is most accurately represented by the high MSE and low PSNR values. As the magnitude change trends down you can see that these statistics begin to approach back to the HVS baselines. The third experiment tested adapting the DCT block size for modification from 8x8 to instead by 16x16 doubling the standard DCT block size. From this you can see the MAE drastically increased despite a decrease in the magnitude used. A consistently positive statistic can be SSIM which indicates that the frequency alterations are affecting pixels in a general fashion rather than a targeted one. This can be because targeted alterations can alter the perceived structure of what is in the image, whereas general modifications achieved in the frequency domain do not have the same impact as the colour space alters more universally. As a result, this modification was not adapted in other magnitudes. However, even at the lowest coefficient modification the PSNR value was unable to get higher than 0.255. At a magnitude of 1.1x the visual impact was not dramatic but was significant that any sensitive images would be considered corrupt. Therefore, in some cases, this method may not be preferred to other methods. ### Noise reduction filter A noise reduction filter can be a median filter. Median filters work by taking a user defined value and using this to determine the kernel size to be used. The kernel then scans in an overlapping fashion across the pixels in an image, at each stage the median values of the pixels can be calculated and then assigned to the central pixel. Noise filters can be used to retain sharp edges on images whilst removing outlier/anomalous pixel values across an image. Effectively, this filter can have a high degree of image quality retention at the cost of a more limited effect on the range of pixels and noise it has the capacity to correct/alter. As PVDS and SSIS specifically target edge and noise values, a noise reduction filter was a targeted attempt to address these two IS methods. Our original experiment attempted altering the images using a kernel with a size of 7x7, the idea being that a larger kernel would create a more even modification at the cost of higher calculation resources. The results: | **Noise Filter 7** | Average PSNR | Average MAE | Average MSE | Average SSIM | | --- | --- | --- | --- | --- | | JPG | 0.335361429 | 0.017911429 | 0.001983429 | 0.814094286 | | TIF | 0.33301 | 0.018072857 | 0.001984286 | 0.805405286 | | BMP-P | 0.310104286 | 0.01861 | 0.002164286 | 0.762373 | | PNG-P | 0.310161429 | 0.018565714 | 0.002162857 | 0.762915857 | | BMP-TC | 0.333041429 | 0.019057143 | 0.001982857 | 0.80549 | | PNG-TC | 0.333137143 | 0.018004286 | 0.001978571 | 0.806305714 | _Table 8 – Noise reduction filter with kernel size 7 statistics table_ _Image 15 – Noise reduction filter with kernel size 7 statistics graph_ This filter did achieve full mitigation in PVDS and SSIS as well as all other IS methods employed, however as the statistics show the resulting images were degraded to a detectable degree and did not always succeed with mitigating LSB depending on where the linear sequence was injected. All statistics other than MSE, which can be sensitive only to major pixel deviations, were in breach of their baselines. This indicated that this was a potential method but that adjustments to the alteration threshold may be required. To this end we re-conducted the experiment but this time with a smaller kernel of 3x3 making the median calculation more specific to a particular region of pixels.  The results: | **Noise Filter 3** | Average PSNR | Average MAE | Average MSE | Average SSIM | | --- | --- | --- | --- | --- | | JPG | 0.393608571 | 0.010145714 | 0.000780029 | 0.921681429 | | TIF | 0.383281429 | 0.009665714 | 0.000754286 | 0.920728571 | | BMP-P | 0.317141571 | 0.010155714 | 0.000915714 | 0.885255714 | | PNG-P | 0.317313 | 0.010107143 | 0.000912857 | 0.885791429 | | BMP-TC | 0.383314286 | 0.009661429 | 0.000752857 | 0.92076 | | PNG-TC | 0.383737143 | 0.009587143 | 0.00075 | 0.921502857 | _Table 9 – Noise reduction filter with kernel size 3 statistics table_ _Image 16 – Noise reduction filter with kernel size 3 statistics graph_ The reduced kernel size and subsequent targeted pixel alteration still achieved full mitigation with PVDS and SSIS and all other IS methods again but with a much more promising statistical output. When palette tolerances are accounted for, the resulting statistical differences are much closer to the required baselines but are still breached in relation to the PSNR and SSIM. As the SSIM statistic directly corresponds to noise/corruption being introduced in the image in a perceptible way when compared the original this is a significant deviation. However, the results were much closer to the required limits than the prior larger kernel results. When compared by the HVS it was difficult to perceive differences, however, as the experiments were conducted against a small sample set of images the baselines are enforced to correspond to all images. Due to these factors, this mitigation method can, in some embodiments, be rejected as a potential general mechanism. ### Smoothing filter A smoothing filter can be like a noise reduction filter but instead uses averaging of colours over a specific kernel rather than a median. Smoothing can be typically applied to either a specific region during photo editing or generally across an image to reduce noise and produce a “blur” effect. The filter can be effective at removing artifacts/noise and also can be used for pre-processing other functions such as edge detection by removing minor variations detected in edge pixels making them easier to detect. Whereas a noise reduction filter has the effect of preserving and refining sharp edges, a smoothing filter can have the inverse effect by instead slightly “blurring” the edges as well as other noise pixels in an image. This gives a more general noise reduction capability with the trade-off of a more significant alteration of the image. A smoothing filter can have a more general mitigation capability against a range of IS techniques due to its less targeted intended alterations. This means that opposed to a median filter, the average filter used in smoothing can be more effective against all forms of IS rather than those that focus on specific regions. Another factor to consider is that smoothing filters can use either uniform of gaussian weighted kernels to make the alterations more relative to the central pixel and maintain structural relativity. This can be done by adapting the weights of the kernel relative to the central pixel, with the central weight being significantly higher than the surrounding pixels. This means the influence of pixel values on the resulting alteration decreases depending on their perceived distance from the targeted pixel. In our experiment, we used a standard 3 x 3 kernel size with the gaussian weighting and results: Matrix 1 – Gaussian weighting used in smoothing filter 3x3 kernel | **Smoothing** | Average PSNR | Average MAE | Average MSE | Average SSIM | | --- | --- | --- | --- | --- | | JPG | 0.415606714 | 0.009175714 | 0.000558581 | 0.944788571 | | TIF | 0.408596714 | 0.008201429 | 0.000404948 | 0.960213571 | | BMP-P | 0.378051571 | 0.009721429 | 0.000458043 | 0.945604286 | | PNG-P | 0.378190143 | 0.00969 | 0.000458043 | 0.94584 | | BMP-TC | 0.408627 | 0.008198571 | 0.000403661 | 0.960227143 | | PNG-TC | 0.408958429 | 0.008147143 | 0.000402233 | 0.96059 | _Table 10 – Smooth filter application statistics table_ _Image 17 – Smooth filter application statistics graph_ The results from this experiment were extremely promising with the statistics all being within the baseline specified by the HVS. When palette allowances are considered, all the statistical metrics can be within the threshold with the effect observed on the files being minimal or unobservable when considered as a whole image. However, this may not be directly reflected in the mitigation capabilities of the filter. It was found that LSB and PVDS IS using a linear sequence encoding could avoid mitigation in some cases where the image was simple. This was interpreted as a limitation of the filter to apply weighted alterations across already similar colour range in a linear space Due to the use of gaussian weighting the mitigation against techniques such as PVDS when used in a linear instead of edge targeted fashion could be worse. This indicated that minor variations in linear sequences may be less likely to be mitigated than those found in outlier pixel values such as those encountered in noise and edges generated by more advanced techniques. As such, this had a limited success profile but so far, the most promising results statistically. ### File bouncing File formats can be typically broken down into two primary categories, lossy and lossless. The latter format type can be defined when the pixel information can be compressed and altered by the specification for storage, but upon rendering will always have the original pixel values regardless of the format transformation. Lossy formats instead allow for data loss to maximise compression and minimize required memory space. One such format we mentioned in Chapter 2 is JPEG.  Due to this expected quality degradation in JPEG format transformation, we considered whether this could be a viable mitigation strategy. The process of JPEG transformation can be analysed at a low level for potential mitigation use. The two primary steps during the transformation process which causes image degradation are the colour translation and quantization phases.  When an image is converted to JPEG, first in the spatial domain, the colour space is converted to JPEG standard which is typically the YCbCr colour space and secondly once converted to the frequency domain quantization takes place. In YCbCr, Y is the luminance value and Cb and Cr are the chrominance components of the blue and red differences. This translation causes artefacts to be introduced into the colour space of the new image as the RGB values are not represented completely accurately in the YCbCr colour space, however the differences can be considered imperceptible to the human eye. Another consideration is that different systems can have different translation equations meaning that images created in the JPEG format by one system may have slight variations when created in another system. These variations are typically in what range of values is permitted/expected and accounting for potential negative values (by removing them). _Equation 10 – RGB to YCbCr conversion formula accounting for negative values_ Quantization can be a mechanism applied to the frequency domain of the image after it has undergone DCT transformation. Once the image has been broken down into (typically) 8x8 grids of coefficients representing 8x8 grids of pixels, quantisation can be applied to squash higher frequencies in that grid. The coefficients represent a range of oscillating frequencies relative to the values of the pixels in the original image. Lower frequency coefficients represent base information like large area colour bases for the image whilst higher frequency coefficients represent fine detail such as edges or sharp changes.  Higher frequencies are targeted in this step as the HVS can be less adept at detecting reduction in the fine details relative to the base details and as such can cause perceptible degradation. The quantization itself can be conducted using an 8 x 8 “quantization matrix” which is made up of a pre-set of values. The JPEG specification “itu-81 – Section K.1” \[2\] outlines an example base matrix for luminance quantization with 50% quality retention can be: _Image 18 – ITU-81 JPEG specification basic quantization matrix \[2\]_ The matrix can then be applied to the coefficients of a block by dividing the corresponding entry by the quantization weight and then rounding to the nearest whole integer. These values are what are then used in compression. When the image is rendered, the quantized frequencies are inversely multiplied by the values of the quantization matrix to reconstruct the DCT values. However, as there is a rounding operation during quantization this can lead to deviations between the original and reconstructed frequency values. In the frequency domain matrices, the lower frequencies can be represented at entry 0,0 (top left) and highest frequencies in at entry 7,7 (bottom right). This is reflected in the quantization table above, with the weight trending upwards towards the higher frequency coefficients meaning they can be impacted more. A practical example is provided below: _Image 19 – Original DCT coefficients of image_ _Image 20 – Quantized DCT coefficients of image after reconstruction_ In these two heatmaps above we have the coefficient values which are generated during the DCT translation of a standard image. These values from top left to bottom right indicate low to high frequencies which occur in an 8 x8 pixel grid. The second image shows the same coefficients after quantization has taken place and the coefficients have been reconstructed. In the second heatmap, we can see that most high frequency coefficients have been resolved to 0 because of the weighted values in the quantization matrix. Additionally, even the lower frequency values have been adjusted slightly with only the value “0,2” retaining its original value of -84. This results in a distributed alteration of the pixel values that are generated when converted from the frequency domain back to the spatial domain.  This can be conducted, and the quality decrease can be accepted, as part of the JPEG format to allow for maximum compression. Using techniques such as Huffman Coding, resolving most of the matrix values to 0 means that a high level of compression can occur. Additionally, the range of the other coefficients in its compressible format are normally between 0 and +/-10 further adding to compression capability. The heatmap of what is compressed is the quantized DCT coefficients before reconstructing/multiplying the by quantization matrix. The compressed heatmap in this case looks like: _Image 21 – Quantized DCT coefficients for compression_ The quantisation values of the matrix can be adjusted via a “quality” variable, which can be specified by a user. This variable adjusts the weighting of the quantization matrix to perform higher or lower weighted quantization against the coefficients depending on the tolerance for deviation. So, if we bounce a lossless file to a lossy format and back again, we can expect the values of the pixels to be affected. If we bounce a lossy file to a lossless format and back to lossy, we can also expect pixel values to be changed as a result. This is due to the colour translation phase introducing artefacts and potentially using a slightly different equation as well as the quantisation phase potentially using slightly different matrix values, either due to base values being different or the quality variable being different. In this process, we conducted this file type bouncing with a quality variable of ninety-five with the following results: | **File Bounce** | Average PSNR | Average MAE | Average MSE | Average SSIM | | --- | --- | --- | --- | --- | | JPG | 0.581620857 | 0.000790899 | 8.60089E-06 | 0.998330143 | | TIF | 0.430017143 | 0.006327143 | 0.000131429 | 0.968938286 | | BMP-P | 0.371294286 | 0.010245714 | 0.000257143 | 0.9341 | | PNG-P | 0.37125 | 0.010255429 | 0.000257143 | 0.9336 | | BMP-TC | 0.429988571 | 0.006307143 | 0.000131429 | 0.968898571 | | PNG-TC | 0.430584286 | 0.006262857 | 0.00013 | 0.969382857 | _Table 11 – File Bouncing statistics results_ _Image 22 – File Bounce statistics graph_ This can result in universal mitigation across all IS methods, low impact on the resulting image with palette accommodation and a low resource overhead. Statistically this mitigation method may not breach any set baseline and may not result in any perceivable difference between the original and resulting image as judged by the HVS. This method may have the additional benefit of leveraging a technology which has been refined over a longer period to make it efficient and swift with the conversion operations consuming very little local resources as opposed to other more mathematically intense filters such as a median filter.  As the quality retention variable was set at 95 in these results, further degradation of the image may not be required. As such, using 95 as a base for this filter may be deemed effective; additional degradation, may be employed in a lower risk tolerance environment. ### Additional mitigation details Assessment of the statistical output related to degradation and the binary value of whether steganography mitigation was achieved identified some potential mitigation methods. In some cases, targeted methods of mitigation for specific IS methods can be too costly in terms of image degradation or too specific to be effective in all cases of the IS method being targeted. As such, general methods of mitigation can be more effective in both HDS corruption and image quality retention. File Bouncing can be considered a successful mitigation method due to its distributed alterations of pixel values, its removal of noise (high frequencies) and its exceptional retention of quality. As this method can have a low overhead, it can be practical for large-scale implementation or as a routine operation. A data comparison utility can be used to observe the pixels affected by this process in the following image. _Image 23 – File bouncing pixel alteration map_ Smoothing filter can be an effective method due to its target on noise reduction and universal applicability with relatively low impact. As the resulting image can be more degraded relative to file bouncing this method may be more practical in some use cases than others. Its mitigation effectiveness can also be extremely high with only some edge case implementation of IS being less mitigated or not being mitigated. This filters application can also be extremely fast with the low overhead and so can be applied swiftly to many images. _Image 24 – Smooth filter pixel alteration map_ Finally, palette shuffling can be considered a standard practice for all palette based images. As the resulting image is identical to the original due to colour values not being altered between the two versions, this can have no negative effects on the processed image. As this can help mitigate IS using basic palette based methods there may not be issues in its deployment in most cases. However, as this does not consider potential resequencing it can be paired with a method which directly alters the colour values within the palette also to ensure improved mitigation. _Image 25 – Palette based image file bouncing pixel alteration map_ When testing the combination of file bouncing and smoothing, it was found that the resulting image when averaged over the test set did not breach the benchmarks. This shows that they can be used safely in combination with increased, but not significant, visual degradation. | **Smooth & Bounce** | Average PSNR | Average MAE | Average MSE | Average SSIM | | --- | --- | --- | --- | --- | | JPG | 0.410240571 | 0.009467143 | 0.000571441 | 0.942757143 | | TIF | 0.401245714 | 0.009065714 | 0.000418686 | 0.95563 | | BMP-P | 0.357571429 | 0.011932857 | 0.000535714 | 0.920425714 | | PNG-P | 0.357585714 | 0.011922857 | 0.000534286 | 0.920384286 | | BMP-TC | 0.401408571 | 0.008805714 | 0.000417257 | 0.955687143 | | PNG-TC | 0.401357143 | 0.008831429 | 0.000417257 | 0.955425714 | Table 12 – File Bouncing & Smoothing combination statistics _Image 26 – File bounce & smoothing combination statistics graph_ ## Weaponizing statistics against IS ### How this can be used in general These mitigation methods can be applied to almost many image files that are encountered. This can be done on a file by file basis or for entire directories or file streams. This would, for these batched image files, achieve a high level of confidence of any IS present being corrupted while the resulting image is just as legible and structurally like the original. When integrating mitigation there are several considerations. Certain image types may have structural capabilities that differ from other formats which will be required to be accounted for. Image formats such as GIF which can have multiple panels/images in the same file is one such format may require special handling. In addition, selection of tools and libraries which can achieve these image processing methods may not all work to the same standard calculations or base values when calculating alterations. As such, it is important when deploying such mitigation that the mechanisms employed are fully understood and the internal metrics and equations are in-line with the expected results. ### Pairing with Content Disarmament and Reconstruction technology – IS has nowhere to hide The IS solutions described herein can be used with CDR technology. In particular, file handling and specification adherence used in a CDR system can be incorporated in the IS solutions described herein. to the solution. As CDR can be tailored to handle file formats of a wide variety of types, including image types, we can ensure that the internal structure is as we expect and analyse each component individually. This can ensure that the image is within specification and not corrupt or masquerading as something else. This can also allow the handling of file type specific use cases such as multiple panels in the case of GIF, dynamic palette sizes and handling of images using transparency palettes. A benefit to incorporating CDR technology to the mitigation process is the handling of embedded images. Image files may be transmitted as standalone files; however, it is increasingly common that the image may be embedded within another file such as an Office document, PDF, or even an email file. CDR has the capacity to extract/export the embedded images, process them as standalone files and then reintegrate/import them to the original document without any deviation being detected. This is a complex process which is extremely difficult to achieve especially when binary format files are considered such as those in old Microsoft Office documents.  Although extraction and mitigation of embedded images may be possible without CDR, it can be highly likely that the document will either have the images removed or that the resulting document will be heavily modified. This is normally caused by misalignments in the graphical representation but can also be caused by errors in the memory offset of the embedded files within the binary structure of such files. CDR accounts for all these cases and allows for mass processing of both standalone and document files.  _Image 27 – CDR processing flow chart for both standalone and document files_ An additional benefit of CDR integration is the use of security profiles based on risk tolerance. Whereas automated deployment may be effective on batches, migration of files between different domains with potentially different risk tolerances can be complex. CDR is designed to sit on domain boundaries with each boundary having its own configuration. As such, we can employ palette shuffling on files entering high tolerance domains and palette shuffling, file bouncing and image smoothing on migration to low tolerance domains. Additionally or alternatively, filter intensities can be adjusted per profile also meaning that in the case of file bouncing the quality variable can be decreased in-line with risk tolerance to achieve a higher level of distortion on the images. This means that where security rather than usability is the primary concern alterations can be made appropriately. ## Conclusion  From our analysis, we have assessed and understood some of the most prevalent IS method that are commonly and widely available today. Our assessment shows that the ways IS encodes data into images is varied with differing levels of complexity depending on their aversion to detection or anticipated scrutiny. As such, a general mitigation solution is something that can be a valuable tool in any security arsenal to proactively defend against such secret communications channels. As described herein, mitigations can be applied in novel and unique ways to address different IS methods. Malicious actors may not expect the encoded images to be put through image filters and/or type conversions as a default action. Although they may encounter compression in some systems, most file transfer operations will not conduct any image specific operations. As such they have been able to use IS, with considerations for their target’s security infrastructure, at will. Employing these mitigation methods as a standard against all images that enter an estate will eliminate this covert channel in most cases, if not all of them.  As described herein, CDR with IS mitigation can address the use case of embedded files. As attackers typically try to blend in with anticipated traffic, sending an office document to a recipient on a regular basis as opposed to an image is less likely to draw attention. Additionally, embedded images are much harder to deal with due to the inherent issue of extraction and reapplication. Reapplication is a primary issue with any slight modification to images in documents potentially causing multiple format issues which can make them incomprehensible or even worse corrupt. CDR technology holds a potential solution to this by being built specifically to achieve this extraction and reapplication process at scale. By incorporating the image processing technologies described herein into the control flow of a CDR verification operation, IS mitigation can be achieved in many if not all formats whether they be standalone or embedded. This leaves malicious actors trying to use IS with nowhere to hide. #### Author: James Brimer (Associate Security Researcher) [Download PDF](https://app.box.com/s/umr4q8eh3bee66r6psc6blox9tqcarxs) ## Purpose outline This document presents a comprehensive analysis of the cybersecurity threats posed by QR codes embedded within digital files. It aims to: • Highlight the methods by which threat actors exploit QR codes in various file formats. • Discuss the limitations of traditional security measures in detecting and mitigating QR code-based threats. • Introduce Glasswall’s Content Disarm and Reconstruction (CDR) technology as a proactive solution to neutralize these threats. • Provide a statistical evaluation of existing QR code detection methods and their effectiveness. • Demonstrate practical mitigation techniques for removing malicious QR codes from digital files. ## Summary QR codes are now a common part of digital communication, but they also present a growing cybersecurity risk. Threat actors are increasingly embedding malicious payloads, like phishing links and malware, into QR codes found in everyday documents like PDFs and images. These threats often go undetected by traditional tools like antivirus software and sandboxing. This research paper highlights the limitations of reactive security approaches and makes the case for a proactive solution. It introduces **Glasswall’s Content Disarm and Reconstruction (CDR)** technology, which neutralizes threats at the file level, including those hidden in QR codes, by rebuilding documents to a safe and clean standard. With QR code-based attacks on the rise, organizations must shift from detection to prevention. Glasswall CDR offers a reliable, scalable way to eliminate hidden threats while preserving document fidelity, helping enterprises and governments to protect sensitive data and maintain trust in digital interactions. ## Introduction ### How QR codes are used by threat actors QR codes are a novel attack vector used by threat actors to persecute targets. QR codes enable threat actors to prompt staff to access malicious domains, which enables initial access via browser exploitation or extraction of a user’s credentials[1]. ### Hypothetical attacks Hypothetically, QR codes can be used for on-device exploitation, provided there is a vulnerability in the parser/decompressor on the user's device for the data type in the QR codes encoded region. QR codes also facilitate the transmission of any type of data across a trust boundary: this is because of an optional feature that can be found on page sixty of the 2015 QR code standard under Section 8: Structured Append. [2] This feature enables any file of any type to be encoded in up to sixteen QR codes, which can then be scanned in any order to recover the original file. The amount of data that individual QR codes can store is still relatively small, but the extraction of cryptographic keys or other sensitive data from an estate is achievable and can do a lot of damage. ### Problem statement Our primary objective is to detect and remove QR codes from image files. QR codes, whilst typically being clearly defined in an image, may also be occluded or blurred which needs to be accounted for. Detection and removal are a two-stage process which is: * Detection of the polygon which surrounds a QR code * Replacement of all the pixels in the polygon with black pixels Acceptance criteria are that the technique used to detect QR codes should have a true positive rate of 99% and a false positive rate of 0.1%. The technique used should only parse structural data and not attempts content decompression or processing due to potential security concerns. This research doesn’t deal with the detection and mitigation of micro QR codes. ## QR Code ISO Standard ### Relevant QR code structural data A QR code is a square matrix of modules. A module is a square pattern made up exclusively of zeros or ones. Three unique finder patterns, detailed in image 2, are located at the upper-left, upper-right and lower-left corners of the symbol. These finder patterns enable a QR code’s position, size and inclination to be determined. An example QR code is shown in image 1. QR codes can be reversed, in which case the lower finder pattern is placed in the lower right-hand corner of the QR code rather than the left. QR codes can be inverted, also known as reflectance reversed, where normally nominally black modules are white and vice versa. _Image 1: An example QR code._ _Image 2: The structure of a QR code finder pattern in a non-inverted QR code. This diagram can be found under Section 6.3.3 Finder Pattern of [2]_ Each finder pattern is made up of three concentric squares/modules. The innermost module is square A, the next module is square B, and the outermost module is square C. A horizontal or vertical line through a finder pattern produces a `b:w:b:w:b` pattern of modules. This `b:w:b:w:b` pattern is used by the reference decode algorithm described in the next section to detect a QR code in an image. The ratios between each run of black and white modules in the `b:w:b:w:b` pattern is expected to be `1:1:3:1:1`. Pixel runs, which could potentially be part of a finder pattern, should be relatively rare in scenes which don't contain a QR code. ### Reference decode algorithm The reference decode algorithm for QR codes can be found under Section 12: Reference Decode Algorithm of the 2015 QR Code ISO Standard[2]. Relevant to this report are steps (b) and (c) of the reference decode algorithm. These are the first steps executed by the algorithm to find the locations of QR codes and two other finder patterns once one has been found. A polygon which surrounds the code can be found once the finder patterns have all been located by using them as corner locations. The polygon which surrounds the code can be used to specify where in the image to overwrite the QR code with black pixels. _Image 3: Steps (b) and (c) of the QR code reference decode algorithm. See Section 12 Reference Decode Algorithm for QR Code of [2]_ _Image 4: The lines used by the QR code reference decode algorithm to locate the two other finder patterns once one finder pattern has been found._ In image 4, the red lines are for illustrative purposes only: they don’t pass through the center of each finder pattern. These lines must be orthogonal (at right angles) for the finder patterns to be part of a standard conformant QR code. ## Statistical evaluation of QR Code detection methods The test dataset is made up of images which belong to one of four categories: * Images which don’t contain QR codes (n = 153). * Images which only contain one manually generated QR code (n = 97) * Real-world scenes which contain one or more detectable QR codes (n = 32). * Real-world scenes which contain occluded or blurred QR codes (n = 24). There are 153 images which don’t contain QR codes and 153 which do. The dataset is unbalanced e.g. The portion of the test dataset which contains QR codes is made up of multiple categories which are not the same size, some categories contain images which have a much higher resolution than others, etc. This will affect the explanatory power of the statistics in this report. I’m not qualified to comment on how the lack of balance will affect the statistics. _Image 5: An example image from each category in the test dataset._ All the images which are not manually generated QR codes were saved from Unsplash[3], which provides copyright free images. QR codes are determined to be recognisable if an iPhone SE 2nd Ed which runs iOS 18.2.1 can detect and decode any QR codes in the image using the Camera app. As such, the Camera app served as the ground truth for whether an image contains a readable QR code. The Camera app uses industry-standard technology to detect QR codes in images and emulates the expected behaviour of QR code recipients. The frameworks used to write the Camera app implement state-of-the-art algorithms: if the Camera app can’t detect a QR code, then it’s unlikely any detector can. Four well-known QR code detection libraries were run over the test dataset: [Quirc](https://github.com/dlbeer/quirc), [Pyzbar](https://github.com/naturalhistorymuseum/pyzbar/tree/master), [QRDet](https://github.com/eric-canas/QRdet) and the [WeChat QR code detector](https://github.com/opencv/opencv_contrib/tree/4.x/modules/wechat_qrcode). Detection Method| Accuracy (2 d.p.)| Sensitivity (2 d.p.)| Specificity (3 d.p.) ---|---|---|--- Pyzbar| 0.82| 0.66| 0.980 QRDet| 0.91| 0.84| 0.987 Quirc| 0.84| 0.68| 1 WeChat| 0.91| 0.66| 1 _Table 1: The accuracy, sensitivity, and specificity of each of the tested QR code detection methods._ The error matrices are shown below. N = 306 | Predicted | - --- | --- | --- Actual | QR Code | No QR Code QR code | 104 | 49 No QR code | 0 | 153 | _Table 2: The error matrix of Quirc when it’s run over the test dataset._ | N = 306 | Predicted | - | | --- | --- | --- | | Actual | QR Code | No QR Code | | QR code | 101 | 52 | | No QR code | 3 | 150 | _Table 3: The error matrix of Pyzbar when it’s run over the test dataset._ | N = 306 | Predicted | - | | --- | --- | --- | | Actual | QR Code | No QR Code | | QR code | 128 | 25 | | No QR code | 2 | 151 | _Table 4: The error matrix of QRDet when it’s run over the test dataset._ | N = 306 | Predicted | - | | --- | --- | --- | | Actual | QR Code | No QR Code | | QR code | 102 | 51 | | No QR code | 0 | 153 | _Table 5: The error matrix of the WeChat QR code detector when it’s run over the test dataset._ The WeChat QR code detector’s inference speed was much slower than that of YOLO, the ML model which underpins QRDet. The QRDet test ran in less than ten minutes. The test using the WeChat QR code detector had to be run in excess of eight hours. ## Detection of generated QR codes only A subset of the test dataset made up of images which belong to one of two categories: * Images which don’t contain QR codes (n = 97). * Images which only contain one manually generated QR code (n = 97) Was used to evaluate how well Quirc and Pyzbar perform on generated QR codes only. If we want to detect QR codes in a non-adversarial scenario, where we only try to detect and clean generated QR codes, it’s preferable to use a near real-time, deterministic detector which is easy to inspect. Detection Method| Accuracy (2 d.p.)| Sensitivity (2 d.p.)| Specificity (3 d.p.) ---|---|---|--- Pyzbar| 0.99| 0.99| 1 Quirc| 1| 1| 1 _Table 6: The accuracy, sensitivity, and specificity of Quirc and Pyzbar when tested against generated QR codes only._ Even though it uses a more sophisticated approach to scanning an image for potential QR codes, pyzbar performs worse than quirc. The cause of this decreased accuracy is yet to be determined. | N = 194 | Predicted | - | | --- | --- | --- | | Actual | QR Code | No QR Code | | QR code | 97 | 0 | | No QR code | 0 | 97 | _Table 7: The error matrix of Quirc when tested against generated QR codes only._ | N = 194 | Predicted | - | | --- | --- | --- | | Actual | QR Code | No QR Code | | QR code | 97 | 0 | | No QR code | 1 | 96 | _Table 8: The error matrix of Pyzbar when tested against generated QR codes only._ ## Mitigation Once the polygon which bounds a QR code has been found, removal of the QR code from the image using OpenCV can be done with one function call in Python: _Image 6: How to call cv2.fillPoly to destroy a QR code_ The cv2.fillPoly function accepts a polygon as an argument. The polygon is passed into cv2.fillPoly as a list of vertex coordinates where each vertex is a list which contains an x and a y value. Each pixel in the polygon is overwritten with a pixel of the colour of the color argument. In this case each pixel is replaced with a black pixel, as the RGB value for a black pixel is (0, 0, 0). The image argument in the call to cv2.fillPoly is a NumPy representation of the original image in raw pixel format. _Image 7: A cleaned image where multiple QR codes have been overwritten with black pixels._ ## Conclusion When dealing with adversarial use-cases of QR code use, it is not recommended to use Quirc, Pyzbar or related QR code detectors which rely on simple approaches to object segmentation. Such approaches struggle with simple, reversible transformations that cause a QR code not to be detected even when it’s standard conformant and recognizable by other detectors. It would be easy for an adversary to bypass these detection methods. An example of how this can be achieved is shown in the Appendix: _"How use of Otsu’s Method Impacts QR Code Detection"_ section of this report. The accuracy of Quirc and Pyzbar over the full test dataset, 0.84 and 0.82, respectively, is worse than QRDet and below what we considered the acceptance criteria. Both QRDet and the WeChat QR code detector use a convolutional neural network (CNN), a type of ML model, to detect QR codes in an image. CNNs are state-of-the-art in computer vision. ML models are deployed widely and effectively across the commercial space to detect QR codes e.g. Google’s QR code detection API uses ML to detect QR codes in an image. [4] Because of this we recommend deploying an ML-based solution for the adversarial actor use-case. The performance of QRDet compared to Quirc and Pyzbar on the full test dataset shows the advantages of this approach. QRDet is more sensitive than the WeChat detector but less specific, see Table 1, which means that QRDet is more likely to detect a QR code in an image than the WeChat detector, but it’s also more likely to detect a QR code in an image which doesn’t contain a QR code and generate a false positive or remove non-QR image content. On generated QR codes, Quirc marginally outperforms Pyzbar, however even though Quirc detects QR codes in all the images which contain generated QR codes, the returned polygons do not always correctly bind some QR codes. More research is needed to understand why this is the case. Although the polygon returned in these cases does not cover all the QR code, enough of the QR code is covered by the polygon that mitigation would still be effective. The polygons returned by Pyzbar correctly bound every generated QR code, however on the full dataset Pyzbar returns three false positives whereas Quirc returns none. We would therefore recommend the use of Quirc. Implementation should use a more sophisticated algorithm to threshold the input image as the use of Otsu’s method, which is used by Quirc to threshold an input image, can have a detrimental effect on QR code detection. See the Appendix: _"How Otsu’s Method impacts QR code detection"_ section of this report to understand why this is the case. #### References | [1] | Microsoft, "Hunting for QR code AiTm Phishing and User Compromise," Microsoft, 12 February 2024. [Online]. Available: https://techcommunity.microsoft.com/blog/microsoftsecurityexperts/hunting-for-qr-code-aitm-phishing-and-user-compromise/4053324. [Accessed 03 February 2025]. | | [2] | ISO, "ISO/IEC 18004:2015(E) Information Technology – Automatic identification and data capture techniques – QR Code barcode symbology specification," ISO, 2015. | | [3] | Various, "Unsplash. The internet's source for visuals. Powered by creators everywhere.," Unsplash, [Online]. Available: https://unsplash.com/. [Accessed 03 February 2025]. | | [4] | Google, "Barcode Scanning," Google, 21 January 2025. [Online]. Available: https://developers.google.com/ml-kit/vision/barcode-scanning. [Accessed 03 February 2025]. | ## Appendix ### How does the use of Otsu’s method affect QR code detection? The image intensity histogram of an image is a graph which shows the number of pixels in an image which take a given pixel intensity value. In RGB images, each channel has its own intensity histogram: _Image 8: The image intensity histograms of the red, green and blue channels of Image 9._ A binary threshold is a pixel value used to transform an image into a binary image, i.e. An image where each pixel is zero or one. To transform an image into a binary image given a threshold value, all the pixels with a value strictly less than the threshold are set to zero. All other pixels are set to one. Pixels which are zero are said to be part of the background of the thresholded image. Pixels which are one are said to belong to the foreground of the thresholded image. A threshold is typically computed by an algorithm, which can be global or local. Global algorithms derive a threshold from the properties of an image’s intensity histogram. Local algorithms use the properties of a region of an image to derive a threshold for that region only. Global algorithms often don’t work well when used to process complex scenes, such as scenes where light causes part of an object to appear to be in both the foreground and the background of the image because of the different pixel intensity values across the object. In contrast, local algorithms perform better when run over complex scenes but are computationally expensive, so they don’t tend to be used when performance needs to be near-real or real-time. For systems which need to have a high throughput the choice between using a global or local algorithm will depend on the amount of compute available. Otsu’s method is an automatic global thresholding algorithm that operates on grayscale images. Otsu’s method iterates through all possible threshold values: 0 to 255. For each threshold, the intra-class variance of the background and foreground classes is recorded. Once all possible thresholds have been tried, the threshold that minimizes the intra-class variance of the two classes is returned. The intra-class variance of a class is a descriptive statistic which quantifies how well members of a class resemble each other. Minimization of the intra-class variance of the background and foreground classes is useful when an image’s intensity histogram is bimodal, as this ensures that the foreground and background classes will cluster around the two modes. Advanced versions of Otsu’s method exist, which can deal with some noise in the image. A connected component is a set of adjacent pixels which are the same colour. An example of a connected component in Image 9 is square C of the upper-right finder pattern. Connected component analysis is important in classical approaches to QR code detection because it allows simple tests to be performed to see if a run of pixels belongs to a finder pattern e.g. In a run of pixels which follow the 1:1:3:1:1 pattern, which defines a finder pattern row or column if the black pixels all belong to the same connected component then the run of pixels cannot be part of a QR code finder pattern. Most non-proprietary QR code detectors use a standard computer vision pipeline to pre-process an image before the reference decode algorithm is run over the image. The reference decode algorithm described in Section 12 of [2] is used to locate and decode QR codes in an image; steps (b) and (c) of the algorithm are reproduced in Image 3. Steps (b) and (c) are used to locate a QR code in an image. A common approach is to use Otsu’s method to transform a grayscale image into a binary image, after which connected component analysis or edge detection is used to emphasize potential finder patterns in the image. The use of Otsu’s method to threshold an image can be problematic, as Otsu’s method performs very poorly when used to threshold more complex scenes (both Quirc and Pyzbar use Otsu’s method). This often disrupts connected component analysis, which in turn disrupts QR code detection. _Image 9: A QR code where the upper-left finder pattern is partly covered by a shadow._ _Image 10: Otsu’s method produces a poorly binarized image when used to threshold Image 9._ The shadow in Image 9 has caused part of the upper-left finder pattern to be classified as part of the foreground in Image 10. This means that the finder pattern is no longer standard conformant. _Image 11: The grayscale image intensity histogram of Image 9._ The image intensity histogram of Image 9 - see Image 11 - is not bimodal. This breaks the key assumption of Otsu’s method. How should pixels clustered around one of the three modes be clustered around the other two? If pixels around the mode are not clustered exclusively around one of the other modes, then pixels associated with the mode will be assigned to both the foreground and the background, which leads to poor binarization. If they are clustered around only one of the other modes, this forces pixels which may belong to both the background and the foreground to be assigned to one class. This removes important details: _Image 14: Image 9 thresholded with a pixel intensity value of 150._ _Image 15: Image 9 thresholded with a pixel intensity value of 200._ In both cases, the binarization isn’t ideal. Other algorithms often find better thresholds: _Image 16: A binary image produced by the application of a normalized 3x3 box filter and Gaussian Adaptive thresholding to Image 9, where the block size is 3 and C = 0._ In Image 16 the upper-left finder pattern is much better preserved when compared to Image 10. This is a good example of how local methods can outperform global methods on more complex scenes, and how other methods outperform Otsu’s method when it comes to preservation of a QR code’s key features, such as the codes finder patterns. ### Example false positives _Image 17: A false positive returned by Pyzbar._ It is currently unclear why the polygon in Image 17 has five sides when it should only have four. This oddity has been noted for further investigation in future work. _Image 18: A false positive returned by QRDet._