Azure Blob Storage Integration

Objective

Set up automated file sanitization in Azure Blob Storage using Glasswall Halo API via the Event Grid functionality and Service Bus queues.

Azure Blob Storage Integration Guide

• Prerequisites
• Step 1 - Create a Service Bus namespace
• Step 2 - Create a Service Bus queue
• Step 3 - Set up source Blob storage account events
• Step 4 - Create a function app
• Step 5 - Create a function

Prerequisites

• Azure Portal account
• Source Azure Blob storage account
• Destination Azure Blob storage account
• Glasswall Example Code

Notes:

• Ensure that all the configuration parameters are correctly named and added to the Azure Function App. Any mismatches will lead to failures.
• We've only provided a sample code to demonstrate this use case. It only handles flat files and does not have sophisticated logging mechanisms.

Step 1 - Create a Service Bus namespace

First, you need to create a Service Bus namespace and a queue so that you can recieve the Event Grid messages.

Note: the tier and settings are related to your load or IT.

1. Sign in to Azure Portal.

2. Navigate to All services > Integration > Service Bus.

3. In the Basics tag of the Create namespace page, create a namespace.

   Note: the sample code provided will work with all tiers so select the settings appropriate for your usage, and enter all mandatory configuration.

For more detailed information about creating a Service Bus namespace and queue, please refer to Azure Documentation.

Step 2 - Create a Service Bus queue

Once the namespace is created, we need to create a queue to post the events to.

1. On the Service Bus Namespace page, select Queues in the left navigational menu.

2. On the Queues page, select + Queue on the toolbar.

   

   Note: as with the namespace, the settings are down to preference – for this sample we are simply taking the defaults.

3. Enter a name for the queue, and leave the other values with their defaults.

   The name you choose doesn't matter as the sample code allows it to be configured to whatever is set.

   

4. Select Create.

For more detailed information about creating a Service Bus namespace and queue, please refer to Azure Documentation.

Step 3 - Set up source Blob storage account events

Once the Service Bus namespace and queue is created, we need to create and configure our source Blob storage account to emit events as messages to the queue.

It's possible for these events to directly trigger an Azure function; however any event must be acknowledged within 30 seconds – which while most files are processed in seconds, could mean, that under load or with large files this becomes an issue. To avoid this, we recommend you get the Event Grid to deliver the message to a queue and for the function to consume from that. This gives the added resilience and flexibility of Service Bus.

1. To set up events, navigate to the source storage account and select Events > More Options > Service Bus Queue.

2. Enter a Name under Event Subscription Details.

3. Enter a System Topic Name under Topic Details.

4. From the Filter to Event Type dropdown, select Blob Created.

   This means there will be a message for every new file in Blob Storage across the storage account.

5. Finally, choose the queue you created in the previous step under Endpoint Detail.

A new service bus message is now placed in the queue for every file uploaded to that storage account, which will be consumed via a function we will create next.

Step 4 - Create a function app

You must have a function app to host the execution of your functions. The Azure function that consumes the service bus messages you just created will be able to use the Event Grid details to identify the new file.

Glasswall has a working example in C# that performs the following:

• Identifies the new file.
• Accesses the file.
• Sends it in a request to Glasswall Halo’s Synchronous API.
• Creates (if it doesn’t already exist) a container of the same name in the destination storage account.
• Writes either the regenerated file (as the same name as original) or a text file if it couldn’t be regenerated.

1. To create a function app, navigate to the Create a resource page.

2. In the New page, select Compute > Function App.

3. The settings below will work for Glasswall's example function. You should adjust these settings to suit your intended function code and hosting option:

   

For more detailed information about creating a function app, please refer to Azure Documentation.

Step 5 - Create a function

Once the function app is created, you can create the function.

1. In the function app overview, it provides guidance for various methods of getting your code deployed.

   Choose the method most suitable for your selected codebase.

2. If you are using Glasswall’s sample code then the following configuration needs to be added to the function app via Settings > Configuration.

   All values can be sourced from the relevant resources in the Azure Portal. For access to Glasswall Halo either use your instance and authentication, or for trial purposes contact our sales team for basic authentication credentials to use our public endpoint.

   Configuration

   Key	Value
   AZURE_SERVICEBUS_CONNECTIONSTRING	Connection string for service bus
   AZURE_SERVICEBUS_NEWFILES_QUEUENAME	Queue name for new blob events
   AZURE_STORAGE_ACCOUNT_NAME	Source storage account name
   AZURE_STORAGE_ACCOUNT_KEY	Source storage account key
   AZURE_STORAGE_DESTINATION_CONNECTIONSTRING	Destination storage account connection string
   CDR_USERNAME	Username for Halo API basic auth
   CDR_PASSWORD	Password for Halo API basic auth
   CDR_URL	URL for cdr-file endpoint (e.g. https://api.glasswall.com/api/v3/cdr-file)

With all these steps followed, placing a file in the source Azure Blob Storage account will result in either a regenerated clean document or a benign text file with an error message.