Azure Blob Storage Integration
    • PDF

    Azure Blob Storage Integration

    • PDF

    Article summary

    Objective

    Set up automated file sanitization in Azure Blob Storage using Glasswall Halo API via the Event Grid functionality and Service Bus queues.

    Azure Blob Storage Integration Guide


    Prerequisites

    • Azure Portal account
    • Source Azure Blob storage account
    • Destination Azure Blob storage account
    • Glasswall Example Code

    Notes:

    • Ensure that all the configuration parameters are correctly named and added to the Azure Function App. Any mismatches will lead to failures.
    • We've only provided a sample code to demonstrate this use case. It only handles flat files and does not have sophisticated logging mechanisms.

    Step 1 - Create a Service Bus namespace

    First, you need to create a Service Bus namespace and a queue so that you can recieve the Event Grid messages.

    Note: the tier and settings are related to your load or IT.

    1. Sign in to Azure Portal.

    2. Navigate to All services > Integration > Service Bus.

    3. In the Basics tag of the Create namespace page, create a namespace.

      Note: the sample code provided will work with all tiers so select the settings appropriate for your usage, and enter all mandatory configuration.

    ABS%20-%20Step%201c



    For more detailed information about creating a Service Bus namespace and queue, please refer to Azure Documentation.

    Step 2 - Create a Service Bus queue

    Once the namespace is created, we need to create a queue to post the events to.

    1. On the Service Bus Namespace page, select Queues in the left navigational menu.

    2. On the Queues page, select + Queue on the toolbar.

      ABS%20-%20Step%201d

      Note: as with the namespace, the settings are down to preference – for this sample we are simply taking the defaults.

    3. Enter a name for the queue, and leave the other values with their defaults.

      The name you choose doesn't matter as the sample code allows it to be configured to whatever is set.

      ABS%20-%20Step%201e

    4. Select Create.



    For more detailed information about creating a Service Bus namespace and queue, please refer to Azure Documentation.


    Step 3 - Set up source Blob storage account events

    Once the Service Bus namespace and queue is created, we need to create and configure our source Blob storage account to emit events as messages to the queue.

    It's possible for these events to directly trigger an Azure function; however any event must be acknowledged within 30 seconds – which while most files are processed in seconds, could mean, that under load or with large files this becomes an issue. To avoid this, we recommend you get the Event Grid to deliver the message to a queue and for the function to consume from that. This gives the added resilience and flexibility of Service Bus.

    1. To set up events, navigate to the source storage account and select Events > More Options > Service Bus Queue.

    ABS%20-%20Step%202a

    1. Enter a Name under Event Subscription Details.

    2. Enter a System Topic Name under Topic Details.

    3. From the Filter to Event Type dropdown, select Blob Created.

      This means there will be a message for every new file in Blob Storage across the storage account.

    4. Finally, choose the queue you created in the previous step under Endpoint Detail.

    ABS - Step 2b.jpg

    ABS%20-%20Step%202c

    A new service bus message is now placed in the queue for every file uploaded to that storage account, which will be consumed via a function we will create next.


    Step 4 - Create a function app

    You must have a function app to host the execution of your functions. The Azure function that consumes the service bus messages you just created will be able to use the Event Grid details to identify the new file.

    Glasswall has a working example in C# that performs the following:

    • Identifies the new file.
    • Accesses the file.
    • Sends it in a request to Glasswall Halo’s Synchronous API.
    • Creates (if it doesn’t already exist) a container of the same name in the destination storage account.
    • Writes either the regenerated file (as the same name as original) or a text file if it couldn’t be regenerated.
    1. To create a function app, navigate to the Create a resource page.

    2. In the New page, select Compute > Function App.

    3. The settings below will work for Glasswall's example function. You should adjust these settings to suit your intended function code and hosting option:

      ABS - Step 3a.jpg



    For more detailed information about creating a function app, please refer to Azure Documentation.

    Step 5 - Create a function

    Once the function app is created, you can create the function.

    1. In the function app overview, it provides guidance for various methods of getting your code deployed.

      Choose the method most suitable for your selected codebase.

    ABS%20-%20Step%203b

    ABS%20-%20Step%203c

    1. If you are using Glasswall’s sample code then the following configuration needs to be added to the function app via Settings > Configuration.

      All values can be sourced from the relevant resources in the Azure Portal. For access to Glasswall Halo either use your instance and authentication, or for trial purposes contact our sales team for basic authentication credentials to use our public endpoint.

      Configuration

      KeyValue
      AZURE_SERVICEBUS_CONNECTIONSTRINGConnection string for service bus
      AZURE_SERVICEBUS_NEWFILES_QUEUENAMEQueue name for new blob events
      AZURE_STORAGE_ACCOUNT_NAMESource storage account name
      AZURE_STORAGE_ACCOUNT_KEYSource storage account key
      AZURE_STORAGE_DESTINATION_CONNECTIONSTRINGDestination storage account connection string
      CDR_USERNAMEUsername for Halo API basic auth
      CDR_PASSWORDPassword for Halo API basic auth
      CDR_URLURL for cdr-file endpoint (e.g. https://api.glasswall.com/api/v3/cdr-file)

    ABS%20-%20Step%203d



    With all these steps followed, placing a file in the source Azure Blob Storage account will result in either a regenerated clean document or a benign text file with an error message.


    Was this article helpful?