Google Cloud Storage Integration
    • PDF

    Google Cloud Storage Integration

    • PDF

    Article summary

    Objective

    Set up automated file sanitization in Google Cloud Storage using Glasswall Halo API via a Google Cloud Run service and Eventarc.

    Google Cloud Storage Integration Guide

    Prerequisites

    Notes:

    • Before you begin, ensure that all the configuration parameters are correctly named and added to the Google Cloud Run application since any mismatches will lead to failures.
    • We've only provided a sample code to demonstrate this use case. It only handles flat files and does not have sophisticated logging mechanisms.

    Step 1 - Setup

    Google provides a similar scenario that can be used to get started. Please follow the before you begin guide for permissions, service accounts and command line setup.

    Once completed you should be ready to deploy the example code.

    Also ensure that you have done the following:

    1. Sign in to Google Cloud Console.
    2. Clone sample code to a working folder.
    3. Run the following commands
    cd ./GCloudStorageEvents/dotnet
    gcloud auth login
    gcloud config set project [PROJECT_ID]
    

    'PROJECT_ID' can be found in the welcome page of the Google Cloud Platform Portal.

    Step 2 - Deploy Google Cloud Run

    In the following steps, please replace [SERVICE] with the desired name of the deployed cloud run application.

    1. To deploy from command line, the gcloud run deploy command is used.
    gcloud run deploy ${SERVICE_NAME} --source . \
     --region ${SERVICE_REGION} \
     --set-env-vars OutputBucket=${OUTPUT_BUCKET_NAME} \
     --set-env-vars HALO_URL=${HALO_URL} \
     --set-env-vars HALO_USERNAME=${HALO_USERNAME} \
     --set-env-vars HALO_PASSWORD=${HALO_PASSWORD}
    
    1. Configuring the application is also done at this stage, the following arguments need to be set by replacing. Please replace the varaibles enclosed by '${}' with the value.
    KeyValue
    SERVICE_NAMEThis is the name of gcloud run service to be created
    SERVICE_REGIONThis is the name of region in which the gcloud run service will be created
    OUTPUT_BUCKET_NAMEDestination storage bucket name. (Should exist already)
    HALO_URLURL for cdr-file endpoint (e.g. https://api.glasswall.com/api/v3/cdr-file)
    HALO_USERNAMEUsername for Halo API basic auth
    HALO_PASSWORDPassword for Halo API basic auth

    Please refer to gcloud command documentation for more information.

    Step 3 - Set up storage events

    In order to set up storage events, the following command will need to be run to set up Google Eventarc.

    Note: this command assumes that the buckets and the Cloud Run service are in the same region - the command may need further customization if this is not the case.

    gcloud eventarc triggers create ${SERVICE_NAME} \
     --destination-run-service=${SERVICE_NAME} \
     --destination-run-region=${SERVICE_REGION} \
     --destination-run-path="/" \
     --location=${SERVICE_REGION} \
     --event-filters="type=google.cloud.storage.object.v1.finalized" \
     --event-filters="bucket=${INPUT_BUCKET_NAME}" \
     --service-account=${PROJECT_NUMBER}-compute@developer.gserviceaccount.com
    

    See the following table for argument parameters.

    KeyValue
    SERVICE_NAMEThis is the name of gcloud run service, for simplicity the same name is used for the event arc resource
    SERVICE_REGIONThis is the name of region in which the gcloud run sits, this is also used for the --location flag, which is used to place the eventarc resource.
    INPUT_BUCKET_NAMESource storage bucket name. This bucket contains the files to be processed. (Should exist already)
    PROJECT_NUMBERThis is the unique project number associated with your google project, can be found at the welcome screen of the google cloud portal

    This will automatically set up a event arc resource that will listen to changes in the configured bucket. Internally it will publish to a Pub/Sub message queue and automatically call and retry invocations to the Google Cloud Run service.

    This may take up to 2 minutes to take affect. After which, any new documents on the storage will trigger.

    Step 4 - Test application

    Finally to test the application, you can simply place files in the input bucket. For supported files this will then result in rebuilt files created in the output bucket.

    To monitor logs on the application you can either navigate to the cloud run resource or run the following command:

    gcloud config set run/region ${SERVICE_REGION}
    gcloud auth application-default login
    gcloud beta run services logs tail ${SERVICE_NAME} --project ${PROJECT_ID}
    

    Reference arguments:

    KeyValue
    SERVICE_NAMEThis is the name of gcloud run service
    SERVICE_REGIONThis is the name of region in which the gcloud run sits, this is also used for the --location flag, which is used to place the eventarc resource.
    PROJECT_IDThis is the unique project id associated with your google project, can be found at the welcome screen of the google cloud portal

    Please refer to gcloud logging documentation for more information and options.


    Was this article helpful?

    What's Next