Google Cloud Storage Integration

Prev Next

Objective

Set up automated file sanitization in Google Cloud Storage using Glasswall Halo API via a Google Cloud Run service and Eventarc.

Google Cloud Storage Integration Guide

Prerequisites

Notes:

  • Before you begin, ensure that all the configuration parameters are correctly named and added to the Google Cloud Run application since any mismatches will lead to failures.
  • We've only provided a sample code to demonstrate this use case. It only handles flat files and does not have sophisticated logging mechanisms.

Step 1 - Setup

Google provides a similar scenario that can be used to get started. Please follow the before you begin guide for permissions, service accounts and command line setup.

Once completed you should be ready to deploy the example code.

Also ensure that you have done the following:

  1. Sign in to Google Cloud Console.
  2. Clone sample code to a working folder.
  3. Run the following commands
cd ./GCloudStorageEvents/dotnet
gcloud auth login
gcloud config set project [PROJECT_ID]

'PROJECT_ID' can be found in the welcome page of the Google Cloud Platform Portal.

Step 2 - Deploy Google Cloud Run

In the following steps, please replace [SERVICE] with the desired name of the deployed cloud run application.

  1. To deploy from command line, the gcloud run deploy command is used.
gcloud run deploy ${SERVICE_NAME} --source . \
 --region ${SERVICE_REGION} \
 --set-env-vars OutputBucket=${OUTPUT_BUCKET_NAME} \
 --set-env-vars HALO_URL=${HALO_URL} \
 --set-env-vars HALO_USERNAME=${HALO_USERNAME} \
 --set-env-vars HALO_PASSWORD=${HALO_PASSWORD}
  1. Configuring the application is also done at this stage, the following arguments need to be set by replacing. Please replace the varaibles enclosed by '${}' with the value.
Key Value
SERVICE_NAME This is the name of gcloud run service to be created
SERVICE_REGION This is the name of region in which the gcloud run service will be created
OUTPUT_BUCKET_NAME Destination storage bucket name. (Should exist already)
HALO_URL URL for cdr-file endpoint (e.g. https://api.glasswall.com/api/v3/cdr-file)
HALO_USERNAME Username for Halo API basic auth
HALO_PASSWORD Password for Halo API basic auth

Please refer to gcloud command documentation for more information.

Step 3 - Set up storage events

In order to set up storage events, the following command will need to be run to set up Google Eventarc.

Note: this command assumes that the buckets and the Cloud Run service are in the same region - the command may need further customization if this is not the case.

gcloud eventarc triggers create ${SERVICE_NAME} \
 --destination-run-service=${SERVICE_NAME} \
 --destination-run-region=${SERVICE_REGION} \
 --destination-run-path="/" \
 --location=${SERVICE_REGION} \
 --event-filters="type=google.cloud.storage.object.v1.finalized" \
 --event-filters="bucket=${INPUT_BUCKET_NAME}" \
 --service-account=${PROJECT_NUMBER}-compute@developer.gserviceaccount.com

See the following table for argument parameters.

Key Value
SERVICE_NAME This is the name of gcloud run service, for simplicity the same name is used for the event arc resource
SERVICE_REGION This is the name of region in which the gcloud run sits, this is also used for the --location flag, which is used to place the eventarc resource.
INPUT_BUCKET_NAME Source storage bucket name. This bucket contains the files to be processed. (Should exist already)
PROJECT_NUMBER This is the unique project number associated with your google project, can be found at the welcome screen of the google cloud portal

This will automatically set up a event arc resource that will listen to changes in the configured bucket. Internally it will publish to a Pub/Sub message queue and automatically call and retry invocations to the Google Cloud Run service.

This may take up to 2 minutes to take affect. After which, any new documents on the storage will trigger.

Step 4 - Test application

Finally to test the application, you can simply place files in the input bucket. For supported files this will then result in rebuilt files created in the output bucket.

To monitor logs on the application you can either navigate to the cloud run resource or run the following command:

gcloud config set run/region ${SERVICE_REGION}
gcloud auth application-default login
gcloud beta run services logs tail ${SERVICE_NAME} --project ${PROJECT_ID}

Reference arguments:

Key Value
SERVICE_NAME This is the name of gcloud run service
SERVICE_REGION This is the name of region in which the gcloud run sits, this is also used for the --location flag, which is used to place the eventarc resource.
PROJECT_ID This is the unique project id associated with your google project, can be found at the welcome screen of the google cloud portal

Please refer to gcloud logging documentation for more information and options.