Google Cloud Storage Integration

Objective

Set up automated file sanitization in Google Cloud Storage using Glasswall Halo API via a Google Cloud Run service and Eventarc.

Google Cloud Storage Integration Guide

• Prerequisites
• Step 1 - Setup
• Step 2 - Deploy Google Cloud Run
• Step 3 - Set up storage events
• Step 4 - Test application

Prerequisites

• Google cloud console account
• Source Google Cloud bucket
• Destination Google Cloud bucket
• gcloud CLI
• Glasswall Example Code

Notes:

• Before you begin, ensure that all the configuration parameters are correctly named and added to the Google Cloud Run application since any mismatches will lead to failures.
• We've only provided a sample code to demonstrate this use case. It only handles flat files and does not have sophisticated logging mechanisms.

Step 1 - Setup

Google provides a similar scenario that can be used to get started. Please follow the before you begin guide for permissions, service accounts and command line setup.

Once completed you should be ready to deploy the example code.

Also ensure that you have done the following:

1. Sign in to Google Cloud Console.
2. Clone sample code to a working folder.
3. Run the following commands

cd ./GCloudStorageEvents/dotnet
gcloud auth login
gcloud config set project [PROJECT_ID]

'PROJECT_ID' can be found in the welcome page of the Google Cloud Platform Portal.

Step 2 - Deploy Google Cloud Run

In the following steps, please replace [SERVICE] with the desired name of the deployed cloud run application.

1. To deploy from command line, the gcloud run deploy command is used.

gcloud run deploy ${SERVICE_NAME} --source . \
 --region ${SERVICE_REGION} \
 --set-env-vars OutputBucket=${OUTPUT_BUCKET_NAME} \
 --set-env-vars HALO_URL=${HALO_URL} \
 --set-env-vars HALO_USERNAME=${HALO_USERNAME} \
 --set-env-vars HALO_PASSWORD=${HALO_PASSWORD}

2. Configuring the application is also done at this stage, the following arguments need to be set by replacing. Please replace the varaibles enclosed by '${}' with the value.

Please refer to gcloud command documentation for more information.

Step 3 - Set up storage events

In order to set up storage events, the following command will need to be run to set up Google Eventarc.

Note: this command assumes that the buckets and the Cloud Run service are in the same region - the command may need further customization if this is not the case.

gcloud eventarc triggers create ${SERVICE_NAME} \
 --destination-run-service=${SERVICE_NAME} \
 --destination-run-region=${SERVICE_REGION} \
 --destination-run-path="/" \
 --location=${SERVICE_REGION} \
 --event-filters="type=google.cloud.storage.object.v1.finalized" \
 --event-filters="bucket=${INPUT_BUCKET_NAME}" \
 --service-account=${PROJECT_NUMBER}-compute@developer.gserviceaccount.com

See the following table for argument parameters.

This will automatically set up a event arc resource that will listen to changes in the configured bucket. Internally it will publish to a Pub/Sub message queue and automatically call and retry invocations to the Google Cloud Run service.

This may take up to 2 minutes to take affect. After which, any new documents on the storage will trigger.

Step 4 - Test application

Finally to test the application, you can simply place files in the input bucket. For supported files this will then result in rebuilt files created in the output bucket.

To monitor logs on the application you can either navigate to the cloud run resource or run the following command:

gcloud config set run/region ${SERVICE_REGION}
gcloud auth application-default login
gcloud beta run services logs tail ${SERVICE_NAME} --project ${PROJECT_ID}

Reference arguments:

Please refer to gcloud logging documentation for more information and options.