Published on

Comprehensive Guide to Google Cloud Storage Using CLI/SDK and Advanced Bucket Lock Features

Authors

Introduction

Google Cloud Storage offers a scalable, secure, and durable storage solution for developers and enterprises. In this comprehensive guide, we'll walk you through the process of using the Google Cloud command-line interface (CLI) to interact with Cloud Storage. You'll learn how to create buckets, upload and download objects, manage folders, control access permissions, and leverage advanced features like Bucket Lock—all from the command line.

Table of Contents

Prerequisites

  • A Google Cloud account with access to the Google Cloud Console.
  • Basic knowledge of command-line operations.
  • Familiarity with Google Cloud Storage concepts.

Note: For this guide, we recommend using the Google Cloud Shell for all command-line operations.

Setting Up Your Environment

Activate Cloud Shell

  1. Log in to the Google Cloud Console.

  2. Click on the Activate Cloud Shell button in the top-right corner of the console:

    Activate Cloud Shell

  3. A Cloud Shell session will open at the bottom of your browser window.

Cloud Shell is a virtual machine that comes pre-loaded with development tools. It offers a persistent 5GB home directory and runs on Google Cloud.

Set the Compute Region

Set the default compute region to us-east1 (or any region of your choice):

gcloud config set compute/region us-east1

Output:

Updated property [compute/region].

Basic Cloud Storage Operations

Task 1: Create a Storage Bucket

Use the gsutil command-line tool to create a new storage bucket.

Bucket Naming Rules

  • Must be globally unique across all Google Cloud projects.
  • Can contain only lowercase letters, numbers, dashes (-), underscores (_), and dots (.).
  • Must start and end with a letter or number.
  • Length must be between 3 and 63 characters.

Create the Bucket

Replace <YOUR_BUCKET_NAME> with a unique bucket name following the naming rules:

gsutil mb gs://<YOUR_BUCKET_NAME>/

Example:

gsutil mb gs://my-unique-bucket-12345/

Output:

Creating gs://my-unique-bucket-12345/...

Task 2: Upload an Object into Your Bucket

Download an image file and upload it to your newly created bucket.

Download the Image

Use curl to download an image of Ada Lovelace:

curl https://upload.wikimedia.org/wikipedia/commons/thumb/a/a4/Ada_Lovelace_portrait.jpg/800px-Ada_Lovelace_portrait.jpg --output ada.jpg

Output:

  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100  360k  100  360k    0     0   850k      0 --:--:-- --:--:-- --:--:--  850k

Upload the Image to the Bucket

Use gsutil cp to copy the image to your bucket:

gsutil cp ada.jpg gs://<YOUR_BUCKET_NAME>/

Example:

gsutil cp ada.jpg gs://my-unique-bucket-12345/

Output:

Copying file://ada.jpg [Content-Type=image/jpeg]...
/ [1 files][360.1 KiB/360.1 KiB]
Operation completed over 1 objects/360.1 KiB.

Remove the Local Image File

rm ada.jpg

Task 3: Download an Object from Your Bucket

Download the image from your bucket back to the Cloud Shell environment.

gsutil cp gs://<YOUR_BUCKET_NAME>/ada.jpg .

Example:

gsutil cp gs://my-unique-bucket-12345/ada.jpg .

Output:

Copying gs://my-unique-bucket-12345/ada.jpg...
/ [1 files][360.1 KiB/360.1 KiB]
Operation completed over 1 objects/360.1 KiB.

Task 4: Copy an Object to a Folder in the Bucket

Create a folder within your bucket and copy the image into it.

Copy the Image to a Folder

gsutil cp gs://<YOUR_BUCKET_NAME>/ada.jpg gs://<YOUR_BUCKET_NAME>/image-folder/

Example:

gsutil cp gs://my-unique-bucket-12345/ada.jpg gs://my-unique-bucket-12345/image-folder/

Output:

Copying gs://my-unique-bucket-12345/ada.jpg [Content-Type=image/jpeg]...
/ [1 files][360.1 KiB/360.1 KiB]
Operation completed over 1 objects/360.1 KiB.

Task 5: List Contents of a Bucket or Folder

List all objects in your bucket.

List Bucket Contents

gsutil ls gs://<YOUR_BUCKET_NAME>/

Example:

gsutil ls gs://my-unique-bucket-12345/

Output:

gs://my-unique-bucket-12345/ada.jpg
gs://my-unique-bucket-12345/image-folder/

Task 6: List Details for an Object

Get detailed information about the image file.

List Object Details

gsutil ls -l gs://<YOUR_BUCKET_NAME>/ada.jpg

Example:

gsutil ls -l gs://my-unique-bucket-12345/ada.jpg

Output:

  368543  2023-10-01T12:00:00Z  gs://my-unique-bucket-12345/ada.jpg
TOTAL: 1 objects, 368543 bytes (360.1 KiB)

Task 7: Make Your Object Publicly Accessible

Change the Access Control List (ACL) of the object to make it publicly readable.

Grant Public Read Access

gsutil acl ch -u AllUsers:R gs://<YOUR_BUCKET_NAME>/ada.jpg

Example:

gsutil acl ch -u AllUsers:R gs://my-unique-bucket-12345/ada.jpg

Output:

Updated ACL on gs://my-unique-bucket-12345/ada.jpg

Verify Public Access

  1. In the Google Cloud Console, navigate to Cloud Storage > Buckets.
  2. Click on your bucket name.
  3. Locate ada.jpg and copy its Public URL.
  4. Paste the URL into a new browser tab to view the image.

Task 8: Remove Public Access

Revoke public access to the object by removing the AllUsers permission.

Remove Public Read Access

gsutil acl ch -d AllUsers gs://<YOUR_BUCKET_NAME>/ada.jpg

Example:

gsutil acl ch -d AllUsers gs://my-unique-bucket-12345/ada.jpg

Output:

Updated ACL on gs://my-unique-bucket-12345/ada.jpg

Verify Access Removal

  • Refresh the Cloud Storage page in the console.
  • The Public column for ada.jpg should no longer indicate public access.
  • Attempting to access the image via the public URL should result in an access denied error.

Task 9: Delete Objects

Remove objects from your bucket.

Delete the Original Image

gsutil rm gs://<YOUR_BUCKET_NAME>/ada.jpg

Example:

gsutil rm gs://my-unique-bucket-12345/ada.jpg

Output:

Removing gs://my-unique-bucket-12345/ada.jpg...

Delete the Copied Image in the Folder

gsutil rm gs://<YOUR_BUCKET_NAME>/image-folder/ada.jpg

Example:

gsutil rm gs://my-unique-bucket-12345/image-folder/ada.jpg

Output:

Removing gs://my-unique-bucket-12345/image-folder/ada.jpg...

Verify Deletion

List the contents of your bucket to confirm deletion:

gsutil ls gs://<YOUR_BUCKET_NAME>/

Example:

gsutil ls gs://my-unique-bucket-12345/

Output:

gs://my-unique-bucket-12345/image-folder/

Advanced Cloud Storage Operations

Task 10: Using Bucket Lock and Retention Policies

Bucket Lock allows you to configure a data retention policy for a Cloud Storage bucket, governing how long objects in the bucket must be retained. Once locked, the retention policy cannot be removed or reduced.

Overview

In scenarios requiring strict compliance and data retention (e.g., financial regulations like FINRA, SEC, CFTC, or healthcare regulations), Bucket Lock ensures that data cannot be altered or deleted prematurely. Combined with detailed audit logging and Object Lifecycle Management, it provides a robust solution for regulatory compliance.

What You'll Learn

In this section, you will learn how to:

  • Create a bucket
  • Define an object retention policy
  • Lock the retention policy
  • Set a temporary hold
  • Use event-based holds
  • Remove a retention policy

Create a New Bucket

First, define an environment variable for your new bucket and use your project ID as the bucket name:

export BUCKET=$(gcloud config get-value project)

Output:

my-project-id

Create the Bucket:

gsutil mb gs://$BUCKET/

Example:

gsutil mb gs://qwiklabs-gcp-04-b7c24845b55b/

Output:

Creating gs://qwiklabs-gcp-04-b7c24845b55b/...

Define a Retention Policy

Set a retention policy of 10 seconds (for demonstration purposes):

gsutil retention set 10s gs://$BUCKET/

Example:

gsutil retention set 10s gs://qwiklabs-gcp-04-b7c24845b55b/

Output:

Setting retention policy on gs://qwiklabs-gcp-04-b7c24845b55b/...

Verify the Retention Policy:

gsutil retention get gs://$BUCKET/

Example:

gsutil retention get gs://qwiklabs-gcp-04-b7c24845b55b/

Sample Output:

Retention Policy (UNLOCKED):
  Duration: 10 Second(s)
  Effective Time: Tue, 23 Jan 2018 01:04:05 GMT

Upload a Transaction Record Object:

Add a transaction record object to test the retention policy:

gsutil cp gs://spls/gsp297/dummy_transactions gs://$BUCKET/

Example:

gsutil cp gs://spls/gsp297/dummy_transactions gs://qwiklabs-gcp-04-b7c24845b55b/

Output:

Copying gs://spls/gsp297/dummy_transactions...
/ [1 files][xx.xx KiB/xx.xx KiB]
Operation completed over 1 objects/xx.xx KiB.

Review the Retention Expiration:

gsutil ls -L gs://$BUCKET/dummy_transactions

Example:

gsutil ls -L gs://qwiklabs-gcp-04-b7c24845b55b/dummy_transactions

Sample Output:

gs://qwiklabs-gcp-04-b7c24845b55b/dummy_transactions:
    Creation time:          Tue, 23 Jan 2018 00:45:21 GMT
    Update time:            Thu, 25 Jan 2018 20:14:49 GMT
    Retention Expiration:   Thu, 25 Jan 2018 20:14:59 GMT

Extend a Retention Policy:

To extend the retention policy, use the gsutil retention set command again with the desired duration.

gsutil retention set 20s gs://$BUCKET/

Example:

gsutil retention set 20s gs://qwiklabs-gcp-04-b7c24845b55b/

Lock the Retention Policy

Locking the retention policy makes it immutable. Once locked, it cannot be removed or reduced.

gsutil retention lock gs://$BUCKET/

Example:

gsutil retention lock gs://qwiklabs-gcp-04-b7c24845b55b/

Sample Output to Confirm the Lock:

This will PERMANENTLY set the Retention Policy on gs://qwiklabs-gcp-04-b7c24845b55b/ to:

  Retention Policy (UNLOCKED):
    Duration: 10 Second(s)
    Effective Time: Wed, 07 Feb 2018 01:37:52 GMT

This setting cannot be reverted!  Continue? [y|N]:

Type y and press Enter:

Retention policy for gs://qwiklabs-gcp-04-b7c24845b55b/ is now locked.

Verify the Locked Retention Policy:

gsutil retention get gs://$BUCKET/

Example:

gsutil retention get gs://qwiklabs-gcp-04-b7c24845b55b/

Sample Output:

Retention Policy (LOCKED):
  Duration: 10 Second(s)
  Effective Time: Wed, 07 Feb 2018 01:37:52 GMT

Set a Temporary Hold

Temporary holds prevent objects from being deleted, even if they are older than the retention period.

Set a Temporary Hold on the Transactions Object:

gsutil retention temp set gs://$BUCKET/dummy_transactions

Example:

gsutil retention temp set gs://qwiklabs-gcp-04-b7c24845b55b/dummy_transactions

Output:

Setting Temporary Hold on gs://qwiklabs-gcp-04-b7c24845b55b/dummy_transactions...

Attempt to Delete the Object:

gsutil rm gs://$BUCKET/dummy_transactions

Example:

gsutil rm gs://qwiklabs-gcp-04-b7c24845b55b/dummy_transactions

Expected Error Message:

AccessDeniedException: 403 Object 'qwiklabs-gcp-04-b7c24845b55b/dummy_transactions is under active Temporary hold and cannot be deleted, overwritten or archived until hold is removed.

Release the Temporary Hold:

gsutil retention temp release gs://$BUCKET/dummy_transactions

Example:

gsutil retention temp release gs://qwiklabs-gcp-04-b7c24845b55b/dummy_transactions

Output:

Releasing Temporary Hold on gs://qwiklabs-gcp-04-b7c24845b55b/dummy_transactions...

Delete the Object After Releasing the Hold:

gsutil rm gs://$BUCKET/dummy_transactions

Example:

gsutil rm gs://qwiklabs-gcp-04-b7c24845b55b/dummy_transactions

Output:

Removing gs://qwiklabs-gcp-04-b7c24845b55b/dummy_transactions...

Use Event-Based Holds

Event-based holds delay the start of the retention period until a specific event occurs.

Enable Default Event-Based Hold on the Bucket:

gsutil retention event-default set gs://$BUCKET/

Example:

gsutil retention event-default set gs://qwiklabs-gcp-04-b7c24845b55b/

Output:

Setting default event-based hold on gs://qwiklabs-gcp-04-b7c24845b55b/...

Upload an Object with Event-Based Hold:

gsutil cp gs://spls/gsp297/dummy_loan gs://$BUCKET/

Example:

gsutil cp gs://spls/gsp297/dummy_loan gs://qwiklabs-gcp-04-b7c24845b55b/

Output:

Copying gs://spls/gsp297/dummy_loan...
/ [1 files][xx.xx KiB/xx.xx KiB]
Operation completed over 1 objects/xx.xx KiB.

Verify Event-Based Hold:

gsutil ls -L gs://$BUCKET/dummy_loan

Example:

gsutil ls -L gs://qwiklabs-gcp-04-b7c24845b55b/dummy_loan

Sample Output:

gs://qwiklabs-gcp-04-b7c24845b55b/dummy_loan:
    Creation time:          Fri, 26 Jan 2018 07:40:28 GMT
    Update time:            Fri, 26 Jan 2018 07:40:28 GMT
    Event-Based Hold:       Enabled

Release Event-Based Hold When Event Occurs:

gsutil retention event release gs://$BUCKET/dummy_loan

Example:

gsutil retention event release gs://qwiklabs-gcp-04-b7c24845b55b/dummy_loan

Output:

Releasing Event-Based Hold on gs://qwiklabs-gcp-04-b7c24845b55b/dummy_loan...

Verify Retention Expiration:

gsutil ls -L gs://$BUCKET/dummy_loan

Example:

gsutil ls -L gs://qwiklabs-gcp-04-b7c24845b55b/dummy_loan

Sample Output:

gs://qwiklabs-gcp-04-b7c24845b55b/dummy_loan:
    Creation time:          Fri, 26 Jan 2018 08:14:16 GMT
    Update time:            Fri, 26 Jan 2018 08:14:25 GMT
    Retention Expiration:   Fri, 26 Jan 2018 08:14:45 GMT

Attempt to Delete Before Retention Period Expires:

gsutil rm gs://$BUCKET/loan_record.txt

Example:

gsutil rm gs://qwiklabs-gcp-04-b7c24845b55b/loan_record.txt

Expected Error Message:

AccessDeniedException: 403 gs://qwiklabs-gcp-04-b7c24845b55b/loan_record.txt is subject to bucket's retention policy and cannot be deleted, overwritten or archived until Fri, 26 Jan 2018 08:14:45 GMT

Remove a Retention Policy

Once all objects are deleted and the retention periods have expired, you can delete the bucket even if it has a locked retention policy.

Delete the Bucket:

gsutil rb gs://$BUCKET/

Example:

gsutil rb gs://qwiklabs-gcp-04-b7c24845b55b/

Output:

Removing gs://qwiklabs-gcp-04-b7c24845b55b/...

Note: The bucket must be empty to be deleted.

Conclusion

Congratulations! You've successfully navigated through both basic and advanced features of Google Cloud Storage using the command-line interface. You've learned how to:

  • Create and manage storage buckets: Establishing storage containers with unique names adhering to naming conventions.
  • Upload, download, and organize objects: Efficiently handling your data within Cloud Storage.
  • Modify access permissions for objects: Controlling public and private access through ACLs.
  • Implement and manage retention policies with Bucket Lock: Ensuring data integrity and compliance with regulatory requirements.
  • Use temporary holds and event-based holds: Providing flexibility in data retention and audit scenarios.
  • Delete objects and buckets securely: Maintaining a clean and organized storage environment.

These skills are essential for efficiently managing your data and meeting regulatory compliance in Google Cloud Storage. You can now integrate these commands into scripts or workflows to automate your cloud storage tasks and ensure robust data governance.

Additional Resources

Disclaimer: The bucket and object names used in this guide are for demonstration purposes. Always ensure that your bucket names are unique and comply with Google's naming conventions.