Product overview of Cloud Storage (original) (raw)

This page provides an overview of Cloud Storage and how it works.

Cloud Storage is a service for storing your objects in Google Cloud. An object is an immutable piece of data consisting of a file of any format. You store objects in containers called buckets, and you can optionally organize objects stored in buckets by using folders.

All buckets are associated with a project, and you can group your projects under an organization. Each project, bucket, managed folder, and object in Google Cloud is a resource in Google Cloud, as are things such as Compute Engine instances.

After you create a project, you can create Cloud Storage buckets,upload objects to your buckets, and download objects from your buckets. You can also grant permissions to make your data accessible to principals you specify or accessible to everyone on the public internet.

The Google Cloud hierarchy

Here's how the Cloud Storage structure can apply to a real-world case:

• Organization: Your company, called Example Inc., creates a Google Cloud organization called exampleinc.org.
• Project: Example Inc. is building several applications, and each one is associated with a project. Each project has its own set of Cloud Storage APIs, as well as other resources.
• Bucket: Each project can contain multiple buckets, which are containers to store your objects. For example, you might create a photos bucket for all the image files your app generates and a separate videos bucket.
  • Hierarchical namespace enabled buckets: Buckets can havehierarchical namespace enabled, which lets you store your data in a logical file system structure using folders. You can manage folders by using folder-specific operations, including creating, deleting, listing, and renaming. Buckets with hierarchical namespace enabled offer up to 8 times higher initial QPS limits for reading and writing objects compared to buckets without hierarchical namespace enabled, and is beneficial for data-intensive tasks like analytics and AI/ML workloads.
• Object: Buckets contain objects, such as an image called puppy.png. Each bucket can contain essentially unlimited individual objects.
• Folder: Buckets with hierarchical namespace enabled can contain folders. Folders enable a real file system for storing objects, as opposed to a simulated file system. You can atomically rename a folder and all the objects within it in one operation.
• Managed folder: Each bucket can also contain managed folders, which grant or revoke additional access beyond the IAM permissions set on the bucket. Managed folders don't use a true directory tree structure; rather, a managed folder is a resource overlay used only for permission checking.

You can interact with Cloud Storage by using the following tools:

• Console: The Google Cloud console provides a visual interface for you to manage your data in a browser.
• Google Cloud CLI: The gcloud CLI allows you to interact with Cloud Storage through a terminal usinggcloud storage commands.
• Client libraries: The Cloud Storage client libraries allow you to manage your data using one of your preferred languages, including C++, C#, Go, Java, Node.js, PHP, Python, and Ruby.
• REST APIs: Manage your data using the JSON or XML API.
• Terraform: Terraform is an infrastructure-as-code (IaC) tool that you can use to provision the infrastructure for Cloud Storage. For more information, seeProvision resources with Cloud Storage.
• gRPC: gRPC lets you interact with Cloud Storage. gRPC is a high performance, open source universal RPC framework developed by Google that you can use to define your services using Protocol Buffers.
• Cloud Storage FUSE: Cloud Storage FUSE lets you mount Cloud Storage buckets to your local file system. This enables your applications to read from a bucket or write to a bucket by using standard file system semantics.

Securing your data

Once you upload your objects to Cloud Storage, you have fine-grained control over how you secure and share your data. Here are some ways to secure the data you upload to Cloud Storage:

• Identity and Access Management: Use IAM to control who has access to the resources in your Google Cloud project. Resources include Cloud Storage buckets and objects, as well as other Google Cloud entities such asCompute Engine instances. You can grant principals certain types of access to buckets and objects, such as update, create, or delete.
• Data encryption: Cloud Storage uses server-side encryption toencrypt your data by default. You can also use supplemental data encryption options such as customer-managed encryption keys andcustomer-supplied encryption keys.
• Authentication: Ensure that anyone who accesses your data has proper credentials.
• Soft delete: Prevent permanent loss of data against accidental or malicious deletion by retaining recently deleted objects and buckets. By default, Cloud Storage enables soft deletefor all buckets with a seven day retention period.
• Object Versioning: When a live version of an object is replaced or deleted, it can be retained as a noncurrent version if youenable Object Versioning.
• Bucket IP filtering: With bucket IP filtering, you can restrict access to a bucket based on the source IP address of the request and secure your data from unauthorized access from specific IP addresses or Virtual Private Cloud (VPC).
• Bucket Lock: Govern how long objects in buckets must be retained byspecifying a retention policy.

Resource names

Each resource has a unique name that identifies it, much like a filename. Buckets have a resource name in the form ofprojects/_/buckets/BUCKET_NAME, whereBUCKET_NAME is the ID of the bucket. Objects have a resource name in the form ofprojects/_/buckets/BUCKET_NAME/objects/OBJECT_NAME, where OBJECT_NAME is the ID of the object.

A #NUMBER appended to the end of the resource name indicates a specific generation of the object. #0 is a special identifier for the most recent version of an object. #0 is useful to add when the name of the object ends in a string that would otherwise be interpreted as a generation number.

Quickstart guides

To learn the fundamentals of using Cloud Storage, visit the following guides:

• Google Cloud console quickstart
• gcloud quickstart
• Terraform quickstart

Looking for other products?

If Cloud Storage is not the right storage solution for you, see more information about the following storage services:

• Google Cloud Managed Lustre: Store your data in a high-performance, fully managed parallel file system that's optimized for AI and HPC workloads.
• Google Drive: Store, manage, and share your personal files.
• Cloud Storage for Firebase: Manage data for your mobile applications.
• Persistent Disk: Add block storage to your Compute Engine virtual machine.
• Filestore: Add file storage for multiwriter access to your GKE clusters.
• Explore more storage services offered by Google.

What's next

• Learn the fundamentals of Cloud Storage through theGoogle Cloud console or Google Cloud CLI.
• Try Google Cloud jump start solutions that use Cloud Storage.
• Get started with client libraries.
• Quickly import online data into Cloud Storage or between Cloud Storage buckets using Storage Transfer Service.