Streaming uploads (original) (raw)

Cloud Storage supports streaming data to a bucket without requiring that the data first be saved to a file. This is useful when you want to upload data but don't know the final size at the start of the upload, such as when generating the upload data from a process, or when compressing an object on-the-fly.

Using checksum validation when streaming

Because a checksum can only be supplied in the initial request of an upload, it's often not feasible to use Cloud Storage's checksum validationwhen streaming. It's recommended that you always use checksum validation, and you can manually do so after a streaming upload completes; however, validating after the transfer completes means that any corrupted data is accessible during the time it takes to confirm the corruption and remove it.

If you require checksum validation prior to the upload completing and the data becoming accessible, then you shouldn't use a streaming upload. You should use a different upload option that performs checksum validation prior to finalizing the object.

Required roles

To get the permissions that you need to stream uploads, ask your administrator to grant you one of the following roles:

• For uploads that include an Object Retention Lock, ask your administrator to grant you the Storage Object Admin (roles/storage.objectAdmin) IAM role for the bucket.
• For all other cases, ask your administrator to grant you the Storage Object User (roles/storage.objectUser) IAM role for the bucket.

These predefined roles contain the permissions required to stream uploads to Cloud Storage. To see the exact permissions that are required, expand the Required permissions section:

Required permissions

• storage.objects.create
• storage.objects.delete
  • This permission is only required for uploads that overwrite an existing object.
• storage.objects.list
  • This permission is only required for using the Google Cloud CLI to perform the instructions on this page.
• storage.objects.setRetention
  • This permission is only required for uploads that include an Object Retention Lock.

You can also get these permissions with other predefined roles orcustom roles.

For information about granting roles on buckets, seeSet and manage IAM policies on buckets.

Stream an upload

The following examples show how to perform a streaming upload from a process to a Cloud Storage object:

Console

The Google Cloud console does not support streaming uploads. Use the gcloud CLI instead.

Command line

1. Pipe the data to the gcloud storage cp command and use a dash for the source URL:
   PROCESS_NAME | gcloud storage cp - gs://BUCKET_NAME/OBJECT_NAME
   Where:
   • PROCESS_NAME is the name of the process from which you are collecting data. For example,collect_measurements.
   • BUCKET_NAME is the name of the bucket containing the object. For example, my_app_bucket.
   • OBJECT_NAME is the name of the object that is created from the data. For example, data_measurements.

Client libraries

C++

For more information, see theCloud Storage C++ API reference documentation.

To authenticate to Cloud Storage, set up Application Default Credentials. For more information, seeSet up authentication for client libraries.

C#

For more information, see theCloud Storage C# API reference documentation.

To authenticate to Cloud Storage, set up Application Default Credentials. For more information, seeSet up authentication for client libraries.

Go

For more information, see theCloud Storage Go API reference documentation.

To authenticate to Cloud Storage, set up Application Default Credentials. For more information, seeSet up authentication for client libraries.

Java

For more information, see theCloud Storage Java API reference documentation.

To authenticate to Cloud Storage, set up Application Default Credentials. For more information, seeSet up authentication for client libraries.

Node.js

For more information, see theCloud Storage Node.js API reference documentation.

To authenticate to Cloud Storage, set up Application Default Credentials. For more information, seeSet up authentication for client libraries.

PHP

For more information, see theCloud Storage PHP API reference documentation.

To authenticate to Cloud Storage, set up Application Default Credentials. For more information, seeSet up authentication for client libraries.

Python

For more information, see theCloud Storage Python API reference documentation.

To authenticate to Cloud Storage, set up Application Default Credentials. For more information, seeSet up authentication for client libraries.

Ruby

For more information, see theCloud Storage Ruby API reference documentation.

To authenticate to Cloud Storage, set up Application Default Credentials. For more information, seeSet up authentication for client libraries.

Rust

REST APIs

JSON API

To perform a streaming upload, use one of the following methods:

• A resumable upload, with the following adjustments:
  • When uploading the file data itself, use amultiple chunk upload.
  • Since you don't know the total file size until you get to the final chunk, use a * for the total file size in theContent-Range header of intermediate chunks.
    For example, if the first chunk you upload has a size of 512 KiB, the Content-Range header for the chunk is bytes 0-524287/*. If your upload has 64000 bytes remaining after the first chunk, you then send a final chunk that contains the remaining bytes and has a Content-Range header with the valuebytes 524288-588287/588288.
• A single-request upload, with the following adjustments:
  • Include a Transfer-Encoding: chunked header, and exclude the Content-Length header.
  • Construct the request according to the specification, sending the object data in chunks as it becomes available.

XML API

To perform a streaming upload, use one of the following methods:

• An XML API multipart upload
• A resumable upload, with the following adjustments:
  • When uploading the file data itself, use amultiple chunk upload.
  • Since you don't know the total file size until you get to the final chunk, use a * for the total file size in theContent-Range header of intermediate chunks.
    For example, if the first chunk you upload has a size of 512 KiB, the Content-Range header for the chunk is bytes 0-524287/*. If your upload has 64000 bytes remaining after the first chunk, you then send a final chunk that contains the remaining bytes and has a Content-Range header with the valuebytes 524288-588287/588288.
• A single-request upload, with the following adjustments:
  • Include a Transfer-Encoding: chunked header, and exclude the Content-Length header.
  • Construct the request according to the specification, sending the object data in chunks as it becomes available.
  • Note that you cannot perform a streaming upload using this method if the request uses a signature in its Authorizationheader.

What's next

• Stream a download.
• Learn more about decompressive transcoding.
• Learn more about object uploads.