Bucket relocation (original) (raw)

Bucket relocation

Stay organized with collections Save and categorize content based on your preferences.

This document describes the Cloud Storage bucket relocation service that relocates buckets serverlessly between geographic locations. Using bucket relocation, you can move an existing bucket from one location to another without changing its name or requiring manual data transfer of data within the bucket.

Based on your objectives and bucket usage, you'll need to plan your bucket relocation carefully to minimize disruption and relocate your bucket successfully. For more information about how to relocate buckets, see Relocate buckets.

Benefits

The benefits of bucket relocation are as follows:

Simplified migration: You can relocate buckets with minimal operational overhead. No complex scripting or multi-step processes are required.
Continuous operation: Your applications remain accessible throughout the relocation process, with no downtime for read operations and minimal downtime for write operations.
Improved performance: Colocating Compute Engine and Cloud Storage resources within the same region can reduce latency and enhance performance.
Metadata preservation: The bucket relocation process retains the object metadata. Retaining the object metadata ensures compatibility with existing applications and workflows after the bucket is moved.
Storage class configurations: You can maintain existing Cloud Storage class settings, including Autoclass. Preserving the storage class ensures that your cost structure remains consistent after the relocation.

Use cases

The following are use cases you can achieve by relocating your buckets:

Reduce data transfer cost: Avoid data transfer costs by relocating your bucket closer to the workloads accessing the bucket's data. For example, if your data is stored in the United States and primarily accessed from Europe, you can move your bucket to a European location to reduce data transfer costs.
Improve performance: Improve your application's speed and response by moving your data closer to your Compute Engine workloads. For example, if your application runs in us-central1 but your data resides inasia-east1, you can relocate your bucket to us-central1 to reduce latency.
Enhance resilience: Safeguard your critical data from regional outages. For example, if your data is stored in a single region, you can relocate it to a dual-region or multi-region for increased availability and disaster recovery.

Relocation types

There are two types of bucket relocations:

Bucket relocation with write downtime: In bucket relocation with write downtime, there is a period where you can't perform object write operations during the bucket relocation process.
Bucket relocation without write downtime: In bucket relocation without write downtime, you can continue to perform object write operations without interruption while the bucket relocation happens in the background.

The bucket's source and destination locations determine whether a bucket relocation involves write downtime. The following table shows how your bucket's location affects write downtime during a relocation, including the differences between relocations with and without downtime.

Specification	Bucket relocation with write downtime	Bucket relocation without write downtime
Bucket location	Relocating a bucket between the following locations causes downtime: RegionsDual-regionsMulti-regionsMulti-regions and predefined dual-regionsMulti-regions and configurable dual-regions if the two locations have different multi-region codes	Relocating a bucket between the following locations causes no downtime if the two locations share the same multi-region code: Configurable dual-regionsMulti-regions and configurable dual-regions
Write availability	You can't perform write operations during the final synchronization step.	Write operations continue uninterrupted during the relocation.
User involvement	You must initiate the write downtime finalization step.	No explicit finalization step is required.
Performance impact	You can't write or update objects in the bucket during the final synchronization step.	Object read and write latency might increase during the relocation.
Bucket relocation cancellation	Faster than relocations without write downtime.	Cancellation is not instantaneous and can take longer due to the need to backfill objects.
Feature support	Provides less feature support than relocations without write downtime. For more information about the unsupported features, see Unsupported features.	Limitations exist for features such as multi-part uploads, retention policies,Firebase, and appspot. For more information about these limitations, see Limitations.
Minimum relocation duration	None	Seven days

Understand the bucket relocation process

Bucket relocation helps you move your data from a source bucket to a destination bucket. The source bucket holds the data that you want to move and the destination bucket is where you want to move your data.

The following diagram shows the bucket relocation process flow:

Figure 1. Bucket relocation process flow (click to enlarge).

* Final synchronization is only required for relocations with write downtime.

The following table lists the three primary steps and the description for each step:

Step	Description
Perform a dry run(Optional)	Simulates the bucket relocation process to identify potential issues before the actual data transfer begins.
Initiate the incremental data copy step	Copies data from the source bucket to the destination bucket. The bucket metadata is write-locked to prevent any changes to the bucket that might affect the relocation process. However, you can write, modify, and delete objects in the bucket. The factors influencing the duration are as follows: The frequency of object updates, deletions, or additions within the bucket directly impacts the copy duration. A higher rate of change requires more time. There's a maximum object movement rate `Rm, objects/second`. With `N` total objects and an update rate of `R objects/second`, the copy step duration can be estimated as `N / (Rm - R)` seconds.Large buckets require more relocation time due to finite bandwidth. The size of individual objects affects the copy time. Objects larger than 10 GB take longer to transfer than objects under 10 GB due to bandwidth constraints. For example, a 1 TB object takes one day to copy.
Initiate the final synchronization step(Only required for relocations with write downtime)	Once you initiate the final synchronization, the bucket is write-locked. As a result, you can't write or update any objects within the bucket during this time, preventing data inconsistencies. However, you can continue to read from the bucket.Once all data has been transferred, verified, and the bucket is operational in the new location, the write-lock is automatically removed. You can then resume writing and updating objects in the bucket.

Limitations

The bucket relocation service supports up to five concurrent relocations from the same location within a project.

The following sections describe the limitations that apply to relocations with write downtime and without write downtime.

Relocation with write downtime limitations

Relocation with write downtime has the limitations listed in the following sections.

Data handling limitations

The following are the limitations when handling data during the relocation:

Table breakage: BigLake external tables and BigQuery tables using Apache Iceberg will break and require manual recreation. Automatic detection of impacted tables is unavailable.
Autoclass object handling: Autoclass uses access patterns to determine when to transition objects to colder storage classes. Duringfinal synchronization of the bucket relocation process, Autoclass is paused and objects aren't transitioned to colder storage classes. Once final synchronization is complete, Autoclass resumes.
- Objects in a Standard storage class are handled as follows:
  * Standard storage class objects have a 30-day no-access period before they can be transitioned to a colder class like Nearline storage. When an object in the Standard storage class is moved during the relocation, it's treated as if it has been accessed. As a result, the relocation process resets the no-access period and even if an object was close to being transitioned to Nearline storage before the move, it has to wait another 30 days after the relocation is complete.
- Objects in a non-Standard storage class are handled as follows:
  * Relocating objects in Nearline storage, Coldline storage, or Archive storage storage classes does not count as accessing them. As a result, the no-access period for these objects is not affected.
  * When relocating a bucket, if you frequently access objects in buckets with a non-Standard storage class, such as Nearline storage, Coldline storage, or Archive storage, the bucket doesn't automatically transition to a warmer storage class. For example, the bucket doesn't automatically transition from Archive storage to Coldline storage or from Coldline storage to a Standard storage, even if the objects are frequently accessed. This behavior prevents automatic storage class transitions during relocation.
  * If an object was scheduled to transition to a colder storage class such as from Nearline storage to Coldline storage, the relocation process won't interfere with the schedule. The transition proceeds as planned after the relocation is finished.
Object size limit: A 2 TB limit applies to object sizes for relocation.

Unsupported features

Buckets using the following features can't be relocated:

Customer-managed encryption keys (CMEK) or Customer-supplied encryption keys (CSEK).
Locked retention policies.
Objects with temporary holds.
You must disable Anywhere Cache before thefinal synchronization step. You can use Anywhere Cache during theincremental data copy step.
Buckets with hierarchical namespace enabled.
Multipart uploads. Multipart uploads are not supported for objects. You must re-upload the objects without using multipart methods.
Tags. Adding tags during relocation is not recommended as it causes the relocation process to fail.
Disable Anywhere Cache. While Anywhere Cache can be enabled during the incremental data copy step, it prevents the final synchronization step from completing.
Appspot buckets. Consider migrating Container Registry to Artifact Registry as a workaround for default buckets created by App Engine.
Firebase buckets. You can't relocate buckets associated with Firebase.

Operational restrictions

Bucket relocation with write downtime has the following operational restrictions:

Project restriction: You can't relocate buckets across projects.
Resumable uploads: In-progress resumable uploads must be finalized before the final synchronization step to avoid data loss.
Metadata updates: You can't update a bucket's metadata during relocation.
Request rate ramp-up: Relocated buckets are subject to the same request rate ramp-up guidelines as newly created buckets.

Relocation without write downtime limitations

Bucket relocation without write downtime has the following limitations:

Multipart uploads: Unfinished multipart uploads are not supported and must be completed or aborted before relocation. New multipart uploads are blocked during the move.
Retention policies: All retention policies must be unlocked before relocation.
Firebase and Appspot buckets: Relocation is not supported for buckets associated with Firebase or Appspot.
Progress updates: Relocation progress updates might not be linear.

Unsupported locations

Bucket relocation isn't supported for the source and destination buckets in the following locations:

Location type	Unsupported locations
Regions	ME-CENTRAL1ME-WEST1
Predefined dual-regions	EUR5EUR7EUR8

Pricing

For details on pricing associated with bucket relocation, see Cloud Storage pricing.

What's next

Learn how to plan a bucket relocation.
Learn how to relocate buckets.