Implement obsolete file deletion (GC) in follower by anand1976 · Pull Request #12657 · facebook/rocksdb (original) (raw)

@anand1976

Summary: This PR implements deletion of obsolete files in a follower RocksDB instance. The follower tails the leader's MANIFEST and creates links to newly added SST files. These links need to be deleted once those files become obsolete in order to reclaim space. There are three cases to be considered -

  1. New files added and links created, but the Version could not be installed due to some missing files. Those links need to be preserved so a subsequent catch up attempt can succeed. We insert the next file number in the VersionSet to pending_outputs_ to prevent their deletion.
  2. Files deleted from the previous successfully installed Version. These are deleted as usual in PurgeObsoleteFiles.
  3. New files added by a VersionEdit and deleted by a subsequent VersionEdit, both processed in the same catchup attempt. Links will be created for the new files when verifying a candidate Version. Those need to be deleted explicitly as they're never added to VersionStorageInfo, and thus not deleted by PurgeObsoleteFiles.

Test plan - New unit tests in db_follower_test.

Pull Request resolved: facebook#12657

Reviewed By: jowlyzhang

Differential Revision: D57462697

Pulled By: anand1976

fbshipit-source-id: 898f15570638dd4930f839ffd31c560f9cb73916