Use SpecialEnv to speed up some slow BackupEngineRateLimitingTestWithParam (#9974)

Summary:
**Context:**
`BackupEngineRateLimitingTestWithParam.RateLimiting` and `BackupEngineRateLimitingTestWithParam.RateLimitingVerifyBackup` involve creating backup and restoring of a big database with rate-limiting. Using the normal env with a normal clock requires real elapse of time (13702 - 19848 ms/per test). As suggested in https://github.com/facebook/rocksdb/pull/8722#discussion_r703698603, this PR is to speed it up with SpecialEnv (`time_elapse_only_sleep=true`) where its clock accepts fake elapse of time during rate-limiting (100 - 600 ms/per test)

**Summary:**
- Added TEST_ function to set clock of the default rate limiters in backup engine
- Shrunk testdb by 10 times while keeping it big enough for testing
- Renamed some test variables and reorganized some if-else branch for clarity without changing the test

Pull Request resolved: https://github.com/facebook/rocksdb/pull/9974

Test Plan:
- Run tests pre/post PR the same time to verify the tests are sped up by 90 - 95%
`BackupEngineRateLimitingTestWithParam.RateLimiting`
Pre:
```
[ RUN      ] RateLimiting/BackupEngineRateLimitingTestWithParam.RateLimiting/0
[       OK ] RateLimiting/BackupEngineRateLimitingTestWithParam.RateLimiting/0 (11123 ms)
[ RUN      ] RateLimiting/BackupEngineRateLimitingTestWithParam.RateLimiting/1
[       OK ] RateLimiting/BackupEngineRateLimitingTestWithParam.RateLimiting/1 (9441 ms)
[ RUN      ] RateLimiting/BackupEngineRateLimitingTestWithParam.RateLimiting/2
[       OK ] RateLimiting/BackupEngineRateLimitingTestWithParam.RateLimiting/2 (11096 ms)
[ RUN      ] RateLimiting/BackupEngineRateLimitingTestWithParam.RateLimiting/3
[       OK ] RateLimiting/BackupEngineRateLimitingTestWithParam.RateLimiting/3 (9339 ms)
[ RUN      ] RateLimiting/BackupEngineRateLimitingTestWithParam.RateLimiting/4
[       OK ] RateLimiting/BackupEngineRateLimitingTestWithParam.RateLimiting/4 (11121 ms)
[ RUN      ] RateLimiting/BackupEngineRateLimitingTestWithParam.RateLimiting/5
[       OK ] RateLimiting/BackupEngineRateLimitingTestWithParam.RateLimiting/5 (9413 ms)
[ RUN      ] RateLimiting/BackupEngineRateLimitingTestWithParam.RateLimiting/6
[       OK ] RateLimiting/BackupEngineRateLimitingTestWithParam.RateLimiting/6 (11185 ms)
[ RUN      ] RateLimiting/BackupEngineRateLimitingTestWithParam.RateLimiting/7
[       OK ] RateLimiting/BackupEngineRateLimitingTestWithParam.RateLimiting/7 (9511 ms)
[----------] 8 tests from RateLimiting/BackupEngineRateLimitingTestWithParam (82230 ms total)
```
Post:
```
[ RUN      ] RateLimiting/BackupEngineRateLimitingTestWithParam.RateLimiting/0
[       OK ] RateLimiting/BackupEngineRateLimitingTestWithParam.RateLimiting/0 (395 ms)
[ RUN      ] RateLimiting/BackupEngineRateLimitingTestWithParam.RateLimiting/1
[       OK ] RateLimiting/BackupEngineRateLimitingTestWithParam.RateLimiting/1 (564 ms)
[ RUN      ] RateLimiting/BackupEngineRateLimitingTestWithParam.RateLimiting/2
[       OK ] RateLimiting/BackupEngineRateLimitingTestWithParam.RateLimiting/2 (358 ms)
[ RUN      ] RateLimiting/BackupEngineRateLimitingTestWithParam.RateLimiting/3
[       OK ] RateLimiting/BackupEngineRateLimitingTestWithParam.RateLimiting/3 (567 ms)
[ RUN      ] RateLimiting/BackupEngineRateLimitingTestWithParam.RateLimiting/4
[       OK ] RateLimiting/BackupEngineRateLimitingTestWithParam.RateLimiting/4 (173 ms)
[ RUN      ] RateLimiting/BackupEngineRateLimitingTestWithParam.RateLimiting/5
[       OK ] RateLimiting/BackupEngineRateLimitingTestWithParam.RateLimiting/5 (176 ms)
[ RUN      ] RateLimiting/BackupEngineRateLimitingTestWithParam.RateLimiting/6
[       OK ] RateLimiting/BackupEngineRateLimitingTestWithParam.RateLimiting/6 (191 ms)
[ RUN      ] RateLimiting/BackupEngineRateLimitingTestWithParam.RateLimiting/7
[       OK ] RateLimiting/BackupEngineRateLimitingTestWithParam.RateLimiting/7 (177 ms)
[----------] 8 tests from RateLimiting/BackupEngineRateLimitingTestWithParam (2601 ms total)
```
`BackupEngineRateLimitingTestWithParam.RateLimitingVerifyBackup`
Pre:
```
[ RUN      ] RateLimiting/BackupEngineRateLimitingTestWithParam.RateLimitingVerifyBackup/0
[       OK ] RateLimiting/BackupEngineRateLimitingTestWithParam.RateLimitingVerifyBackup/0 (7275 ms)
[ RUN      ] RateLimiting/BackupEngineRateLimitingTestWithParam.RateLimitingVerifyBackup/1
[       OK ] RateLimiting/BackupEngineRateLimitingTestWithParam.RateLimitingVerifyBackup/1 (3961 ms)
[ RUN      ] RateLimiting/BackupEngineRateLimitingTestWithParam.RateLimitingVerifyBackup/2
[       OK ] RateLimiting/BackupEngineRateLimitingTestWithParam.RateLimitingVerifyBackup/2 (7117 ms)
[ RUN      ] RateLimiting/BackupEngineRateLimitingTestWithParam.RateLimitingVerifyBackup/3
[       OK ] RateLimiting/BackupEngineRateLimitingTestWithParam.RateLimitingVerifyBackup/3 (3921 ms)
[ RUN      ] RateLimiting/BackupEngineRateLimitingTestWithParam.RateLimitingVerifyBackup/4
[       OK ] RateLimiting/BackupEngineRateLimitingTestWithParam.RateLimitingVerifyBackup/4 (19862 ms)
[ RUN      ] RateLimiting/BackupEngineRateLimitingTestWithParam.RateLimitingVerifyBackup/5
[       OK ] RateLimiting/BackupEngineRateLimitingTestWithParam.RateLimitingVerifyBackup/5 (10231 ms)
[ RUN      ] RateLimiting/BackupEngineRateLimitingTestWithParam.RateLimitingVerifyBackup/6
[       OK ] RateLimiting/BackupEngineRateLimitingTestWithParam.RateLimitingVerifyBackup/6 (19848 ms)
[ RUN      ] RateLimiting/BackupEngineRateLimitingTestWithParam.RateLimitingVerifyBackup/7
[       OK ] RateLimiting/BackupEngineRateLimitingTestWithParam.RateLimitingVerifyBackup/7 (10372 ms)
[----------] 8 tests from RateLimiting/BackupEngineRateLimitingTestWithParam (82587 ms total)
```
Post:
```
[ RUN      ] RateLimiting/BackupEngineRateLimitingTestWithParam.RateLimitingVerifyBackup/0
[       OK ] RateLimiting/BackupEngineRateLimitingTestWithParam.RateLimitingVerifyBackup/0 (157 ms)
[ RUN      ] RateLimiting/BackupEngineRateLimitingTestWithParam.RateLimitingVerifyBackup/1
[       OK ] RateLimiting/BackupEngineRateLimitingTestWithParam.RateLimitingVerifyBackup/1 (152 ms)
[ RUN      ] RateLimiting/BackupEngineRateLimitingTestWithParam.RateLimitingVerifyBackup/2
[       OK ] RateLimiting/BackupEngineRateLimitingTestWithParam.RateLimitingVerifyBackup/2 (160 ms)
[ RUN      ] RateLimiting/BackupEngineRateLimitingTestWithParam.RateLimitingVerifyBackup/3
[       OK ] RateLimiting/BackupEngineRateLimitingTestWithParam.RateLimitingVerifyBackup/3 (158 ms)
[ RUN      ] RateLimiting/BackupEngineRateLimitingTestWithParam.RateLimitingVerifyBackup/4
[       OK ] RateLimiting/BackupEngineRateLimitingTestWithParam.RateLimitingVerifyBackup/4 (155 ms)
[ RUN      ] RateLimiting/BackupEngineRateLimitingTestWithParam.RateLimitingVerifyBackup/5
[       OK ] RateLimiting/BackupEngineRateLimitingTestWithParam.RateLimitingVerifyBackup/5 (151 ms)
[ RUN      ] RateLimiting/BackupEngineRateLimitingTestWithParam.RateLimitingVerifyBackup/6
[       OK ] RateLimiting/BackupEngineRateLimitingTestWithParam.RateLimitingVerifyBackup/6 (146 ms)
[ RUN      ] RateLimiting/BackupEngineRateLimitingTestWithParam.RateLimitingVerifyBackup/7
[       OK ] RateLimiting/BackupEngineRateLimitingTestWithParam.RateLimitingVerifyBackup/7 (153 ms)
[----------] 8 tests from RateLimiting/BackupEngineRateLimitingTestWithParam (1232 ms total)
```

Reviewed By: pdillinger

Differential Revision: D36336345

Pulled By: hx235

fbshipit-source-id: 724c6ba745f95f56d4440a6d2f1e4512a2987589

utilities/backup/backup_engine.cc

@@ -48,6 +48,7 @@
 #include "util/coding.h"
 #include "util/crc32c.h"
 #include "util/math.h"
+#include "util/rate_limiter.h"
 #include "util/string_util.h"
 #include "utilities/backup/backup_engine_impl.h"
 #include "utilities/checkpoint/checkpoint_impl.h"
@@ -181,6 +182,61 @@ class BackupEngineImpl {
            BackupEngineOptions::kMaskNamingFlags;
   }
 
+  void TEST_SetDefaultRateLimitersClock(
+      const std::shared_ptr<SystemClock>& backup_rate_limiter_clock,
+      const std::shared_ptr<SystemClock>& restore_rate_limiter_clock) {
+    if (backup_rate_limiter_clock) {
+      assert(options_.backup_rate_limiter->IsInstanceOf(
+          GenericRateLimiter::kClassName()));
+      auto* backup_rate_limiter_options =
+          options_.backup_rate_limiter
+              ->GetOptions<GenericRateLimiter::GenericRateLimiterOptions>();
+
+      assert(backup_rate_limiter_options);
+      RateLimiter::Mode backup_rate_limiter_mode;
+      if (!options_.backup_rate_limiter->IsRateLimited(
+              RateLimiter::OpType::kRead)) {
+        backup_rate_limiter_mode = RateLimiter::Mode::kWritesOnly;
+      } else if (!options_.backup_rate_limiter->IsRateLimited(
+                     RateLimiter::OpType::kWrite)) {
+        backup_rate_limiter_mode = RateLimiter::Mode::kReadsOnly;
+      } else {
+        backup_rate_limiter_mode = RateLimiter::Mode::kAllIo;
+      }
+      options_.backup_rate_limiter.reset(new GenericRateLimiter(
+          backup_rate_limiter_options->max_bytes_per_sec,
+          backup_rate_limiter_options->refill_period_us,
+          backup_rate_limiter_options->fairness, backup_rate_limiter_mode,
+          backup_rate_limiter_clock, backup_rate_limiter_options->auto_tuned));
+    }
+
+    if (restore_rate_limiter_clock) {
+      assert(options_.restore_rate_limiter->IsInstanceOf(
+          GenericRateLimiter::kClassName()));
+      auto* restore_rate_limiter_options =
+          options_.restore_rate_limiter
+              ->GetOptions<GenericRateLimiter::GenericRateLimiterOptions>();
+      assert(restore_rate_limiter_options);
+
+      RateLimiter::Mode restore_rate_limiter_mode;
+      if (!options_.restore_rate_limiter->IsRateLimited(
+              RateLimiter::OpType::kRead)) {
+        restore_rate_limiter_mode = RateLimiter::Mode::kWritesOnly;
+      } else if (!options_.restore_rate_limiter->IsRateLimited(
+                     RateLimiter::OpType::kWrite)) {
+        restore_rate_limiter_mode = RateLimiter::Mode::kReadsOnly;
+      } else {
+        restore_rate_limiter_mode = RateLimiter::Mode::kAllIo;
+      }
+      options_.restore_rate_limiter.reset(new GenericRateLimiter(
+          restore_rate_limiter_options->max_bytes_per_sec,
+          restore_rate_limiter_options->refill_period_us,
+          restore_rate_limiter_options->fairness, restore_rate_limiter_mode,
+          restore_rate_limiter_clock,
+          restore_rate_limiter_options->auto_tuned));
+    }
+  }
+
  private:
   void DeleteChildren(const std::string& dir,
                       uint32_t file_type_filter = 0) const;
@@ -942,6 +998,15 @@ class BackupEngineImplThreadSafe : public BackupEngine,
     impl_.schema_test_options_.reset(new TEST_BackupMetaSchemaOptions(options));
   }
 
+  // Not public API but used in testing
+  void TEST_SetDefaultRateLimitersClock(
+      const std::shared_ptr<SystemClock>& backup_rate_limiter_clock = nullptr,
+      const std::shared_ptr<SystemClock>& restore_rate_limiter_clock =
+          nullptr) {
+    impl_.TEST_SetDefaultRateLimitersClock(backup_rate_limiter_clock,
+                                           restore_rate_limiter_clock);
+  }
+
  private:
   mutable port::RWMutex mutex_;
   BackupEngineImpl impl_;
@@ -3162,6 +3227,15 @@ void TEST_SetBackupMetaSchemaOptions(
   impl->TEST_SetBackupMetaSchemaOptions(options);
 }
 
+void TEST_SetDefaultRateLimitersClock(
+    BackupEngine* engine,
+    const std::shared_ptr<SystemClock>& backup_rate_limiter_clock,
+    const std::shared_ptr<SystemClock>& restore_rate_limiter_clock) {
+  BackupEngineImplThreadSafe* impl =
+      static_cast_with_check<BackupEngineImplThreadSafe>(engine);
+  impl->TEST_SetDefaultRateLimitersClock(backup_rate_limiter_clock,
+                                         restore_rate_limiter_clock);
+}
 }  // namespace ROCKSDB_NAMESPACE
 
 #endif  // ROCKSDB_LITE

utilities/backup/backup_engine_impl.h

@@ -25,5 +25,12 @@ struct TEST_BackupMetaSchemaOptions {
 void TEST_SetBackupMetaSchemaOptions(
     BackupEngine *engine, const TEST_BackupMetaSchemaOptions &options);
 
+// Modifies the BackupEngine(Impl) to use specified clocks for backup and
+// restore rate limiters created by default if not specified by users for
+// test speedup.
+void TEST_SetDefaultRateLimitersClock(
+    BackupEngine* engine,
+    const std::shared_ptr<SystemClock>& backup_rate_limiter_clock = nullptr,
+    const std::shared_ptr<SystemClock>& restore_rate_limiter_clock = nullptr);
 }  // namespace ROCKSDB_NAMESPACE
 #endif  // ROCKSDB_LITE

utilities/backup/backup_engine_test.cc

@@ -2598,80 +2598,119 @@ INSTANTIATE_TEST_CASE_P(
 
 TEST_P(BackupEngineRateLimitingTestWithParam, RateLimiting) {
   size_t const kMicrosPerSec = 1000 * 1000LL;
-
-  std::shared_ptr<RateLimiter> backupThrottler(NewGenericRateLimiter(1));
-  std::shared_ptr<RateLimiter> restoreThrottler(NewGenericRateLimiter(1));
-
-  bool makeThrottler = std::get<0>(GetParam());
-  if (makeThrottler) {
-    engine_options_->backup_rate_limiter = backupThrottler;
-    engine_options_->restore_rate_limiter = restoreThrottler;
-  }
-
+  const bool custom_rate_limiter = std::get<0>(GetParam());
   // iter 0 -- single threaded
   // iter 1 -- multi threaded
-  int iter = std::get<1>(GetParam());
+  const int iter = std::get<1>(GetParam());
   const std::pair<uint64_t, uint64_t> limit = std::get<2>(GetParam());
-
+  std::unique_ptr<Env> special_env(
+      new SpecialEnv(db_chroot_env_.get(), /*time_elapse_only_sleep*/ true));
   // destroy old data
-  DestroyDB(dbname_, Options());
-  if (makeThrottler) {
-    backupThrottler->SetBytesPerSecond(limit.first);
-    restoreThrottler->SetBytesPerSecond(limit.second);
+  Options options;
+  options.env = special_env.get();
+  DestroyDB(dbname_, options);
+
+  if (custom_rate_limiter) {
+    std::shared_ptr<RateLimiter> backup_rate_limiter =
+        std::make_shared<GenericRateLimiter>(
+            limit.first, 100 * 1000 /* refill_period_us */, 10 /* fairness */,
+            RateLimiter::Mode::kWritesOnly /* mode */,
+            special_env->GetSystemClock(), false /* auto_tuned */);
+    std::shared_ptr<RateLimiter> restore_rate_limiter =
+        std::make_shared<GenericRateLimiter>(
+            limit.second, 100 * 1000 /* refill_period_us */, 10 /* fairness */,
+            RateLimiter::Mode::kWritesOnly /* mode */,
+            special_env->GetSystemClock(), false /* auto_tuned */);
+    engine_options_->backup_rate_limiter = backup_rate_limiter;
+    engine_options_->restore_rate_limiter = restore_rate_limiter;
   } else {
     engine_options_->backup_rate_limit = limit.first;
     engine_options_->restore_rate_limit = limit.second;
   }
+
   engine_options_->max_background_operations = (iter == 0) ? 1 : 10;
   options_.compression = kNoCompression;
+
+  // Rate limiter uses `CondVar::TimedWait()`, which does not have access to the
+  // `Env` to advance its time according to the fake wait duration. The
+  // workaround is to install a callback that advance the `Env`'s mock time.
+  ROCKSDB_NAMESPACE::SyncPoint::GetInstance()->SetCallBack(
+      "GenericRateLimiter::Request:PostTimedWait", [&](void* arg) {
+        int64_t time_waited_us = *static_cast<int64_t*>(arg);
+        special_env->SleepForMicroseconds(static_cast<int>(time_waited_us));
+      });
+  ROCKSDB_NAMESPACE::SyncPoint::GetInstance()->EnableProcessing();
+
   OpenDBAndBackupEngine(true);
-  size_t bytes_written = FillDB(db_.get(), 0, 100000);
+  TEST_SetDefaultRateLimitersClock(backup_engine_.get(),
+                                   special_env->GetSystemClock());
 
-  auto start_backup = db_chroot_env_->NowMicros();
+  size_t bytes_written = FillDB(db_.get(), 0, 10000);
+
+  auto start_backup = special_env->NowMicros();
   ASSERT_OK(backup_engine_->CreateNewBackup(db_.get(), false));
-  auto backup_time = db_chroot_env_->NowMicros() - start_backup;
+  auto backup_time = special_env->NowMicros() - start_backup;
+  CloseDBAndBackupEngine();
   auto rate_limited_backup_time = (bytes_written * kMicrosPerSec) / limit.first;
   ASSERT_GT(backup_time, 0.8 * rate_limited_backup_time);
 
-  CloseDBAndBackupEngine();
-
   OpenBackupEngine();
-  auto start_restore = db_chroot_env_->NowMicros();
+  TEST_SetDefaultRateLimitersClock(backup_engine_.get(),
+                                   special_env->GetSystemClock());
+
+  auto start_restore = special_env->NowMicros();
   ASSERT_OK(backup_engine_->RestoreDBFromLatestBackup(dbname_, dbname_));
-  auto restore_time = db_chroot_env_->NowMicros() - start_restore;
+  auto restore_time = special_env->NowMicros() - start_restore;
   CloseBackupEngine();
   auto rate_limited_restore_time =
       (bytes_written * kMicrosPerSec) / limit.second;
   ASSERT_GT(restore_time, 0.8 * rate_limited_restore_time);
 
-  AssertBackupConsistency(0, 0, 100000, 100010);
+  AssertBackupConsistency(0, 0, 10000, 10100);
+
+  ROCKSDB_NAMESPACE::SyncPoint::GetInstance()->DisableProcessing();
+  ROCKSDB_NAMESPACE::SyncPoint::GetInstance()->ClearCallBack(
+      "GenericRateLimiter::Request:PostTimedWait");
 }
 
 TEST_P(BackupEngineRateLimitingTestWithParam, RateLimitingVerifyBackup) {
   const std::size_t kMicrosPerSec = 1000 * 1000LL;
-  std::shared_ptr<RateLimiter> backupThrottler(NewGenericRateLimiter(
-      1, 100 * 1000 /* refill_period_us */, 10 /* fairness */,
-      RateLimiter::Mode::kAllIo /* mode */));
-
-  bool makeThrottler = std::get<0>(GetParam());
-  if (makeThrottler) {
-    engine_options_->backup_rate_limiter = backupThrottler;
-  }
-
-  bool is_single_threaded = std::get<1>(GetParam()) == 0 ? true : false;
-  engine_options_->max_background_operations = is_single_threaded ? 1 : 10;
-
+  const bool custom_rate_limiter = std::get<0>(GetParam());
   const std::uint64_t backup_rate_limiter_limit = std::get<2>(GetParam()).first;
-  if (makeThrottler) {
-    engine_options_->backup_rate_limiter->SetBytesPerSecond(
-        backup_rate_limiter_limit);
+  const bool is_single_threaded = std::get<1>(GetParam()) == 0 ? true : false;
+  std::unique_ptr<Env> special_env(
+      new SpecialEnv(db_chroot_env_.get(), /*time_elapse_only_sleep*/ true));
+
+  if (custom_rate_limiter) {
+    std::shared_ptr<RateLimiter> backup_rate_limiter =
+        std::make_shared<GenericRateLimiter>(
+            backup_rate_limiter_limit, 100 * 1000 /* refill_period_us */,
+            10 /* fairness */, RateLimiter::Mode::kAllIo /* mode */,
+            special_env->GetSystemClock(), false /* auto_tuned */);
+    engine_options_->backup_rate_limiter = backup_rate_limiter;
   } else {
     engine_options_->backup_rate_limit = backup_rate_limiter_limit;
   }
 
-  DestroyDB(dbname_, Options());
+  engine_options_->max_background_operations = is_single_threaded ? 1 : 10;
+
+  Options options;
+  options.env = special_env.get();
+  DestroyDB(dbname_, options);
+  // Rate limiter uses `CondVar::TimedWait()`, which does not have access to the
+  // `Env` to advance its time according to the fake wait duration. The
+  // workaround is to install a callback that advance the `Env`'s mock time.
+  ROCKSDB_NAMESPACE::SyncPoint::GetInstance()->SetCallBack(
+      "GenericRateLimiter::Request:PostTimedWait", [&](void* arg) {
+        int64_t time_waited_us = *static_cast<int64_t*>(arg);
+        special_env->SleepForMicroseconds(static_cast<int>(time_waited_us));
+      });
+  ROCKSDB_NAMESPACE::SyncPoint::GetInstance()->EnableProcessing();
   OpenDBAndBackupEngine(true /* destroy_old_data */);
-  FillDB(db_.get(), 0, 100000);
+  TEST_SetDefaultRateLimitersClock(backup_engine_.get(),
+                                   special_env->GetSystemClock(), nullptr);
+  FillDB(db_.get(), 0, 10000);
+
   ASSERT_OK(backup_engine_->CreateNewBackup(db_.get(),
                                             false /* flush_before_backup */));
 
@@ -2688,21 +2727,24 @@ TEST_P(BackupEngineRateLimitingTestWithParam, RateLimitingVerifyBackup) {
   for (BackupFileInfo backup_file_info : backup_info.file_details) {
     bytes_read_during_verify_backup += backup_file_info.size;
   }
-
-  auto start_verify_backup = db_chroot_env_->NowMicros();
+  auto start_verify_backup = special_env->NowMicros();
   ASSERT_OK(
       backup_engine_->VerifyBackup(backup_id, true /* verify_with_checksum */));
-  auto verify_backup_time = db_chroot_env_->NowMicros() - start_verify_backup;
+  auto verify_backup_time = special_env->NowMicros() - start_verify_backup;
   auto rate_limited_verify_backup_time =
       (bytes_read_during_verify_backup * kMicrosPerSec) /
       backup_rate_limiter_limit;
-
-  if (makeThrottler) {
+  if (custom_rate_limiter) {
     EXPECT_GE(verify_backup_time, 0.8 * rate_limited_verify_backup_time);
   }
+
   CloseDBAndBackupEngine();
-  AssertBackupConsistency(backup_id, 0, 100000, 100010);
-  DestroyDB(dbname_, Options());
+  AssertBackupConsistency(backup_id, 0, 10000, 10010);
+  DestroyDB(dbname_, options);
+
+  ROCKSDB_NAMESPACE::SyncPoint::GetInstance()->DisableProcessing();
+  ROCKSDB_NAMESPACE::SyncPoint::GetInstance()->ClearCallBack(
+      "GenericRateLimiter::Request:PostTimedWait");
 }
 
 TEST_P(BackupEngineRateLimitingTestWithParam, RateLimitingChargeReadInBackup) {