Fix the potential DB crash caused by call EndTrace before StartTrace (#5130)

Summary:
Although user should first call StartTrace to begin the RocksDB tracing function and call EndTrace to stop the tracing process, user can accidentally call EndTrace first. It will cause segment fault and crash the DB instance. The issue is fixed by checking the pointer first.

Test case added in db_test2.
Pull Request resolved: https://github.com/facebook/rocksdb/pull/5130

Differential Revision: D14691420

Pulled By: zhichao-cao

fbshipit-source-id: 3be13d2f944bc453728ef8eef67b68d7ad0939c8

db/db_impl.cc

@@ -3655,8 +3655,13 @@ Status DBImpl::StartTrace(const TraceOptions& trace_options,
 
 Status DBImpl::EndTrace() {
   InstrumentedMutexLock lock(&trace_mutex_);
-  Status s = tracer_->Close();
-  tracer_.reset();
+  Status s;
+  if (tracer_ != nullptr) {
+    s = tracer_->Close();
+    tracer_.reset();
+  } else {
+    return Status::IOError("No trace file to close");
+  }
   return s;
 }
 

db/db_test2.cc

@@ -2875,6 +2875,8 @@ TEST_F(DBTest2, TraceAndReplay) {
   Random rnd(301);
   Iterator* single_iter = nullptr;
 
+  ASSERT_TRUE(db_->EndTrace().IsIOError());
+
   std::string trace_filename = dbname_ + "/rocksdb.trace";
   std::unique_ptr<TraceWriter> trace_writer;
   ASSERT_OK(NewFileTraceWriter(env_, env_opts, trace_filename, &trace_writer));