Summary:
Support for PowerPC Architecture
Detecting AltiVec Support
Closes https://github.com/facebook/rocksdb/pull/2353

Differential Revision: D5210948

Pulled By: siying

fbshipit-source-id: 859a8c063d37697addd89ba2b8a14e5efd5d24bf

Summary:
We initially had disabled support for write_options.sync when concurrent_prepare_ is set. We later added this support but the statement that asserts this combination is not used was left there. This patch cleans it up.
Closes https://github.com/facebook/rocksdb/pull/2642

Differential Revision: D5496101

Pulled By: maysamyabandeh

fbshipit-source-id: becbc503446f2a51bee24cc861958c090c724ec2

Summary:
The test is failing occasionally on the assert: `ASSERT_TRUE(writer->state == WriteThread::State::STATE_INIT)`. This is because the test don't make the leader wait for long enough before updating state for its followers. The patch move the update to `threads_waiting` to the end of `WriteThread::JoinBatchGroup:Wait` callback to avoid this happening.

Also adding `WriteThread::JoinBatchGroup:Start` and have each thread wait there while another thread is linking to the linked-list. This is to make the check of `is_leader` more deterministic.

Also changing two while-loops of `compare_exchange_strong` to plain `fetch_add`, to make it look cleaner.
Closes https://github.com/facebook/rocksdb/pull/2640

Differential Revision: D5491525

Pulled By: yiwu-arbug

fbshipit-source-id: 6e897f122082bd6f98e6d51b31a25e5fd0a3fb82

Summary:
5.6.1 release blog post
Closes https://github.com/facebook/rocksdb/pull/2638

Differential Revision: D5491168

Pulled By: yiwu-arbug

fbshipit-source-id: 14e3a92a03684afa4bd19bfb3ffb053cc09f5d4a

Summary: These are implied by default platform flags, in particular, `-march=corei7`.

Reviewed By: pixelb

Differential Revision: D5485414

fbshipit-source-id: 85f1329c71fa81a604760844187cc73877fb40e9

Summary:
Currently this test times out with tsan. This is likely due to decreased speed with tsan. By lowering the number of iterations we can still catch a bug as the test is run regularly and multiple runs of the test is equivalent with running the test with more iterations.
Closes https://github.com/facebook/rocksdb/pull/2639

Differential Revision: D5490549

Pulled By: maysamyabandeh

fbshipit-source-id: bd69c42a9728d337ac95a06a401088384e51731a

Summary: Closes https://github.com/facebook/rocksdb/pull/2637

Differential Revision: D5489751

Pulled By: maysamyabandeh

fbshipit-source-id: 0298f8960d4f86ce67959616615beee4d802c2e4

Differential Revision: D5470179

fbshipit-source-id: 475ad3b6a85ead5f8abddefddc607353ea8680af

Summary:
Fixes broken links to the introductory talk I stumbled upon while
reading the documentation.
Closes https://github.com/facebook/rocksdb/pull/2628

Differential Revision: D5483851

Pulled By: sagar0

fbshipit-source-id: 94aab7fb4c4ed2305680a2fbc65b14c7977af6b8

Summary: Closes https://github.com/facebook/rocksdb/pull/2629

Differential Revision: D5483751

Pulled By: sagar0

fbshipit-source-id: 9719ef9edd936dbb89b8988e3f4cb912a234f00e

Summary:
We will divide by zero if `stats.micros` is zero, just add a simple check
This happens sometimes during running tests and UBSAN complains
Closes https://github.com/facebook/rocksdb/pull/2631

Differential Revision: D5481455

Pulled By: IslamAbdelRahman

fbshipit-source-id: 69aa24e64e21de15d9e2b8009adf01675fcc6598

Summary: Closes https://github.com/facebook/rocksdb/pull/2613

Differential Revision: D5476064

Pulled By: sagar0

fbshipit-source-id: 6b30a9eacb93a945bbe499eafb90565fa9f1798b

Summary:
I haven't looked to see if a class variable inside a loop like this is always initialised.
Closes https://github.com/facebook/rocksdb/pull/2602

Differential Revision: D5475937

Pulled By: IslamAbdelRahman

fbshipit-source-id: 8570b308f9a4b49e2a56ccc9e9b84d7c46568c15

Summary:
As explained in the comments, Sometimes we create Slice(nullptr, 0) in our code base which cause us to do calls like
```
memcmp(nullptr, "abc", 0);
```
That's fine since the len is equal 0, but UBSAN is not happy about it
so disable UBSAN for this function and add an assert instead
Closes https://github.com/facebook/rocksdb/pull/2616

Differential Revision: D5458326

Pulled By: IslamAbdelRahman

fbshipit-source-id: cfca32abe30f7d8f760c9f77ecd9543dfb1170dd

Summary:
combining #2568 and #2612.
Closes https://github.com/facebook/rocksdb/pull/2620

Differential Revision: D5464394

Pulled By: IslamAbdelRahman

fbshipit-source-id: 9f71d3058dd6adaf02ce3b2de3a81a1228009778

Summary:
simply enable the macro in internal build, it wont hurt other sanitizers and will fix UBSAN issues
Closes https://github.com/facebook/rocksdb/pull/2625

Differential Revision: D5475897

Pulled By: IslamAbdelRahman

fbshipit-source-id: 262c6fd5de3c1906f4b29e55b39110f125f41057

Summary:
Add and implement Iterator::Refresh(). When this function is called, if the super version doesn't change, update the sequence number of the iterator to the latest one and invalidate the iterator. If the super version changed, recreated the whole iterator. This can help users reuse the iterator more easily.
Closes https://github.com/facebook/rocksdb/pull/2621

Differential Revision: D5464500

Pulled By: siying

fbshipit-source-id: f548bd35e85c1efca2ea69273802f6704eba6ba9

Summary:
The previous implementation of caching `file_size` index made no sense. It only remembered the original span of locked files starting from beginning of `file_size`. We should remember the index after all compactions that have been considered but rejected. This will reduce the work we do while holding the db mutex.
Closes https://github.com/facebook/rocksdb/pull/2624

Differential Revision: D5468152

Pulled By: ajkr

fbshipit-source-id: ab92a4bffe76f9f174d861bb5812b974d1013400

Summary:
This reverts the previous commit 1d7048c5, which broke the build.

Did a `git revert 1d7048c5`.
Closes https://github.com/facebook/rocksdb/pull/2627

Differential Revision: D5476473

Pulled By: sagar0

fbshipit-source-id: 4756ff5c0dfc88c17eceb00e02c36176de728d06

Summary: This uses `clang-tidy` to comment out unused parameters (in functions, methods and lambdas) in fbcode. Cases that the tool failed to handle are fixed manually.

Reviewed By: igorsugak

Differential Revision: D5454343

fbshipit-source-id: 5dee339b4334e25e963891b519a5aa81fbf627b2

Summary:
Major changes in this PR:
* Implement CassandraCompactionFilter to remove expired columns and rows (if all column expired)
* Move cassandra related code from utilities/merge_operators/cassandra to utilities/cassandra/*
* Switch to use shared_ptr<> from uniqu_ptr for Column membership management in RowValue. Since columns do have multiple owners in Merge and GC process, use shared_ptr helps make RowValue immutable.
* Rename cassandra_merge_test to cassandra_functional_test and add two TTL compaction related tests there.
Closes https://github.com/facebook/rocksdb/pull/2588

Differential Revision: D5430010

Pulled By: wpc

fbshipit-source-id: 9566c21e06de17491d486a68c70f52d501f27687

Summary:
Seems the only function of the script is to create a new branch, which can be done easily. I'm removing it.
Closes https://github.com/facebook/rocksdb/pull/2623

Differential Revision: D5468681

Pulled By: yiwu-arbug

fbshipit-source-id: 87dea5ecc4c85e06941ccbc36993f7f589063878

Summary:
Remove some of the per-key logging by blob db to reduce noise.
Closes https://github.com/facebook/rocksdb/pull/2587

Differential Revision: D5429115

Pulled By: yiwu-arbug

fbshipit-source-id: b89328282fb8b3c64923ce48738c16017ce7feaf

Summary:
In this test we are deleting 100 files, and we are expecting DeleteScheduler to delete 26 files in the background and 74 files immediately in the foreground

The main purpose of the test is to make sure that we delete files in foreground thread, which is verified in line 546

But sometimes we may end up with 26 files or 25 files in the trash directory because the background thread may be slow and not be able to delete the first file fast enough, so sometimes this test fail.

Remove
```
ASSERT_EQ(CountFilesInDir(trash_dir_), 25);
```
Since it does not have any benefit any way
Closes https://github.com/facebook/rocksdb/pull/2618

Differential Revision: D5458674

Pulled By: IslamAbdelRahman

fbshipit-source-id: 5556a9edfa049db71dce80b8e6ae0fdd25e1e74e

Summary:
This diff addresses two problems. Both problems cause us to miss scheduling desirable compactions. One side effect is compaction picking can spam logs, as there's no delay after failed attempts to pick compactions.

1. If a compaction pulled in a locked input-level file due to user-key overlap, we would not consider picking another file from the same input level.
2. If a compaction pulled in a locked output-level file due to user-key overlap, we would not consider picking any other compaction on any level.

The code changes are dependent, which is why I solved both problems in a single diff.

- Moved input-level `ExpandInputsToCleanCut` into the loop inside `PickFileToCompact`. This gives two benefits: (1) if it fails, we will try the next-largest file on the same input level; (2) we get the fully-expanded input-level key-range with which we can check for pending compactions in output level.
- Added another call to `ExpandInputsToCleanCut` inside `PickFileToCompact`'s to check for compaction conflicts in output level.
- Deleted call to `IsRangeInCompaction` in `PickFileToCompact`, as `ExpandInputsToCleanCut` also correctly handles the case where original output-level files (i.e., ones not pulled in due to user-key overlap) are pending compaction.
Closes https://github.com/facebook/rocksdb/pull/2615

Differential Revision: D5454643

Pulled By: ajkr

fbshipit-source-id: ea3fb5477d83e97148951af3fd4558d2039e9872

Summary:
I decided not even to keep it as an INFO-level log as it is too normal for compactions to be skipped due to locked input files. Removing logging here makes us consistent with how we treat locked files that weren't pulled in due to overlap.

We may want some error handling on line 422, which should never happen when called by `LevelCompactionBuilder::PickCompaction`, as `SetupInitialFiles` skips compactions where overlap causes the output level to pull in locked files.
Closes https://github.com/facebook/rocksdb/pull/2617

Differential Revision: D5458502

Pulled By: ajkr

fbshipit-source-id: c2e5f867c0a77c1812ce4242ab3e085b3eee0bae

Summary:
Moving static AdaptationContext to outside function to bypass tsan's false report with static initializers.

It is because with optimization enabled std::atomic is simplified to as a simple read with no locks. The existing lock produced by static initializer is __cxa_guard_acquire which is apparently not understood by tsan as it is different from normal locks (__gthrw_pthread_mutex_lock).

This is a known problem with tsan:
https://stackoverflow.com/questions/27464190/gccs-tsan-reports-a-data-race-with-a-thread-safe-static-local
https://stackoverflow.com/questions/42062557/c-multithreading-is-initialization-of-a-local-static-lambda-thread-safe

A workaround that I tried was to move the static variable outside the function. It is not a good coding practice since it gives global visibility to variable but it is a hackish workaround until g++ tsan is improved.
Closes https://github.com/facebook/rocksdb/pull/2598

Differential Revision: D5445281

Pulled By: yiwu-arbug

fbshipit-source-id: 6142bd934eb5852d8fd7ce027af593ba697ed41d

Summary: Closes https://github.com/facebook/rocksdb/pull/2610

Differential Revision: D5445705

Pulled By: siying

fbshipit-source-id: d5f97ffdf5bc18b853c3a106755aca96a650e428

Summary:
as titled
Closes https://github.com/facebook/rocksdb/pull/2601

Differential Revision: D5441618

Pulled By: ajkr

fbshipit-source-id: 334b098d90caadd3d07167d051cfb4ae65db98e1

Summary: Closes https://github.com/facebook/rocksdb/pull/2608

Differential Revision: D5444797

Pulled By: siying

fbshipit-source-id: 690581d03f37822e059a16085088e8e2d8a45016

Summary:
We don't need this format compatibility hack anymore. We should remove it to make things simpler.
Closes https://github.com/facebook/rocksdb/pull/2607

Differential Revision: D5444107

Pulled By: siying

fbshipit-source-id: 7ef587dd0cacfc15a4083a137adba8e6bfddac7e

Summary:
For forward_compatible_checkout_objs the local branch is already created in previous step. This patch avoid recreating it. This should address "fatal: A branch named '3.10.fb' already exists." errors.
Closes https://github.com/facebook/rocksdb/pull/2606

Differential Revision: D5443786

Pulled By: maysamyabandeh

fbshipit-source-id: 69d5a67b87677429cf36e3a467bd114d341f3b9c

Summary:
The new local branch specified with -b cannot be called master. Use tmp prefix to avoid name collision.
Closes https://github.com/facebook/rocksdb/pull/2600

Differential Revision: D5442944

Pulled By: maysamyabandeh

fbshipit-source-id: 4a623d9b21d6cc01bee812b2799790315bdf5f6e

Summary:
This will fix the error: "error: pathspec '2.2.fb.branch' did not match any file(s) known to git."

Tested by manually sshing to sandcastle and running the command.
Closes https://github.com/facebook/rocksdb/pull/2599

Differential Revision: D5441130

Pulled By: maysamyabandeh

fbshipit-source-id: a22fd6a52221471bafbba8990394b499535e5812

Summary:
Unfortunately we can't use -DNDEBUG yet since we don't properly exclude the test libraries/executables from the non-debug builds on non-MSVC platforms. Previously this was failing on Linux for every build type except `CMAKE_BUILD_TYPE=Debug`.

Reverts a48a62d5
Closes https://github.com/facebook/rocksdb/pull/2595

Differential Revision: D5436182

Pulled By: ajkr

fbshipit-source-id: 062f07cc9ce06a073b66054722b27bac1890dca3

Summary:
This patch enables using PinnableSlice for RowCache, changes include
not releasing the cache handle immediately after lookup in TableCache::Get, instead pass a Cleanble function which does Cache::RleaseHandle.
Closes https://github.com/facebook/rocksdb/pull/2492

Differential Revision: D5316216

Pulled By: maysamyabandeh

fbshipit-source-id: d2a684bd7e4ba73772f762e58a82b5f4fbd5d362

Summary:
Fix column_family_test with LITE build. I need this patch to fix 5.6 branch.
Closes https://github.com/facebook/rocksdb/pull/2597

Differential Revision: D5437171

Pulled By: yiwu-arbug

fbshipit-source-id: 88b9dc5925a6b47af10c1b41bc5b07c4251a84b5

Summary:
Should be 1024, obviously :)
Closes https://github.com/facebook/rocksdb/pull/2592

Differential Revision: D5435269

Pulled By: ajkr

fbshipit-source-id: c59338a3900798a4733f0b205e534f21215cf049

Summary:
In gcc-7 the following is an error identified by -Werror=class-memaccess

In file included from ./table/get_context.h:14:0,
                 from db/version_set.cc:43:
./table/block.h: In constructor ‘rocksdb::BlockReadAmpBitmap::BlockReadAmpBitmap(size_t, size_t, rocksdb::Statistics*)’:
./table/block.h:73:53: error: ‘void* memset(void*, int, size_t)’ clearing an object of type ‘struct std::atomic<unsigned int>’ with no trivial copy-assignment; use value-initialization instead [-Werror=class-memaccess]
     memset(bitmap_, 0, bitmap_size * kBytesPersEntry);
                                                     ^
In file included from ./db/version_set.h:23:0,
                 from db/version_set.cc:12:
/toolchain/include/c++/8.0.0/atomic:684:12: note: ‘struct std::atomic<unsigned int>’ declared here
     struct atomic<unsigned int> : __atomic_base<unsigned int>
            ^~~~~~~~~~~~~~~~~~~~

As a solution the default initializer can be applied in list context.
Signed-off-by: NDaniel Black <daniel.black@au.ibm.com>
Closes https://github.com/facebook/rocksdb/pull/2561

Differential Revision: D5398714

Pulled By: siying

fbshipit-source-id: d883fb88ec7535eee60d551038fe91f14488be36