Summary:
This diff introduce new table properties that will be written for block based tables
These properties are
  - comparator name
  - merge operator name
  - property collectors names

Test Plan:
  - Added a new unit test to verify that these tests are written/read correctly
  - Running all other tests right now (wont land until all tests finish)

Reviewers: rven, kradhakrishnan, igor, sdong, anthony, yhchiang

Reviewed By: yhchiang

Subscribers: dhruba

Differential Revision: https://reviews.facebook.net/D34269

Summary:
When a write batch can't join a batch group due to the total
size of the contained batches, the write controller's GetDelay is passed
a size value that includes the rejected batch.

Test Plan: make check

Reviewers: sdong, igor

Reviewed By: igor

Subscribers: dhruba

Differential Revision: https://reviews.facebook.net/D50343

Summary:
TSAN builds for gcc 4.9 need a PIC version of the libraries
taken from the fbcode platform.  This is accomplished by assuming every
.a has a _pic.a sibling, and by fixing the third-party2 zlib build.

Test Plan: make COMPILE_WITH_TSAN=1 check

Reviewers: sdong, igor

Reviewed By: igor

Subscribers: dhruba

Differential Revision: https://reviews.facebook.net/D50331

Summary: Docuemented what is currently supported by SingleDelete based on its current implementation.

Test Plan: n/a

Reviewers: sdong, kradhakrishnan, yhchiang, rven

Reviewed By: rven

Subscribers: dhruba, leveldb

Differential Revision: https://reviews.facebook.net/D50205

Improve concurrency when running tests

Summary: Use IterKey to store prefix_start_ so that it doesn't get freed

Test Plan: PrefixTest.PrefixValid

Reviewers: anthony, IslamAbdelRahman

Reviewed By: IslamAbdelRahman

Subscribers: dhruba

Differential Revision: https://reviews.facebook.net/D50289

  PowerShell seems to have a hard time when a flood of async tasks is
  scheduled at the same time.
  I speculated that WaitForMultipleObjects() in Windows can only take up
  to 64 process handles and if you want to handle more than you should write
  some additional code which can be sub-optimal. I.e to implement Wait-Job -Any.
  I decided to test that suggestion and introduced a $Concurrency parameter with a default value of 62.
  So in the new version the script fires up up to $Concurrency value
  and wait for anything to complete before starting any more processes.
  This improved matters greatly. Individual tests against ramdrive now
  run in 8 minutes and all of the 200+ db_tests run in 9 minutes with concurrency
  values of 8-16. About 48 is required to load a CPU on my box running against HD
  but that does not improve running times much.

  Other changes include respect -EnableJE for the individual test exes.
  Enforce exclusions for the individual tests.

Summary:
Valgrind reports an issue with the test for GeoIterator.
This diff explicitly deletes the two iterators used in this test.

Test Plan: This diff is for a test. The test still passes.

Reviewers: IslamAbdelRahman

Reviewed By: IslamAbdelRahman

Subscribers: dhruba, leveldb

Differential Revision: https://reviews.facebook.net/D50193

Summary:
MyRocks testing found an issue that while iterating over keys
that are outside the prefix, sometimes wrong results were seen for keys
outside the prefix. We now tighten the range of keys seen with a new
read option called prefix_seen_at_start. This remembers the starting
prefix and then compares it on a Next for equality of prefix. If they
are from a different prefix, it sets valid to false.

Test Plan: PrefixTest.PrefixValid

Reviewers: IslamAbdelRahman, sdong, yhchiang, anthony

Reviewed By: anthony

Subscribers: spetrunia, hermanlee4, yoshinorim, dhruba, leveldb

Differential Revision: https://reviews.facebook.net/D50211

Summary: As title. Let's run it for 1 hour.

Test Plan: How can I test legocastle changes?

Reviewers: IslamAbdelRahman, yhchiang, rven, sdong, anthony, kradhakrishnan

Reviewed By: kradhakrishnan

Subscribers: dhruba, leveldb

Differential Revision: https://reviews.facebook.net/D49653

Summary:
Move some of the items in public API changes section to to new
features section.

Test Plan: no code change.

Reviewers: igor, sdong, anthony, IslamAbdelRahman

Subscribers: dhruba, leveldb

Differential Revision: https://reviews.facebook.net/D50229

Summary: Fix build for clang

Test Plan:
USE_CLANG=1 make all -j64
make clean
make check -j64

Reviewers: yhchiang

Subscribers: dhruba

Differential Revision: https://reviews.facebook.net/D50217

Fix appveyor build failure

Summary:
This patch introduces OptionsSanityCheckLevel internally to enable
sanity check rocksdb options.

Utilities API will be added in the follow-up diffs.

Test Plan: Added more tests in options_test

Reviewers: igor, IslamAbdelRahman, sdong, anthony

Reviewed By: anthony

Subscribers: dhruba, leveldb

Differential Revision: https://reviews.facebook.net/D49515

Summary:
Fixed the following compile error in RocksDBLite:

18:00:33   CC       utilities/memory/memory_test.o
18:00:33 utilities/memory/memory_test.cc: In function ‘int main(int, char**)’:
18:00:33 utilities/memory/memory_test.cc:268:66: error: ‘printf’ was not declared in this scope
18:00:33    printf("Skipped in RocksDBLite as utilities are not supported.");
18:00:33                                                                   ^

Test Plan: make OPT=-DROCKSDB_LITE memory_test

Reviewers: igor, sdong, anthony, IslamAbdelRahman

Subscribers: dhruba

Differential Revision: https://reviews.facebook.net/D50145

Summary:
This patch introduces utilities/memory, which currently includes
GetApproximateMemoryUsageByType that reports different types of
rocksdb memory usage given a list of input DBs.

The API also take care of the case where Cache could be shared
across multiple column families / multiple db instances.

Currently, it reports memory usage of memtable, table-readers
and cache.

Test Plan: utilities/memory/memory_test.cc

Reviewers: igor, anthony, IslamAbdelRahman, sdong

Reviewed By: sdong

Subscribers: dhruba, leveldb

Differential Revision: https://reviews.facebook.net/D49257

Summary:
This patch adds GetAggregatedIntProperty() that returns the aggregated
value from all CFs

Test Plan: Added a test in db_test

Reviewers: igor, sdong, anthony, IslamAbdelRahman, rven

Reviewed By: rven

Subscribers: rven, dhruba, leveldb

Differential Revision: https://reviews.facebook.net/D49497

Summary:
This diff is a first step towards an iterator based interface for the
SearchRadial method which replaces a vector of GeoObjects with an
iterator for GeoObjects. This diff works by just wrapping the iterator
for the encapsulated vector of GeoObjects. A future diff could extend
this approach by defining an interator in terms of the underlying
iteration in SearchRadial which would then remove the need to have
an in-memory representation for all the matching GeoObjects.
Fixes T8421387

Test Plan:
The existing tests have been modified to work with the new
interface.

Reviewers: IslamAbdelRahman, kradhakrishnan, dhruba, igor

Reviewed By: igor

Subscribers: igor, dhruba, leveldb

Differential Revision: https://reviews.facebook.net/D50031

Add Option to Skip Flushing in TableBuilder

Enable compiler warnings

  C4101 'identifier' : unreferenced local variable
  C4189 'identifier' : local variable is initialized but not referenced
  C4100 'identifier' : unreferenced formal parameter
  C4296 'operator' : expression is always false

Fix introduced in 2ab7065a was reverted by 18285c1e.

Corrects:

db/memtablerep_bench.cc:135:22: error: ‘FLAGS_env’ defined but not used [-Werror=unused-variable]
 static rocksdb::Env* FLAGS_env = rocksdb::Env::Default();
                      ^
cc1plus: all warnings being treated as errors
Makefile:1147: recipe for target 'db/memtablerep_bench.o' failed

Do not suppress C4018 'expression' : signed/unsigned mismatch

  The code compiles cleanly for the most part. Fix db_test.
  Move debug file to testutil library.

Summary:
Update DB::AddFile() restrictions to be
  - Key range in loaded table file don't overlap with existing keys or tombstones in DB.
  - No other writes happen during AddFile call.

The updated AddFile() will verify that the file key range don't overlap with any keys or tombstones in the DB, and then add the file to L0

Test Plan: unit tests

Reviewers: igor, rven, anthony, kradhakrishnan, sdong

Reviewed By: sdong

Subscribers: adsharma, ameyag, dhruba

Differential Revision: https://reviews.facebook.net/D49233

Summary: Currently db_bnech's --compaction_pri default is set to be rocksdb::Options().compaction_style. Change it to rocksdb::Options().compaction_pri. Although, for now both is 0.

Test Plan: Build db_bench

Reviewers: anthony, rven, IslamAbdelRahman, kradhakrishnan, igor

Reviewed By: igor

Subscribers: leveldb, dhruba

Differential Revision: https://reviews.facebook.net/D49773

Summary: new_cf_opt.table_factory->Name() is char*, ASSERT_EQ doesn't work with char* directly. Construct a string using it.

Test Plan: Run the test that failed.

Reviewers: igor, kradhakrishnan, rven, IslamAbdelRahman, anthony

Reviewed By: anthony

Subscribers: leveldb, dhruba

Differential Revision: https://reviews.facebook.net/D49767

Support PlainTableOption in option_helper.cc

Summary:
CreateLoggerFromOptions have some parameters like  db_log_dir and env, these parameters are redundant since they already exist in DBOptions

this patch remove the redundant parameters and expose CreateLoggerFromOptions to users

Test Plan: make check

Reviewers: igor, anthony, yhchiang, rven, kradhakrishnan, sdong

Reviewed By: sdong

Subscribers: dhruba, hermanlee4

Differential Revision: https://reviews.facebook.net/D49713

Summary: Run "make format" for some recent commits.

Test Plan: Build and run tests

Reviewers: IslamAbdelRahman

Reviewed By: IslamAbdelRahman

Subscribers: leveldb, dhruba

Differential Revision: https://reviews.facebook.net/D49707

Implement smart buffer management in Windows Env.

Fix compilation problem on Windows.