Test Plan: compile

Reviewers: dhruba

Reviewed By: dhruba

CC: leveldb

Differential Revision: https://reviews.facebook.net/D7857

Summary:
Pretty much a blind copy of the patch in open source.
Hope to get this in before we make a release

Test Plan: make clean check

Reviewers: dhruba, heyongqiang

Reviewed By: dhruba

CC: leveldb

Differential Revision: https://reviews.facebook.net/D7809

Summary: Ran ./build_java.sh bump_version 1.5.7

Test Plan: automated change

Reviewers: dhruba

Reviewed By: dhruba

CC: leveldb

Differential Revision: https://reviews.facebook.net/D7833

Summary:
`Table::Open()` assumes that `size` correctly describes the size of `file`, added a check that the footer is actually the right size and for good measure added assertions to `Footer::DecodeFrom()`.

This was discovered by running `valgrind ./db_test` and seeing that `Footer::DecodeFrom()` was accessing uninitialized memory.

Test Plan:
make clean check

ran `valgrind ./db_test` and saw DBTest.NoSpace no longer complains about a conditional jump being dependent on uninitialized memory.

Reviewers: dhruba, vamsi, emayanke, sheki

Reviewed By: dhruba

CC: leveldb

Differential Revision: https://reviews.facebook.net/D7815

Summary:

Test Plan:

Reviewers:

CC:

Task ID: #

Blame Rev:

Summary:
The asserts introduced in https://reviews.facebook.net/D7629 are
wrong.

The direction of iteration is changed after the function call so they
assert's fail.

Test Plan: make clean check

Reviewers: dhruba

Reviewed By: dhruba

CC: leveldb

Differential Revision: https://reviews.facebook.net/D7827

Summary: Found some issues running Valgrind on `db_test` (there are still some outstanding ones) and fixed them.

Test Plan:
make check

ran `valgrind ./db_test` and saw that errors no longer occur

Reviewers: dhruba, vamsi, emayanke, sheki

Reviewed By: dhruba

CC: leveldb

Differential Revision: https://reviews.facebook.net/D7803

Summary: `~ShardedLRUCache()` was empty despite `init()` allocating memory on the heap. Fixed the leak by freeing memory allocated by `init()`.

Test Plan:
make check

Ran valgrind on db_test before and after patch and saw leaked memory went down

Reviewers: vamsi, dhruba, emayanke, sheki

Reviewed By: dhruba

CC: leveldb

Differential Revision: https://reviews.facebook.net/D7791

Summary:
This was a peformance regression caused by https://reviews.facebook.net/D6729.
The default value of max_grandparent_overlap_factor was erroneously
set to 0 in db_bench.

This was causing compactions to create really really small files because the max_grandparent_overlap_factor was erroneously set to zero in the benchmark.

Test Plan: Run --benchmarks=overwrite

Reviewers: heyongqiang, emayanke, sheki, MarkCallaghan

Reviewed By: sheki

CC: leveldb

Differential Revision: https://reviews.facebook.net/D7797

Summary:
Changed CreateDir() to CreateDirIfMissing() so a directory that already exists now causes and error.

Fixed CreateDirIfMissing() and added Env.DirExists()

Test Plan:
make check to test for regessions

Ran the following to test if the error message is not about lock files not existing
./db_bench --db=dir/testdb

After creating a file "testdb", ran the following to see if it failed with sane error message:
./db_bench --db=testdb

Reviewers: dhruba, emayanke, vamsi, sheki

Reviewed By: emayanke

CC: leveldb

Differential Revision: https://reviews.facebook.net/D7707

Summary:
Adds the option --seed to db_bench to specify the base for the per-thread RNG.
When not set each thread uses the same value across runs of db_bench which defeats
IO stress testing.

Adds the option --read_range. When set to a value > 1 an iterator is created and
each query done for the randomread benchmark will do a range scan for that many
rows. When not set or set to 1 the existing behavior (a point lookup) is done.

Fixes a bug where a printf format string was missing.

Test Plan: run db_bench

Reviewers: dhruba

Reviewed By: dhruba

CC: leveldb

Differential Revision: https://reviews.facebook.net/D7749

Summary:
`MemTableList::Add()` neglected to mention that it took ownership of the reference held by its caller.

The comment in `MemTable::Get()` was wrong in describing the format of the key.

Test Plan: None

Reviewers: dhruba, sheki, emayanke, vamsi

Reviewed By: dhruba

CC: leveldb

Differential Revision: https://reviews.facebook.net/D7755

Summary:
Use a std::priority_queue in merger.cc instead of doing a o(n) search
every time.
Currently only the ForwardIteration uses a Priority Queue.

Test Plan: make all check

Reviewers: dhruba

Reviewed By: dhruba

CC: emayanke, zshao

Differential Revision: https://reviews.facebook.net/D7629

Summary:
There was a bug in the ExtendOverlappingInputs method so that
the terminating condition for the backward search was incorrect.

Test Plan: make clean check

Reviewers: sheki, emayanke, MarkCallaghan

Reviewed By: MarkCallaghan

CC: leveldb

Differential Revision: https://reviews.facebook.net/D7725

Summary:
1. The thrift libraries do not need to be built anyore.
2. SSE is dynamically detected via
   https://github.com/facebook/rocksdb/commit/1aae609b920f8cd4d93ac49798fa96367b9b864c

Test Plan: compile and build

Reviewers: sheki, emayanke

Reviewed By: sheki

CC: leveldb

Differential Revision: https://reviews.facebook.net/D7665

Summary:
There is a compilation error while using gcc 4.7.1.
util/ldb_cmd.cc:381:3: error: ‘leveldb::ReadOptions::ReadOptions’ names the constructor, not the type
util/ldb_cmd.cc:381:37: error: expected ‘;’ before ‘read_options’
util/ldb_cmd.cc:381:49: error: statement cannot resolve address of overloaded function

Test Plan: make clean check

Reviewers: sheki, emayanke, zshao

Reviewed By: emayanke

CC: leveldb

Differential Revision: https://reviews.facebook.net/D7659

Summary: The queries will come from stdin.  One key per line.  The output will be in stdout, in the format of "<key> ==> <value>" if found, or "<key>" if not found.  "--hex" uses HEX-encoded keys and values in both input and output.

Test Plan: ldb query --db=leveldb_db --hex

Reviewers: dhruba, emayanke, sheki

Reviewed By: dhruba

CC: leveldb

Differential Revision: https://reviews.facebook.net/D7617

Summary: We were ignoring additional chars at the end of an arg.  This can create confusion, e.g. --disable_wal=0 will act the same as --disable_wal without any warnings.

Test Plan:
Tried this:
[zshao@dev485 ~/git/rocksdb] ./ldb dump --statsAAA
Failed: Unknown argument:--statsAAA

Reviewers: dhruba, sheki, emayanke

Reviewed By: dhruba

CC: leveldb

Differential Revision: https://reviews.facebook.net/D7635

Summary: The current parsing logic ignores any additional chars after the arg.

Test Plan: "./sst_dump --verify_checksumAAA" now outputs error.

Reviewers: dhruba, sheki, emayanke

Reviewed By: dhruba

CC: leveldb

Differential Revision: https://reviews.facebook.net/D7611

Summary:
Create an Executable with
* A thread to do Put's
* A thread to use GetUpdatesSince. Check if we miss any sequence
  Numbers.

Test Plan: It runs.

Reviewers: dhruba

Reviewed By: dhruba

CC: leveldb

Differential Revision: https://reviews.facebook.net/D7383

Summary: ldb uses --output_hex so make manifest_dump do the same thing.

Test Plan:
[zshao@dev485 ~/git/rocksdb] ./manifest_dump --output_hex --file=/data/users/zshao/test_leveldb/MANIFEST-000034
manifest_file_number 42 next_file_number 43 last_sequence 2311567 log_number 36  prev_log_number 0
--- level 0 --- version# 0 ---
--- level 1 --- version# 0 ---
--- level 2 --- version# 0 ---
--- level 3 --- version# 0 ---
 5:27788699['0000027F4FBE0000' @ 1 : 1 .. '11CE749602C90000' @ 160642 : 1]
 7:27785313['11CE773DA7E00000' @ 160643 : 1 .. '23A4C63EC55D0000' @ 321094 : 1]
 9:27784288['23A4D581FCD30000' @ 321095 : 1 .. '3576291D12D00000' @ 481428 : 1]
 38:64378271['35762BF0E0CE0000' @ 481429 : 1 .. '5E987E0604700000' @ 852910 : 1]
 39:64379046['5E987EB0BDD50000' @ 852911 : 1 .. '87C954330E840000' @ 1224603 : 1]
 40:10169201['87C95507E49C0000' @ 1224604 : 1 .. '8E48DC0933B70000' @ 1283317 : 1]
 21:27798825['8E48DFB0D7CE0000' @ 1283318 : 1 .. 'A00675F8AD7E0000' @ 1443826 : 1]
 23:27793751['A006777536E30000' @ 1443827 : 1 .. 'B1D1787FE8670000' @ 1604553 : 1]
 25:27801659['B1D179289BB30000' @ 1604554 : 1 .. 'C396D3A69DCE0000' @ 1765012 : 1]
 27:27792661['C396DA1E03B10000' @ 1765013 : 1 .. 'D55C9974FCC10000' @ 1925513 : 1]
 29:27789095['D55C9B47CBC00000' @ 1925514 : 1 .. 'E71F67D11CCC0000' @ 2085789 : 1]
 31:27793145['E71F7A667E740000' @ 2085790 : 1 .. 'F8D4712EF3D90000' @ 2246454 : 1]
 41:11246031['F8D4715916A70000' @ 2246455 : 1 .. 'FFFFFCAE97DF0000' @ 2311567 : 1]
--- level 4 --- version# 0 ---
--- level 5 --- version# 0 ---
--- level 6 --- version# 0 ---

Reviewers: dhruba, sheki, emayanke

Reviewed By: dhruba

CC: leveldb

Differential Revision: https://reviews.facebook.net/D7575

Summary: Now sst_dump has the same option --output_hex as "ldb dump" and also share the same output format.  So we can do "sst_dump ... | ldb load ..." for an experiment.

Test Plan:
[zshao@dev485 ~/git/rocksdb] ./sst_dump --file=/data/users/zshao/test_leveldb/000005.sst  --output_hex | head -n 2
0000027F4FBE00000101000000000000 ==> D901000000000000000057596F7520726563656976656420746F6461792773207370656369616C20676966742120436C69636B2041636365707420746F207669657720796F75722047696674206265666F72652069742064697361707065617273210000000000000000
000007F9C2D400000102000000000000 ==> D1010000000000000000544974277320676F6F6420746F206265204B696E67E280A6206F7220517565656E212048657265277320796F75722054756573646179204D79737465727920476966742066726F6D20436173746C6556696C6C65219B7B227A636F6465223A22633566306531633039663764222C227470223A22613275222C227A6B223A22663638663061343262666264303966383435666239626235366365396536643024246234506C345139382E734C4D33522169482D4F31315A64794C4B7A4F4653766D7863746534625F2A3968684E3433786521776C427636504A414355795F70222C227473223A313334343936313737357D0000000000000000

Reviewers: dhruba, sheki, emayanke

Reviewed By: dhruba

CC: leveldb

Differential Revision: https://reviews.facebook.net/D7587

Summary:
This allows ldb to control the write_buffer_size (which reflects to L0 file size) and file_size (which reflects to L1 file size).  Since the target_file_size_ratio is 1 by default, all other levels will also have the same file size as L1.

As part of the diff, I also cleaned up some unused code and help messages.

Test Plan: ./ldb load --db=/data/users/zshao/test_leveldb --file_size=64000000 --write_buffer_size=32000000 --create_if_missing --input_hex --disable_wal

Reviewers: dhruba, sheki, emayanke

Reviewed By: dhruba

CC: leveldb

Differential Revision: https://reviews.facebook.net/D7569

Test Plan: make all check

Reviewers: dhruba

Reviewed By: dhruba

CC: leveldb, zshao

Differential Revision: https://reviews.facebook.net/D7503

Summary: It took me almost a day to debug this. :(  Although I got to learn the file format as a by-product, this time could be saved if we have better error messages.

Test Plan: gmake clean all; sst_dump --hex --file=000005.sst

Reviewers: dhruba

Reviewed By: dhruba

CC: leveldb

Differential Revision: https://reviews.facebook.net/D7551

Summary: This is a standalone unit test to test the format of a block.

Test Plan: ./block_test

Reviewers: sheki

Reviewed By: sheki

Differential Revision: https://reviews.facebook.net/D7533

Summary:
Without this fix, I see failures like this:
[zshao@dev1049 /data/users/zshao/rocksdb] . fbcode.gcc471.sh; gmake clean libleveldb.a
. . .
./thrift/lib/cpp/async/TEventUtil.h:22:32: fatal error: event.h: No such file or directory

Test Plan: . fbcode.gcc471.sh; make clean libleveldb.a

Reviewers: dhruba, emayanke, sheki

Reviewed By: sheki

CC: leveldb

Differential Revision: https://reviews.facebook.net/D7497

Summary:
Leveldb has an api OpenForReadOnly() that opens the database
in readonly mode. This call had an option to not process the
transaction log.  This patch removes this option and always
processes all transactions that had been committed. It has
been done in such a way that it does not create/write to
any new files in the process. The invariant of "no-writes"
to the leveldb data directory is still true.

This enhancement allows multiple threads to open the same database
in readonly mode and access all trancations that were committed right
upto the OpenForReadOnly call.

I changed the public API to match the new semantics because
there are no users who are currently using this api.

Test Plan: make clean check

Reviewers: sheki

Reviewed By: sheki

CC: leveldb

Differential Revision: https://reviews.facebook.net/D7479

Summary: This command accepts key-value pairs from stdin with the same format of "ldb dump" command.  This allows us to try out different compression algorithms/block sizes easily.

Test Plan: dump, load, dump, verify the data is the same.

Reviewers: dhruba

Reviewed By: dhruba

CC: leveldb

Differential Revision: https://reviews.facebook.net/D7443

Test Plan: it compiles and deploys.

Reviewers: dhruba

CC: leveldb

Differential Revision: https://reviews.facebook.net/D7341

Summary:
1. The OpenForReadOnly() call should not lock the db. This is useful
so that multiple processes can open the same database concurrently
for reading.
2. GetUpdatesSince should not error out if the archive directory
does not exist.
3. A new constructor for WriteBatch that can takes a serialized
string as a parameter of the constructor.

Test Plan: make clean check

Reviewers: sheki

Reviewed By: sheki

CC: leveldb

Differential Revision: https://reviews.facebook.net/D7449

Summary:
Added kMetaDatabase for meta-databases in db/filename.h along with supporting
fuctions.
Fixed switch in DBImpl so that it also handles kMetaDatabase.
Fixed DestroyDB() that it can handle destroying meta-databases.

Test Plan: make check

Reviewers: sheki, emayanke, vamsi, dhruba

Reviewed By: dhruba

Differential Revision: https://reviews.facebook.net/D7245

Summary:
C tests would fail sometimes as DestroyDB would return a Failure Status
message when deleting an archival directory which was not created
(WAL_ttl_seconds = 0).

Fix: Ignore the Status returned on Deleting Archival Directory.

Test Plan: * make check

Reviewers: dhruba, emayanke

Reviewed By: dhruba

CC: leveldb

Differential Revision: https://reviews.facebook.net/D7395

Summary: The old code was omitting the 0 if the char is less than 16.

Test Plan:
Tried the following program:
int main() {
  unsigned char c = 1;
  printf("%X\n", c);
  printf("%02X\n", c);
  return 0;
}
The output is:
1
01

Reviewers: dhruba

Reviewed By: dhruba

CC: leveldb

Differential Revision: https://reviews.facebook.net/D7437

Summary: This allows us to use ldb to do more experiments like block_size changes.

Test Plan: run it by hand.

Reviewers: dhruba, sheki, emayanke

Reviewed By: dhruba

CC: leveldb

Differential Revision: https://reviews.facebook.net/D7431

Summary: Without this option, manifest_dump does not print binary keys for files in a human-readable way.

Test Plan:
./manifest_dump --hex=1 --verbose=0 --file=/data/users/zshao/fdb_comparison/leveldb/fbobj.apprequest-0_0_original/MANIFEST-000002
manifest_file_number 589 next_file_number 590 last_sequence 2311567 log_number 543  prev_log_number 0
--- level 0 --- version# 0 ---
 532:1300357['0000455BABE20000' @ 2183973 : 1 .. 'FFFCA5D7ADE20000' @ 2184254 : 1]
 536:1308170['000198C75CE30000' @ 2203313 : 1 .. 'FFFCF94A79E30000' @ 2206463 : 1]
 542:1321644['0002931AA5E50000' @ 2267055 : 1 .. 'FFF77B31C5E50000' @ 2270754 : 1]
 544:1286390['000410A309E60000' @ 2278592 : 1 .. 'FFFE470A73E60000' @ 2289221 : 1]
 538:1298778['0006BCF4D8E30000' @ 2217050 : 1 .. 'FFFD77DAF7E30000' @ 2220489 : 1]
 540:1282353['00090D5356E40000' @ 2231156 : 1 .. 'FFFFF4625CE40000' @ 2231969 : 1]
--- level 1 --- version# 0 ---
 510:2112325['000007F9C2D40000' @ 1782099 : 1 .. '146F5B67B8D80000' @ 1905458 : 1]
 511:2121742['146F8A3023D60000' @ 1824388 : 1 .. '28BC8FBB9CD40000' @ 1777993 : 1]
 512:801631['28BCD396F1DE0000' @ 2080191 : 1 .. '3082DBE9ADDB0000' @ 1989927 : 1]

Reviewers: dhruba, sheki, emayanke

Reviewed By: dhruba

CC: leveldb

Differential Revision: https://reviews.facebook.net/D7425

Summary:
old version of linters use "lint_engine" instead of "lint.engine"
Some bookeeping in gitignore.
Reviewers: abhishekk

Summary:
WriteBatch is now used by the GetUpdatesSinceAPI. This API is external
and will be used by the rocks server. Rocks Server and others will need
to know about the Sequence Number in the WriteBatch. This public method
will allow for that.

Test Plan: make all check.

Reviewers: dhruba

Reviewed By: dhruba

CC: leveldb

Differential Revision: https://reviews.facebook.net/D7293

Summary:
Expose the serialized string that represents a WriteBatch. This
is helpful to replicate a writebatch operation from one machine
to another.

Test Plan: make clean check

Reviewers: sheki

Reviewed By: sheki

CC: leveldb

Differential Revision: https://reviews.facebook.net/D7317

Summary:
Debug and ported changes from the Open Source Github repo to our repo.
Wrote a script to easy build the java Library. future compiling java lib should just be running this script.

Test Plan: it compiles.

Reviewers: dhruba, leveldb

Reviewed By: dhruba

Differential Revision: https://reviews.facebook.net/D7323