- 08 12月, 2016 1 次提交
-
-
由 Jonathan Lee 提交于
Summary: The second variable "SHELL" simply tells make explicitly which shell to use, instead of allowing it to default to "/bin/sh", which may or may not be Bash. However, simply defining the second variable by itself causes make to throw an error concerning a circular definition, as it would be attempting to use the "shell" command while simultaneously trying to set which shell to use. Thus, the first variable "BASH_EXISTS" is defined such that make already knows about "/path/to/bash" before trying to use it to set "SHELL". A more technically correct solution would be to edit the makefile itself to make it compatible with non-bash shells (see the original Issue discussion for details). However, as it seems very few of the people working on this project were building with non-bash shells, I figured this solution would be good enough. Closes https://github.com/facebook/rocksdb/pull/1631 Differential Revision: D4295689 Pulled By: yiwu-arbug fbshipit-source-id: e4f9532
-
- 30 11月, 2016 1 次提交
-
-
由 Andrew Kryczka 提交于
Summary: Closes https://github.com/facebook/rocksdb/pull/1578 Differential Revision: D4241171 Pulled By: ajkr fbshipit-source-id: ce5fd83
-
- 19 11月, 2016 1 次提交
-
-
由 Siying Dong 提交于
Summary: Travis now is building for ldb tests. Disable for now to unblock other tests while we are investigating. Closes https://github.com/facebook/rocksdb/pull/1546 Differential Revision: D4209404 Pulled By: siying fbshipit-source-id: 47edd97
-
- 17 11月, 2016 1 次提交
-
-
由 Yueh-Hsuan Chiang 提交于
Summary: This diff includes an implementation of CompactionFilter that allows users to write CompactionFilter in Lua. With this ability, users can dynamically change compaction filter logic without requiring building the rocksdb binary and restarting the database. To compile, WITH_LUA_PATH must be specified to the base directory of lua. Closes https://github.com/facebook/rocksdb/pull/1478 Differential Revision: D4150138 Pulled By: yhchiang fbshipit-source-id: ed84222
-
- 15 11月, 2016 1 次提交
-
-
由 Yi Wu 提交于
Summary: Parallize persistent_cache_test and transaction_test Closes https://github.com/facebook/rocksdb/pull/1506 Differential Revision: D4179392 Pulled By: IslamAbdelRahman fbshipit-source-id: 05507a1
-
- 14 11月, 2016 1 次提交
-
-
由 Yi Wu 提交于
Summary: Currently our skip-list have an optimization to speedup sequential inserts from a single stream, by remembering the last insert position. We extend the idea to support sequential inserts from multiple streams, and even tolerate small reordering wihtin each stream. This PR is the interface part adding the following: - Add `memtable_insert_prefix_extractor` to allow specifying prefix for each key. - Add `InsertWithHint()` interface to memtable, to allow underlying implementation to return a hint of insert position, which can be later pass back to optimize inserts. - Memtable will maintain a map from prefix to hints and pass the hint via `InsertWithHint()` if `memtable_insert_prefix_extractor` is non-null. Closes https://github.com/facebook/rocksdb/pull/1419 Differential Revision: D4079367 Pulled By: yiwu-arbug fbshipit-source-id: 3555326
-
- 01 11月, 2016 1 次提交
-
-
由 ananclub 提交于
Summary: Closes https://github.com/facebook/rocksdb/pull/1441 Differential Revision: D4103477 Pulled By: yiwu-arbug fbshipit-source-id: 071a0dc
-
- 26 10月, 2016 1 次提交
-
-
由 Kefu Chai 提交于
* util/build_verion.cc.in: add this file, so cmake and make can share the template file for generating util/build_version.cc. * CMakeLists.txt: also, cmake v2.8.11 does not support file(GENERATE ...), so we are using configure_file() for creating build_version.cc. * Makefile: use util/build_verion.cc.in for creating build_version.cc. Signed-off-by: NKefu Chai <tchaikov@gmail.com>
-
- 22 10月, 2016 1 次提交
-
-
由 Anirban Rahut 提交于
Summary: Valgrind does not work well with JEMALLOC. If you run a simple make valgrind_check, you will see lots of issues and crashes. When precommit runs, this is taken care of. Here we make sure valgrind_check is passed in DISABLE_JEMALLOC=1 Test Plan: Ran local valgrind_test and noticed the difference Reviewers: IslamAbdelRahman Reviewed By: IslamAbdelRahman Subscribers: andrewkr, dhruba Differential Revision: https://reviews.facebook.net/D65379
-
- 04 10月, 2016 1 次提交
-
-
由 Adam Retter 提交于
-
- 24 9月, 2016 1 次提交
-
-
由 yiwu-arbug 提交于
Summary: Revert the behavior where we don't read sequence id from WAL, but increase it as we replay the log. We still keep the behave for 2PC for now but will fix later. This change fixes github issue 1339, where some writes come with WAL disabled and we may recover records with wrong sequence id. Test Plan: Added unit test. Subscribers: andrewkr, dhruba Differential Revision: https://reviews.facebook.net/D64275
-
- 17 9月, 2016 1 次提交
-
-
由 Yi Wu 提交于
Summary: Fix dependencies in java makefile, to avoid java build failure. Test Plan: run "make rocksdbjava" multiple times. Reviewers: IslamAbdelRahman, sdong, yhchiang Reviewed By: yhchiang Subscribers: andrewkr, dhruba, leveldb Differential Revision: https://reviews.facebook.net/D64023
-
- 08 9月, 2016 1 次提交
-
-
由 Islam AbdelRahman 提交于
Summary: Simply move the tests Test Plan: make check -j64 Reviewers: andrewkr, lightmark, yiwu, yhchiang, kradhakrishnan, sdong Reviewed By: sdong Subscribers: andrewkr, dhruba Differential Revision: https://reviews.facebook.net/D62529
-
- 29 8月, 2016 1 次提交
-
-
由 Adam Retter 提交于
* Fix the download location of Snappy, no longer available from Google Code * Ensure that curl follows any redirect headers when downloading binaries
-
- 26 8月, 2016 1 次提交
-
-
由 Islam AbdelRahman 提交于
Summary: I just realized that when we run parallel valgrind we actually don't run the parallel tests under valgrind (we run the normally) This patch make sure that we run both parallel and non-parallel tests with valgrind Test Plan: DISABLE_JEMALLOC=1 make valgrind_check -j64 Reviewers: andrewkr, yiwu, lightmark, sdong Reviewed By: sdong Subscribers: andrewkr, dhruba Differential Revision: https://reviews.facebook.net/D62469
-
- 20 8月, 2016 1 次提交
-
-
由 Yi Wu 提交于
Summary: Add mid-point insertion functionality to LRU cache. Caller of `Cache::Insert()` can set an additional parameter to make a cache entry have higher priority. The LRU cache will reserve at most `capacity * high_pri_pool_pct` bytes for high-pri cache entries. If `high_pri_pool_pct` is zero, the cache degenerates to normal LRU cache. Context: If we are to put index and filter blocks into RocksDB block cache, index/filter block can be swap out too early. We want to add an option to RocksDB to reserve some capacity in block cache just for index/filter blocks, to mitigate the issue. In later diffs I'll update block based table reader to use the interface to cache index/filter blocks at high priority, and expose the option to `DBOptions` and make it dynamic changeable. Test Plan: unit test. Reviewers: IslamAbdelRahman, sdong, lightmark Reviewed By: lightmark Subscribers: andrewkr, dhruba, march, leveldb Differential Revision: https://reviews.facebook.net/D61977
-
- 11 8月, 2016 4 次提交
-
-
由 krad 提交于
Summary: Adding benchmark tool for persistent read cache. TODO: Add integration to db_bench Test Plan: Compile Subscribers: andrewkr, dhruba, leveldb Differential Revision: https://reviews.facebook.net/D57711
-
由 sdong 提交于
Summary: This is a proof of concept of a RocksDB blob log file. The actual value of the Put() is appended to a blob log using normal data block format, and the handle of the block is written as the value of the key in RocksDB. The prototype only supports Put() and Get(). It doesn't support DB restart, garbage collection, Write() call, iterator, snapshots, etc. Test Plan: Add unit tests. Reviewers: arahut Reviewed By: arahut Subscribers: kradhakrishnan, leveldb, andrewkr, dhruba Differential Revision: https://reviews.facebook.net/D61485
-
由 sdong 提交于
read_options.background_purge_on_iterator_cleanup to cover forward iterator and log file closing too. Summary: With read_options.background_purge_on_iterator_cleanup=true, File deletion and closing can still happen in forward iterator, or WAL file closing. Cover those cases too. Test Plan: I am adding unit tests. Reviewers: andrewkr, IslamAbdelRahman, yiwu Reviewed By: yiwu Subscribers: leveldb, andrewkr, dhruba Differential Revision: https://reviews.facebook.net/D61503
-
由 Andrew Kryczka 提交于
Summary: compact_on_deletion_collector_test does not support --gtest_list_tests since it isn't gtest, so the full program would run for the target gen_parallel_tests. This caused gen_parallel_tests to take 8+ minutes for tsan and prevented compact_on_deletion_collector_test from running during check_0 since no t/run-* script could be generated. Test Plan: run make check, verify generating t/run-* scripts is fast and ./compact_on_deletion_collector_test is now run Reviewers: IslamAbdelRahman, wanning, lightmark, sdong Reviewed By: sdong Subscribers: andrewkr, dhruba, leveldb Differential Revision: https://reviews.facebook.net/D61695
-
- 06 8月, 2016 2 次提交
-
-
由 omegaga 提交于
Summary: Implement a time series database that supports DateTieredCompactionStrategy. It wraps a db object and separate SST files in different column families (time windows). Test Plan: Add `date_tiered_test`. Reviewers: dhruba, sdong Reviewed By: sdong Subscribers: andrewkr, dhruba, leveldb Differential Revision: https://reviews.facebook.net/D61653
-
由 sdong 提交于
Summary: Add a utility function that trigger necessary full compaction and put output to the correct level by looking at new options and old options. Test Plan: Add unit tests for it. Reviewers: andrewkr, igor, IslamAbdelRahman Reviewed By: IslamAbdelRahman Subscribers: muthu, sumeet, leveldb, andrewkr, dhruba Differential Revision: https://reviews.facebook.net/D60783
-
- 02 8月, 2016 2 次提交
-
-
由 Islam AbdelRahman 提交于
Summary: parallel tests are broken because gnu_parallel is reading deprecated options from `/etc/parallel/config` Fix this by passing `--plain` to ignore `/etc/parallel/config` Test Plan: make check -j64 Reviewers: kradhakrishnan, sdong, andrewkr, yiwu, arahut Reviewed By: arahut Subscribers: andrewkr, dhruba Differential Revision: https://reviews.facebook.net/D61359
-
由 omegaga 提交于
Summary: Experiments on column-aware encodings. Supported features: 1) extract data blocks from SST file and encode with specified encodings; 2) Decode encoded data back into row format; 3) Directly extract data blocks and write in row format (without prefix encoding); 4) Get column distribution statistics for column format; 5) Dump data blocks separated by columns in human-readable format. There is still on-going work on this diff. More refactoring is necessary. Test Plan: Wrote tests in `column_aware_encoding_test.cc`. More tests should be added. Reviewers: sdong Reviewed By: sdong Subscribers: arahut, andrewkr, dhruba Differential Revision: https://reviews.facebook.net/D60027
-
- 30 7月, 2016 1 次提交
-
-
由 Adam Retter 提交于
More granular steps in the Makefile, can help with running all or single Java tests (and with ASAN build - https://github.com/facebook/rocksdb/wiki/JNI-Debugging) (#1237)
-
- 28 7月, 2016 1 次提交
-
-
由 Anirban Rahut 提交于
Summary: Removing moreutils from sandcastle and adding gnu parallel. Then passing in J= nproc command Test Plan: Testing on sandcastle Reviewers: sdong, kradhakrishnan Reviewed By: kradhakrishnan Subscribers: andrewkr, dhruba Differential Revision: https://reviews.facebook.net/D61017
-
- 26 7月, 2016 1 次提交
-
-
由 Islam AbdelRahman 提交于
Summary: TickersNameMap is not consistent with Tickers enum. this cause us to report wrong statistics and sometimes to access TickersNameMap outside it's boundary causing crashes (in Fb303 statistics) Test Plan: added new unit test Reviewers: sdong, kradhakrishnan Reviewed By: kradhakrishnan Subscribers: andrewkr, dhruba Differential Revision: https://reviews.facebook.net/D61083
-
- 23 7月, 2016 2 次提交
-
-
由 sdong 提交于
Summary: Tsan crash white-box test was disabled because it never ends. Re-enable it with reduced killing odds. Add a parameter in crash test script to allow we pass it through an environment variable. Test Plan: Run it manually. Reviewers: IslamAbdelRahman Reviewed By: IslamAbdelRahman Subscribers: leveldb, andrewkr, dhruba Differential Revision: https://reviews.facebook.net/D61053
-
- 22 7月, 2016 4 次提交
-
-
由 sdong 提交于
Summary: Multiput atomiciy is broken across multiple column families if we don't sync WAL before flushing one column family. The WAL file may contain a write batch containing writes to a key to the CF to be flushed and a key to other CF. If we don't sync WAL before flushing, if machine crashes after flushing, the write batch will only be partial recovered. Data to other CFs are lost. Test Plan: Add a new unit test which will fail without the diff. Reviewers: yhchiang, IslamAbdelRahman, igor, yiwu Reviewed By: yiwu Subscribers: yiwu, leveldb, andrewkr, dhruba Differential Revision: https://reviews.facebook.net/D60915
-
由 sdong 提交于
Summary: RocksDB behavior is slightly different between data on tmpfs and normal file systems. Add a test case to run RocksDB on normal file system. Test Plan: See the tests launched by Phabricator Reviewers: kradhakrishnan, IslamAbdelRahman, gunnarku Reviewed By: gunnarku Subscribers: leveldb, andrewkr, dhruba Differential Revision: https://reviews.facebook.net/D60963
-
由 ryneli 提交于
EnvLibrados is a customized RocksDB Env to use RADOS as the backend file system of RocksDB. It overrides all file system related API of default Env. The easiest way to use it is just like following: std::string db_name = "test_db"; std::string config_path = "path/to/ceph/config"; DB* db; Options options; options.env = EnvLibrados(db_name, config_path); Status s = DB::Open(options, kDBPath, &db); Then EnvLibrados will forward all file read/write operation to the RADOS cluster assigned by config_path. Default pool is db_name+"_pool". There are some options that users could set for EnvLibrados. - write_buffer_size. This variable is the max buffer size for WritableFile. After reaching the buffer_max_size, EnvLibrados will sync buffer content to RADOS, then clear buffer. - db_pool. Rather than using default pool, users could set their own db pool name - wal_dir. The dir for WAL files. Because RocksDB only has 2-level structure (dir_name/file_name), the format of wal_dir is "/dir_name"(CAN'T be "/dir1/dir2"). Default wal_dir is "/wal". - wal_pool. Corresponding pool name for WAL files. Default value is db_name+"_wal_pool" The example of setting options looks like following: db_name = "test_db"; db_pool = db_name+"_pool"; wal_dir = "/wal"; wal_pool = db_name+"_wal_pool"; write_buffer_size = 1 << 20; env_ = new EnvLibrados(db_name, config, db_pool, wal_dir, wal_pool, write_buffer_size); DB* db; Options options; options.env = env_; // The last level dir name should match the dir name in prefix_pool_map options.wal_dir = "/tmp/wal"; // open DB Status s = DB::Open(options, kDBPath, &db); Librados is required to compile EnvLibrados. Then use "$make LIBRADOS=1" to compile RocksDB. If you want to only compile EnvLibrados test, just run "$ make env_librados_test LIBRADOS=1". To run env_librados_test, you need to have a running RADOS cluster with the configure file located in "../ceph/src/ceph.conf" related to "rocksdb/".
-
由 Yi Wu 提交于
Summary: Fix flush not being commit while writing manifest, which is a recent bug introduced by D60075. The issue: # Options.max_background_flushes > 1 # Background thread A pick up a flush job, flush, then commit to manifest. (Note that mutex is released before writing manifest.) # Background thread B pick up another flush job, flush. When it gets to `MemTableList::InstallMemtableFlushResults`, it notices another thread is commiting, so it quit. # After the first commit, thread A doesn't double check if there are more flush result need to commit, leaving the second flush uncommitted. Test Plan: run the test. Also verify the new test hit deadlock without the fix. Reviewers: sdong, igor, lightmark Reviewed By: lightmark Subscribers: andrewkr, omegaga, dhruba, leveldb Differential Revision: https://reviews.facebook.net/D60969
-
- 21 7月, 2016 2 次提交
-
-
由 Volker Mische 提交于
The tests run by `make check` require Bash. On Debian you'd need to run the test as `make SHELL=/bin/bash check`. This commit makes it work on all POSIX compatible shells (tested on Debian with `dash`).
-
由 sdong 提交于
Summary: As Title. Test Plan: See how the diff works. Reviewers: kradhakrishnan, andrewkr, gunnarku Reviewed By: gunnarku Subscribers: leveldb, andrewkr, dhruba Differential Revision: https://reviews.facebook.net/D60933
-
- 13 7月, 2016 1 次提交
-
-
由 Yi Wu 提交于
Summary: When write stalls because of auto compaction is disabled, or stop write trigger is reached, user may change these two options to unblock writes. Unfortunately we had issue where the write thread will block the attempt to persist the options, thus creating a deadlock. This diff fix the issue and add two test cases to detect such deadlock. Test Plan: Run unit tests. Also, revert db_impl.cc to master (but don't revert `DBImpl::BackgroundCompaction:Finish` sync point) and run db_options_test. Both tests should hit deadlock. Reviewers: sdong Reviewed By: sdong Subscribers: andrewkr, dhruba, leveldb Differential Revision: https://reviews.facebook.net/D60627
-
- 12 7月, 2016 1 次提交
-
-
由 omegaga 提交于
Summary: In D33849 we updated Makefile to generate .d files for all .cc sources. Since we have more types of source files now, this needs to be updated so that this mechanism can work for new files. Test Plan: change a dependent .h file, re-make and see if .o file is recompiled. Reviewers: sdong, andrewkr Reviewed By: andrewkr Subscribers: andrewkr, dhruba, leveldb Differential Revision: https://reviews.facebook.net/D60591
-
- 22 6月, 2016 1 次提交
-
-
由 Islam AbdelRahman 提交于
Summary: Update Makefile to show warnings when we have invalid paths in our make_config.mk file sample output ``` $ make static_lib -j64 Makefile:150: Warning: /mnt/gvfs/third-party2/libgcc/53e0eac8911888a105aa98b9a35fe61cf1d8b278/4.9.x/gcc-4.9-glibc-2.20/024dbc3/libs dont exist Makefile:150: Warning: /mnt/gvfs/third-party2/llvm-fb/b91de48a4974ec839946d824402b098d43454cef/stable/centos6-native/7aaccbe/../../src/clang/tools/scan-build/scan-build dont exist GEN util/build_version.cc ``` Test Plan: check that warning is printed visually Reviewers: sdong, yiwu, andrewkr Reviewed By: andrewkr Subscribers: andrewkr, dhruba Differential Revision: https://reviews.facebook.net/D59523
-
- 08 6月, 2016 1 次提交
-
-
由 krad 提交于
Summary: This provides provides an implementation of PersistentCacheTier that is specialized for RAM. This tier does not persist data though. Why do we need this tier ? This is ideal as tier 0. This tier can host data that is too hot. Why can't we use Cache variants ? Yes you can use them instead. This tier can potentially outperform BlockCache in RAW mode by virtue of compression and compressed cache in block cache doesn't seem very popular. Potentially this tier can be modified to under stand the disadvantage of the tier below and retain data that the tier below is bad at handling (for example index and bloom data that is huge in size) Test Plan: Run unit tests added Subscribers: andrewkr, dhruba, leveldb Differential Revision: https://reviews.facebook.net/D57069
-
- 04 6月, 2016 1 次提交
-
-
由 Andrew Kryczka 提交于
Summary: - Provide env_test as a static library. We will build it for future releases so internal Envs can use env_test by linking against this library. - Add tests for CustomEnv, which is configurable via ENV_TEST_URI environment variable. It uses the URI-based Env lookup (depends on D58449). - Refactor env_basic_test cases to use a unique/configurable directory for test files. Test Plan: built a test binary against librocksdb_env_test.a. It registered the default Env with URI prefix "a://". - verify runs all CustomEnv tests when URI with correct prefix is provided ``` $ ENV_TEST_URI="a://ok" ./tmp --gtest_filter="CustomEnv/*" ... [ PASSED ] 12 tests. ``` - verify runs no CustomEnv tests when URI with non-matching prefix is provided ``` $ ENV_TEST_URI="b://ok" ./tmp --gtest_filter="CustomEnv/*" ... [ PASSED ] 0 tests. ``` Reviewers: ldemailly, IslamAbdelRahman, lightmark, sdong Reviewed By: sdong Subscribers: andrewkr, dhruba, leveldb Differential Revision: https://reviews.facebook.net/D58485
-