- 18 7月, 2015 5 次提交
-
-
由 Igor Canadi 提交于
Summary: When we first started, max_background_flushes was 0 by default and compaction thread was executing flushes (since there was no flush thread). Then, we switched the default max_background_flushes to 1. However, we still support the case where there is no flush thread and flushes are done in compaction. This is making our code a bit more complicated. By not supporting this use-case we can make our code simpler. We have a special case that when you set max_background_flushes to 0, we schedule the flush to execute on the compaction thread. Test Plan: make check (there might be some unit tests that depend on this behavior) Reviewers: IslamAbdelRahman, yhchiang, sdong Reviewed By: sdong Subscribers: dhruba, leveldb Differential Revision: https://reviews.facebook.net/D41931
-
由 agiardullo 提交于
Summary: ROCKSDB_WARNING is only defined if either ROCKSDB_PLATFORM_POSIX or OS_WIN is defined. This works well for building rocksdb with its own build scripts. But this won't work when an outside project(like mongodb) doesn't define ROCKSDB_PLATFORM_POSIX. This fix defines ROCKSDB_WARNING for all platforms. No idea if its defined correctly on non-posix,non-windows platforms but this is no worse that the current situation where this macro is missing on unexpected platforms. This fix should hopefully fix anyone whose build broke now that we've switched from using #warning to Pragma (to support windows). Unfortunately, while mongo-rocks compiles, it ignores the Pragma and doesn't print a warning. I have not been able to figure out a way to implement this portably on all platforms. Of course, an alternate solution would be to just get rid of ROCKSDB_WARNING and live with include file redirects indefinitely. Thoughts? Test Plan: build rocks, build mongorocks Reviewers: igor, kradhakrishnan, IslamAbdelRahman, sdong Reviewed By: sdong Subscribers: dhruba, leveldb Differential Revision: https://reviews.facebook.net/D42477
-
由 Ari Ekmekji 提交于
Summary: Added a new flag --json to the ldb manifest_dump command that prints out the version edits as JSON objects for easier reading and parsing of information. Test Plan: **Sample usage: ** ``` ./ldb manifest_dump --json --path=path/to/manifest/file ``` **Sample output:** ``` {"EditNumber": 0, "Comparator": "leveldb.BytewiseComparator", "ColumnFamily": 0} {"EditNumber": 1, "LogNumber": 0, "ColumnFamily": 0} {"EditNumber": 2, "LogNumber": 4, "PrevLogNumber": 0, "NextFileNumber": 7, "LastSeq": 35356, "AddedFiles": [{"Level": 0, "FileNumber": 5, "FileSize": 1949284, "SmallestIKey": "'", "LargestIKey": "'"}], "ColumnFamily": 0} ... {"EditNumber": 13, "PrevLogNumber": 0, "NextFileNumber": 36, "LastSeq": 290994, "DeletedFiles": [{"Level": 0, "FileNumber": 17}, {"Level": 0, "FileNumber": 20}, {"Level": 0, "FileNumber": 22}, {"Level": 0, "FileNumber": 24}, {"Level": 1, "FileNumber": 13}, {"Level": 1, "FileNumber": 14}, {"Level": 1, "FileNumber": 15}, {"Level": 1, "FileNumber": 18}], "AddedFiles": [{"Level": 1, "FileNumber": 25, "FileSize": 2114340, "SmallestIKey": "'", "LargestIKey": "'"}, {"Level": 1, "FileNumber": 26, "FileSize": 2115213, "SmallestIKey": "'", "LargestIKey": "'"}, {"Level": 1, "FileNumber": 27, "FileSize": 2114807, "SmallestIKey": "'", "LargestIKey": "'"}, {"Level": 1, "FileNumber": 30, "FileSize": 2115271, "SmallestIKey": "'", "LargestIKey": "'"}, {"Level": 1, "FileNumber": 31, "FileSize": 2115165, "SmallestIKey": "'", "LargestIKey": "'"}, {"Level": 1, "FileNumber": 32, "FileSize": 2114683, "SmallestIKey": "'", "LargestIKey": "'"}, {"Level": 1, "FileNumber": 35, "FileSize": 1757512, "SmallestIKey": "'", "LargestIKey": "'"}], "ColumnFamily": 0} ... ``` Reviewers: sdong, anthony, yhchiang, igor Reviewed By: igor Subscribers: dhruba Differential Revision: https://reviews.facebook.net/D41727
-
由 Igor Canadi 提交于
Summary: It has been around for a while and it looks like it never found any uses in the wild. It's also complicating our compaction_job code quite a bit. We're deprecating it in 3.13, but will put it back in 3.14 if we actually find users that need this feature. Test Plan: make check Reviewers: noetzli, yhchiang, sdong Reviewed By: sdong Subscribers: dhruba, leveldb Differential Revision: https://reviews.facebook.net/D42405
-
由 Andres Notzli 提交于
Summary: MergeUntil was not reporting a success when merging an operand with a Value/Deletion despite the comments in MergeHelper and CompactionJob indicating otherwise. This lead to operands being written to the compaction output unnecessarily: M1 M2 M3 P M4 M5 --> (P+M1+M2+M3) M2 M3 M4 M5 (before the diff) M1 M2 M3 P M4 M5 --> (P+M1+M2+M3) M4 M5 (after the diff) In addition, the code handling Values/Deletion was basically identical. This patch unifies the code. Finally, this patch also adds testing for merge_helper. Test Plan: make && make check Reviewers: sdong, rven, yhchiang, tnovak, igor Reviewed By: igor Subscribers: tnovak, dhruba, leveldb Differential Revision: https://reviews.facebook.net/D42351
-
- 17 7月, 2015 7 次提交
-
-
由 Siying Dong 提交于
fix append bug in DumpDBFileSummary()
-
由 Siying Dong 提交于
Ensure Windows build w/o port/port.h in public headers
-
由 sdong 提交于
Summary: Typo seqeuntial -> sequential in db/fault_injection_test.cc Test Plan: Build it Reviewers: ott, igor, anthony, kradhakrishnan, IslamAbdelRahman Reviewed By: IslamAbdelRahman Subscribers: leveldb, dhruba Differential Revision: https://reviews.facebook.net/D42417
-
由 Dmitri Smirnov 提交于
-
由 Dmitri Smirnov 提交于
- Remove make file defines from public headers and use _WIN32 because it is compiler defined - use __GNUC__ and __clang__ to guard non-portable attributes - add #include "port/port.h" to some new .cc files. - minor changes in CMakeLists to reflect recent changes
-
由 sdong 提交于
Summary: Previous run may leave some SST files with higher file numbers than manifest indicates. Compaction or flush may start to run while DB::Open() is still going on. SST file garbage collection may happen interleaving with compaction or flush, and overwrite files generated by compaction of flushes after they are generated. This might cause data loss. This possibility of interleaving is recently introduced. Fix it by not allowing compaction or flush to be scheduled before DB::Open() finishes. Test Plan: Add a unit test. This verification will have a chance to fail without the fix but doesn't fix without the fix. Reviewers: kradhakrishnan, anthony, yhchiang, IslamAbdelRahman, igor Reviewed By: igor Subscribers: dhruba, leveldb Differential Revision: https://reviews.facebook.net/D42399
-
由 Andres Notzli 提交于
Summary: Fixes T7697334. Adds a simple test to check whether CompactionJob deals with corrupted keys correctly. Right now, we preserve corrupted keys. Note: depending on the type of corruption and options like comparators, CompactionJob fails. This test just checks whether corrupted keys that do not fail CompactionJob are preserved. Test Plan: `make compaction_job_test && ./compaction_job_test` -> Tests pass. Add `input->Next(); continue;` in CompactionJob::ProcessKeyValueCompaction() inside then-branch of `!ParseInternalKey(key, &ikey)` -> Tests fail. Reviewers: sdong, rven, yhchiang, igor Reviewed By: igor Subscribers: dhruba, leveldb Differential Revision: https://reviews.facebook.net/D42237
-
- 16 7月, 2015 12 次提交
-
-
由 Igor Canadi 提交于
Summary: as title Test Plan: compiles Reviewers: lovro Subscribers: dhruba, leveldb Differential Revision: https://reviews.facebook.net/D42411
-
由 qinzuoyan 提交于
-
由 Dmitri Smirnov 提交于
- Remove make file defines from public headers and use _WIN32 because it is compiler defined - use __GNUC__ and __clang__ to guard non-portable attributes - add #include "port/port.h" to some new .cc files. - minor changes in CMakeLists to reflect recent changes
-
由 qinzuoyan 提交于
-
由 agiardullo 提交于
Summary: quick fix for now. will figure out a better fix soon Test Plan: build Reviewers: sdong, igor, kradhakrishnan Subscribers: dhruba, leveldb Differential Revision: https://reviews.facebook.net/D42381
-
由 agiardullo 提交于
Summary: Unbreak build for mongo. later, we'll have to figure out if there is a platform independent way to implement ROCKSDB_WARNING Test Plan: build Reviewers: igor, sdong Subscribers: dhruba, leveldb Differential Revision: https://reviews.facebook.net/D42369
-
由 Igor Canadi 提交于
Summary: master is now 3.13-to-be. I'm testing some internal stuff that has #ifdef based on RocksDB version. Test Plan: none Reviewers: sdong, anthony Reviewed By: anthony Subscribers: dhruba, leveldb Differential Revision: https://reviews.facebook.net/D42249
-
由 agiardullo 提交于
Summary: Moved convenience.h out of utilities to remove a dependency on utilities in db. Test Plan: unit tests. Also compiled a link to the old location to verify the _Pragma works. Reviewers: sdong, yhchiang, igor Reviewed By: igor Subscribers: dhruba, leveldb Differential Revision: https://reviews.facebook.net/D42201
-
Summary: Trvial move in universal compaction was failing when trying to move files from levels other than 0. This was because the DeleteFile while trivially moving, was only deleting files of level 0 which caused duplication of same file in different levels. This is fixed by passing the right level as argument in the call of DeleteFile while doing trivial move. Test Plan: ./db_test ran successfully with the new test cases. Reviewers: sdong Reviewed By: sdong Subscribers: leveldb, dhruba Differential Revision: https://reviews.facebook.net/D42135
-
由 krad 提交于
Summary: gcc-4.9-glibc-2.20 complains about uninitialized variable. db/compaction_picker.cc: In member function 'bool rocksdb::CompactionPicker::IsInputNonOverlapping(rocksdb::Compaction*)': db/compaction_picker.cc:1174:17: error: 'prev.rocksdb::{anonymous}::InputFileInfo::f' may be used uninitialized in this function [-Werror=maybe-uninitialized] InputFileInfo prev, curr, next; Test Plan: pmake on local environment Reviewers: sdong igor CC: leveldb@ Task ID: # Blame Rev:
-
由 Aaron Feldman 提交于
Summary: Remove --help entry for readhot. Update read_random_exp_range flag description: The distribution is num * exp(-r), not num * exp(r). Test Plan: Run ./db_bench --help Reviewers: sdong, igor Reviewed By: igor Subscribers: dhruba Differential Revision: https://reviews.facebook.net/D42303
-
由 Andres Notzli 提交于
Summary: Before, writing key/value pairs out to files was done inside ProcessKeyValueCompaction(). To make ProcessKeyValueCompaction() more understandable, this patch moves the writing part to a separate function. This is intended to be a stepping stone for additional changes. Test Plan: make && make check Reviewers: sdong, rven, yhchiang, igor Reviewed By: igor Subscribers: dhruba, leveldb Differential Revision: https://reviews.facebook.net/D42243
-
- 15 7月, 2015 16 次提交
-
-
由 lovro 提交于
Summary: Repeat of b6655a67 (reverted in b7a2369f) with a proper fix for the issue that 57d216ea was trying to fix. Test Plan: make check for i in $(seq 100); do ./db_stress --test_batches_snapshots=1 --threads=32 --write_buffer_size=4194304 --destroy_db_initially=0 --reopen=20 --readpercent=45 --prefixpercent=5 --writepercent=35 --delpercent=5 --iterpercent=10 --db=/tmp/rocksdb_crashtest_KdCI5F --max_key=100000000 --mmap_read=0 --block_size=16384 --cache_size=1048576 --open_files=500000 --verify_checksum=1 --sync=0 --progress_reports=0 --disable_wal=0 --disable_data_sync=1 --target_file_size_base=2097152 --target_file_size_multiplier=2 --max_write_buffer_number=3 --max_background_compactions=20 --max_bytes_for_level_base=10485760 --filter_deletes=0 --memtablerep=prefix_hash --prefix_size=7 --ops_per_thread=200 || break; done Reviewers: anthony, sdong, igor, yhchiang Reviewed By: igor, yhchiang Subscribers: dhruba Differential Revision: https://reviews.facebook.net/D41391
-
由 Igor Canadi 提交于
Summary: I didn't know this can work :) Test Plan: none Reviewers: sdong, yhchiang Reviewed By: yhchiang Subscribers: dhruba, leveldb Differential Revision: https://reviews.facebook.net/D42081
-
由 Igor Canadi 提交于
Summary: This is failing our tsan tests. Our new behavior is to fail DB::Open() if the requested compression is not available. The easiest fix is to make ldb_test not depend on compression. Test Plan: python tools/ldb_test.py Reviewers: sdong, yhchiang, anthony Reviewed By: anthony Subscribers: dhruba, leveldb Differential Revision: https://reviews.facebook.net/D42075
-
由 Andres Notzli 提交于
Summary: Fixes two minor issues in CompactionJob. CompactionJob::Run() dereferences log_buffer_ without a check, so this patch adds an assert in the constructor where log_buffer_ is assigned. compaction_job_stats_ can be null but ProcessKeyValueCompaction was dereferencing it without a check. Test Plan: make && make check Reviewers: sdong, rven, yhchiang, igor Reviewed By: igor Subscribers: dhruba, leveldb Differential Revision: https://reviews.facebook.net/D42231
-
由 Igor Canadi 提交于
Summary: For some reason, parallel tests are failing in Jenkins. We turned off parallel tests for all tests except `make coverage`. Let's turn it off here, too, so that we can have Jenkins test working again. For the error log, see: https://ci-builds.fb.com/job/rocksdb_coverage/166/console Test Plan: commit and wait for jenkins Reviewers: meyering, sdong, anthony Reviewed By: anthony Subscribers: dhruba, leveldb Differential Revision: https://reviews.facebook.net/D42057
-
由 Yueh-Hsuan Chiang 提交于
Summary: Move UniversalCompaction related db-tests to db_universal_compaction_test.cc Test Plan: db_test db_universal_compaction_test Reviewers: igor, sdong, IslamAbdelRahman, anthony Reviewed By: anthony Subscribers: dhruba, leveldb Differential Revision: https://reviews.facebook.net/D42225
-
由 Yueh-Hsuan Chiang 提交于
Summary: Move TailingIterator tests from db_test.cc to db_test_tailing_iterator.cc Test Plan: db_test db_test_tailing_iterator Reviewers: igor, anthony, IslamAbdelRahman, sdong Reviewed By: sdong Subscribers: dhruba Differential Revision: https://reviews.facebook.net/D42021
-
由 Yueh-Hsuan Chiang 提交于
Summary: Make TransactionLogIterator related tests from db_test.cc to db_log_iter_test.cc Test Plan: db_test db_log_iter_test Reviewers: sdong, IslamAbdelRahman, igor, anthony Reviewed By: igor Subscribers: dhruba, leveldb Differential Revision: https://reviews.facebook.net/D42045
-
由 Yueh-Hsuan Chiang 提交于
Summary: Move CompactionFilter tests in db_test.cc to db_compaction_filter_test.cc Test Plan: db_test db_compaction_filter_test Reviewers: igor, sdong, IslamAbdelRahman, anthony Reviewed By: anthony Subscribers: dhruba, leveldb Differential Revision: https://reviews.facebook.net/D42207
-
由 Yueh-Hsuan Chiang 提交于
Summary: Block SyncPoint in util/db_test_util.h in released Windows mode. Test Plan: db_test Reviewers: igor, anthony, sdong, IslamAbdelRahman Reviewed By: sdong, IslamAbdelRahman Subscribers: dhruba, leveldb Differential Revision: https://reviews.facebook.net/D42213
-
由 Jörg Maier 提交于
Add JNI interface to set max_table_files_size for FIFO compaction
-
由 Jörg Maier 提交于
Looks good to me.
-
由 Igor Canadi 提交于
Summary: Couple of changes here: * NewBackupEngine() and NewReadOnlyBackupEngine() are now removed. They were deprecated since RocksDB 3.8. Changing these to new functions should be pretty straight-forward. As a followup, I'll fix all fbcode callsights * Instead of initializing backup engine in the constructor, we initialize it in a separate function now. That way, we can catch all errors and return appropriate status code. * We catch all errors during initializations and return them to the client properly. * Added new tests to backupable_db_test, to make sure that we can't open BackupEngine when there are Env errors. * Transitioned backupable_db_test to use BackupEngine rather than BackupableDB. From the two available APIs, judging by the current use-cases, it looks like BackupEngine API won. It's much more flexible since it doesn't require StackableDB. Test Plan: Added a new unit test to backupable_db_test Reviewers: yhchiang, sdong, AaronFeldman Reviewed By: AaronFeldman Subscribers: dhruba, leveldb Differential Revision: https://reviews.facebook.net/D41925
-
由 agiardullo 提交于
Summary: Seems like the cleanest way to resolve this is to move CompactedDBImpl into db/. CompactedDBImpl should probably live in the same place as DBImplReadonly since Opening the latter could end up instantiating the former. Both DBImplReadonly and CompactedDBImpl inherit from DBImpl access protected members of DBImpl( and the latter access friendly private methods). Test Plan: unit tests Reviewers: sdong, yhchiang, kradhakrishnan, igor Reviewed By: igor Subscribers: dhruba, leveldb Differential Revision: https://reviews.facebook.net/D42027
-
由 Adam Retter 提交于
-
由 Islam AbdelRahman 提交于
Summary: Remove ColumnFamiliesBatchWithIndexTest from ROCKSDB_LITE since WriteBatchWithIndex is not supported Test Plan: write_batch_test Reviewers: yhchiang, sdong, igor Reviewed By: igor Subscribers: dhruba Differential Revision: https://reviews.facebook.net/D42033
-