Skip to content

R
rocksdb

kvdb / rocksdb
12 个月 前同步成功

通知 0
Star 0
Fork 0
R
rocksdb

体验新版 GitCode，发现更多精彩内容 >>

提交 7242dae7 编写于 作者: A Adam Retter 提交者： Facebook Github Bot
浏览文件
操作

Improve RocksJava Comparator (#6252) 

Summary:
This is a redesign of the API for RocksJava comparators with the aim of improving performance. It also simplifies the class hierarchy.

**NOTE**: This breaks backwards compatibility for existing 3rd party Comparators implemented in Java... so we need to consider carefully which release branches this goes into.

Previously when implementing a comparator in Java the developer had a choice of subclassing either `DirectComparator` or `Comparator` which would use direct and non-direct byte-buffers resepectively (via `DirectSlice` and `Slice`).

In this redesign there we have eliminated the overhead of using the Java Slice classes, and just use `ByteBuffer`s. The `ComparatorOptions` supplied when constructing a Comparator allow you to choose between direct and non-direct byte buffers by setting `useDirect`.

In addition, the `ComparatorOptions` now allow you to choose whether a ByteBuffer is reused over multiple comparator calls, by setting `maxReusedBufferSize > 0`. When buffers are reused, ComparatorOptions provides a choice of mutex type by setting `useAdaptiveMutex`.

 ---
[JMH benchmarks previously indicated](https://github.com/facebook/rocksdb/pull/6241#issue-356398306) that the difference between C++ and Java for implementing a comparator was ~7x slowdown in Java.

With these changes, when reusing buffers and guarding access to them via mutexes the slowdown is approximately the same. However, these changes offer a new facility to not reuse mutextes, which reduces the slowdown to ~5.5x in Java. We also offer a `thread_local` mechanism for reusing buffers, which reduces slowdown to ~5.2x in Java (closes https://github.com/facebook/rocksdb/pull/4425).

These changes also form a good base for further optimisation work such as further JNI lookup caching, and JNI critical.

 ---
These numbers were captured without jemalloc. With jemalloc, the performance improves for all tests, and the Java slowdown reduces to between 4.8x and 5.x.

```
ComparatorBenchmarks.put                                                native_bytewise  thrpt   25  124483.795 ± 2032.443  ops/s
ComparatorBenchmarks.put                                        native_reverse_bytewise  thrpt   25  114414.536 ± 3486.156  ops/s
ComparatorBenchmarks.put              java_bytewise_non-direct_reused-64_adaptive-mutex  thrpt   25   17228.250 ± 1288.546  ops/s
ComparatorBenchmarks.put          java_bytewise_non-direct_reused-64_non-adaptive-mutex  thrpt   25   16035.865 ± 1248.099  ops/s
ComparatorBenchmarks.put                java_bytewise_non-direct_reused-64_thread-local  thrpt   25   21571.500 ±  871.521  ops/s
ComparatorBenchmarks.put                  java_bytewise_direct_reused-64_adaptive-mutex  thrpt   25   23613.773 ± 8465.660  ops/s
ComparatorBenchmarks.put              java_bytewise_direct_reused-64_non-adaptive-mutex  thrpt   25   16768.172 ± 5618.489  ops/s
ComparatorBenchmarks.put                    java_bytewise_direct_reused-64_thread-local  thrpt   25   23921.164 ± 8734.742  ops/s
ComparatorBenchmarks.put                              java_bytewise_non-direct_no-reuse  thrpt   25   17899.684 ±  839.679  ops/s
ComparatorBenchmarks.put                                  java_bytewise_direct_no-reuse  thrpt   25   22148.316 ± 1215.527  ops/s
ComparatorBenchmarks.put      java_reverse_bytewise_non-direct_reused-64_adaptive-mutex  thrpt   25   11311.126 ±  820.602  ops/s
ComparatorBenchmarks.put  java_reverse_bytewise_non-direct_reused-64_non-adaptive-mutex  thrpt   25   11421.311 ±  807.210  ops/s
ComparatorBenchmarks.put        java_reverse_bytewise_non-direct_reused-64_thread-local  thrpt   25   11554.005 ±  960.556  ops/s
ComparatorBenchmarks.put          java_reverse_bytewise_direct_reused-64_adaptive-mutex  thrpt   25   22960.523 ± 1673.421  ops/s
ComparatorBenchmarks.put      java_reverse_bytewise_direct_reused-64_non-adaptive-mutex  thrpt   25   18293.317 ± 1434.601  ops/s
ComparatorBenchmarks.put            java_reverse_bytewise_direct_reused-64_thread-local  thrpt   25   24479.361 ± 2157.306  ops/s
ComparatorBenchmarks.put                      java_reverse_bytewise_non-direct_no-reuse  thrpt   25    7942.286 ±  626.170  ops/s
ComparatorBenchmarks.put                          java_reverse_bytewise_direct_no-reuse  thrpt   25   11781.955 ± 1019.843  ops/s
```
Pull Request resolved: https://github.com/facebook/rocksdb/pull/6252

Differential Revision: D19331064

Pulled By: pdillinger

fbshipit-source-id: 1f3b794e6a14162b2c3ffb943e8c0e64a0c03738
上级 800d24dd
展开全部 隐藏空白更改
内联 并排
Showing with 2673 addition and 1098 deletion
HISTORY.md
浏览文件 @ 7242dae7
# Rocksdb Change Log
## Unreleased
### Public API Change
* Major breaking changes to Java comparators, toward standardizing on ByteBuffer for performant, locale-neutral operations on keys (#6252).
### Bug Fixes
* Fix incorrect results while block-based table uses kHashSearch, together with Prev()/SeekForPrev().
* Fix a bug that prevents opening a DB after two consecutive crash with TransactionDB, where the first crash recovers from a corrupted WAL with kPointInTimeRecovery but the second cannot.
......
java/CMakeLists.txt
浏览文件 @ 7242dae7
......@@ -122,7 +122,6 @@ set(JAVA_MAIN_CLASSES
src/main/java/org/rocksdb/CompactRangeOptions.java
src/main/java/org/rocksdb/CompactionStopStyle.java
src/main/java/org/rocksdb/CompactionStyle.java
src/main/java/org/rocksdb/Comparator.java
src/main/java/org/rocksdb/ComparatorOptions.java
src/main/java/org/rocksdb/ComparatorType.java
src/main/java/org/rocksdb/CompressionOptions.java
......@@ -131,7 +130,6 @@ set(JAVA_MAIN_CLASSES
src/main/java/org/rocksdb/DBOptionsInterface.java
src/main/java/org/rocksdb/DBOptions.java
src/main/java/org/rocksdb/DbPath.java
src/main/java/org/rocksdb/DirectComparator.java
src/main/java/org/rocksdb/DirectSlice.java
src/main/java/org/rocksdb/EncodingType.java
src/main/java/org/rocksdb/Env.java
......@@ -181,6 +179,7 @@ set(JAVA_MAIN_CLASSES
src/main/java/org/rocksdb/ReadTier.java
src/main/java/org/rocksdb/RemoveEmptyValueCompactionFilter.java
src/main/java/org/rocksdb/RestoreOptions.java
src/main/java/org/rocksdb/ReusedSynchronisationType.java
src/main/java/org/rocksdb/RocksCallbackObject.java
src/main/java/org/rocksdb/RocksDBException.java
src/main/java/org/rocksdb/RocksDB.java
......@@ -236,9 +235,10 @@ set(JAVA_MAIN_CLASSES
src/main/java/org/rocksdb/WriteBatchWithIndex.java
src/main/java/org/rocksdb/WriteOptions.java
src/main/java/org/rocksdb/WriteBufferManager.java
src/main/java/org/rocksdb/util/ByteUtil.java
src/main/java/org/rocksdb/util/BytewiseComparator.java
src/main/java/org/rocksdb/util/DirectBytewiseComparator.java
src/main/java/org/rocksdb/util/Environment.java
src/main/java/org/rocksdb/util/IntComparator.java
src/main/java/org/rocksdb/util/ReverseBytewiseComparator.java
src/main/java/org/rocksdb/util/SizeUnit.java
src/main/java/org/rocksdb/UInt64AddOperator.java
......@@ -400,11 +400,9 @@ if(${CMAKE_VERSION} VERSION_LESS "3.11.4" OR (${Java_VERSION_MINOR} STREQUAL "7"
org.rocksdb.CompactionOptionsFIFO
org.rocksdb.CompactionOptionsUniversal
org.rocksdb.CompactRangeOptions
org.rocksdb.Comparator
org.rocksdb.ComparatorOptions
org.rocksdb.CompressionOptions
org.rocksdb.DBOptions
org.rocksdb.DirectComparator
org.rocksdb.DirectSlice
org.rocksdb.Env
org.rocksdb.EnvOptions
......
java/Makefile
浏览文件 @ 7242dae7
NATIVE_JAVA_CLASSES = org.rocksdb.AbstractCompactionFilter\
NATIVE_JAVA_CLASSES = \
org.rocksdb.AbstractCompactionFilter\
org.rocksdb.AbstractCompactionFilterFactory\
org.rocksdb.AbstractComparator\
org.rocksdb.AbstractSlice\
org.rocksdb.AbstractTableFilter\
org.rocksdb.AbstractTraceWriter\
......@@ -21,11 +23,9 @@ NATIVE_JAVA_CLASSES = org.rocksdb.AbstractCompactionFilter\
org.rocksdb.CompactionOptionsFIFO\
org.rocksdb.CompactionOptionsUniversal\
org.rocksdb.CompactRangeOptions\
org.rocksdb.Comparator\
org.rocksdb.ComparatorOptions\
org.rocksdb.CompressionOptions\
org.rocksdb.DBOptions\
org.rocksdb.DirectComparator\
org.rocksdb.DirectSlice\
org.rocksdb.Env\
org.rocksdb.EnvOptions\
......@@ -98,10 +98,13 @@ ifeq ($(PLATFORM), OS_MACOSX)
ROCKSDB_JAR = rocksdbjni-$(ROCKSDB_MAJOR).$(ROCKSDB_MINOR).$(ROCKSDB_PATCH)-osx.jar
endif
JAVA_TESTS = org.rocksdb.BackupableDBOptionsTest\
JAVA_TESTS = \
org.rocksdb.BackupableDBOptionsTest\
org.rocksdb.BackupEngineTest\
org.rocksdb.BlockBasedTableConfigTest\
org.rocksdb.BuiltinComparatorTest\
org.rocksdb.util.BytewiseComparatorTest\
org.rocksdb.util.BytewiseComparatorIntTest\
org.rocksdb.CheckPointTest\
org.rocksdb.ClockCacheTest\
org.rocksdb.ColumnFamilyOptionsTest\
......@@ -115,16 +118,16 @@ JAVA_TESTS = org.rocksdb.BackupableDBOptionsTest\
org.rocksdb.CompactionPriorityTest\
org.rocksdb.CompactionStopStyleTest\
org.rocksdb.ComparatorOptionsTest\
org.rocksdb.ComparatorTest\
org.rocksdb.CompressionOptionsTest\
org.rocksdb.CompressionTypesTest\
org.rocksdb.DBOptionsTest\
org.rocksdb.DirectComparatorTest\
org.rocksdb.DirectSliceTest\
org.rocksdb.util.EnvironmentTest\
org.rocksdb.EnvOptionsTest\
org.rocksdb.HdfsEnvTest\
org.rocksdb.IngestExternalFileOptionsTest\
org.rocksdb.util.EnvironmentTest\
org.rocksdb.util.IntComparatorTest\
org.rocksdb.util.JNIComparatorTest\
org.rocksdb.FilterTest\
org.rocksdb.FlushTest\
org.rocksdb.InfoLogLevelTest\
......@@ -148,6 +151,7 @@ JAVA_TESTS = org.rocksdb.BackupableDBOptionsTest\
org.rocksdb.RateLimiterTest\
org.rocksdb.ReadOnlyTest\
org.rocksdb.ReadOptionsTest\
org.rocksdb.util.ReverseBytewiseComparatorIntTest\
org.rocksdb.RocksDBTest\
org.rocksdb.RocksDBExceptionTest\
org.rocksdb.DefaultEnvTest\
......
java/jmh/pom.xml
浏览文件 @ 7242dae7
......@@ -50,7 +50,7 @@
<dependency>
<groupId>org.rocksdb</groupId>
<artifactId>rocksdbjni</artifactId>
<version>6.4.6</version>
<version>6.6.0-SNAPSHOT</version>
</dependency>
<dependency>
......
java/jmh/src/main/java/org/rocksdb/jmh/ComparatorBenchmarks.java
浏览文件 @ 7242dae7
......@@ -9,7 +9,6 @@ package org.rocksdb.jmh;
import org.openjdk.jmh.annotations.*;
import org.rocksdb.*;
import org.rocksdb.util.BytewiseComparator;
import org.rocksdb.util.DirectBytewiseComparator;
import org.rocksdb.util.FileUtils;
import org.rocksdb.util.ReverseBytewiseComparator;
......@@ -26,13 +25,24 @@ public class ComparatorBenchmarks {
@Param({
"native_bytewise",
"native_reverse_bytewise",
"java_bytewise_adaptive_mutex",
"java_bytewise_non-adaptive_mutex",
"java_reverse_bytewise_adaptive_mutex",
"java_reverse_bytewise_non-adaptive_mutex",
"java_direct_bytewise_adaptive_mutex",
"java_direct_bytewise_non-adaptive_mutex"
"java_bytewise_non-direct_reused-64_adaptive-mutex",
"java_bytewise_non-direct_reused-64_non-adaptive-mutex",
"java_bytewise_non-direct_reused-64_thread-local",
"java_bytewise_direct_reused-64_adaptive-mutex",
"java_bytewise_direct_reused-64_non-adaptive-mutex",
"java_bytewise_direct_reused-64_thread-local",
"java_bytewise_non-direct_no-reuse",
"java_bytewise_direct_no-reuse",
"java_reverse_bytewise_non-direct_reused-64_adaptive-mutex",
"java_reverse_bytewise_non-direct_reused-64_non-adaptive-mutex",
"java_reverse_bytewise_non-direct_reused-64_thread-local",
"java_reverse_bytewise_direct_reused-64_adaptive-mutex",
"java_reverse_bytewise_direct_reused-64_non-adaptive-mutex",
"java_reverse_bytewise_direct_reused-64_thread-local",
"java_reverse_bytewise_non-direct_no-reuse",
"java_reverse_bytewise_direct_no-reuse"
})
public String comparatorName;
......@@ -50,42 +60,49 @@ public class ComparatorBenchmarks {
options = new Options()
.setCreateIfMissing(true);
if (comparatorName == null || "native_bytewise".equals(comparatorName)) {
if ("native_bytewise".equals(comparatorName)) {
options.setComparator(BuiltinComparator.BYTEWISE_COMPARATOR);
} else if ("native_reverse_bytewise".equals(comparatorName)) {
options.setComparator(BuiltinComparator.REVERSE_BYTEWISE_COMPARATOR);
} else if ("java_bytewise_adaptive_mutex".equals(comparatorName)) {
comparatorOptions = new ComparatorOptions()
.setUseAdaptiveMutex(true);
comparator = new BytewiseComparator(comparatorOptions);
options.setComparator(comparator);
} else if ("java_bytewise_non-adaptive_mutex".equals(comparatorName)) {
comparatorOptions = new ComparatorOptions()
.setUseAdaptiveMutex(false);
comparator = new BytewiseComparator(comparatorOptions);
options.setComparator(comparator);
} else if ("java_reverse_bytewise_adaptive_mutex".equals(comparatorName)) {
comparatorOptions = new ComparatorOptions()
.setUseAdaptiveMutex(true);
comparator = new ReverseBytewiseComparator(comparatorOptions);
options.setComparator(comparator);
} else if ("java_reverse_bytewise_non-adaptive_mutex".equals(comparatorName)) {
comparatorOptions = new ComparatorOptions()
.setUseAdaptiveMutex(false);
comparator = new ReverseBytewiseComparator(comparatorOptions);
options.setComparator(comparator);
} else if ("java_direct_bytewise_adaptive_mutex".equals(comparatorName)) {
comparatorOptions = new ComparatorOptions()
.setUseAdaptiveMutex(true);
comparator = new DirectBytewiseComparator(comparatorOptions);
options.setComparator(comparator);
} else if ("java_direct_bytewise_non-adaptive_mutex".equals(comparatorName)) {
comparatorOptions = new ComparatorOptions()
.setUseAdaptiveMutex(false);
comparator = new DirectBytewiseComparator(comparatorOptions);
} else if (comparatorName.startsWith("java_")) {
comparatorOptions = new ComparatorOptions();
if (comparatorName.indexOf("non-direct") > -1) {
comparatorOptions.setUseDirectBuffer(false);
} else if (comparatorName.indexOf("direct") > -1) {
comparatorOptions.setUseDirectBuffer(true);
}
if (comparatorName.indexOf("no-reuse") > -1) {
comparatorOptions.setMaxReusedBufferSize(-1);
} else if (comparatorName.indexOf("_reused-") > -1) {
final int idx = comparatorName.indexOf("_reused-");
String s = comparatorName.substring(idx + 8);
s = s.substring(0, s.indexOf('_'));
comparatorOptions.setMaxReusedBufferSize(Integer.parseInt(s));
}
if (comparatorName.indexOf("non-adaptive-mutex") > -1) {
comparatorOptions.setReusedSynchronisationType(ReusedSynchronisationType.MUTEX);
} else if (comparatorName.indexOf("adaptive-mutex") > -1) {
comparatorOptions.setReusedSynchronisationType(ReusedSynchronisationType.ADAPTIVE_MUTEX);
} else if (comparatorName.indexOf("thread-local") > -1) {
comparatorOptions.setReusedSynchronisationType(ReusedSynchronisationType.THREAD_LOCAL);
}
if (comparatorName.startsWith("java_bytewise")) {
comparator = new BytewiseComparator(comparatorOptions);
} else if (comparatorName.startsWith("java_reverse_bytewise")) {
comparator = new ReverseBytewiseComparator(comparatorOptions);
}
options.setComparator(comparator);
} else {
throw new IllegalArgumentException("Unknown comparator name: " + comparatorName);
throw new IllegalArgumentException("Unknown comparatorName: " + comparatorName);
}
db = RocksDB.open(options, dbDir.toAbsolutePath().toString());
......
java/rocksjni/comparator.cc
浏览文件 @ 7242dae7
......@@ -12,42 +12,33 @@
#include <functional>
#include <string>
#include "include/org_rocksdb_Comparator.h"
#include "include/org_rocksdb_DirectComparator.h"
#include "include/org_rocksdb_AbstractComparator.h"
#include "include/org_rocksdb_NativeComparatorWrapper.h"
#include "rocksjni/comparatorjnicallback.h"
#include "rocksjni/portal.h"
// <editor-fold desc="org.rocksdb.Comparator>
/*
* Class: org_rocksdb_Comparator
* Method: createNewComparator0
* Signature: ()J
* Class: org_rocksdb_AbstractComparator
* Method: createNewComparator
* Signature: (J)J
*/
jlong Java_org_rocksdb_Comparator_createNewComparator0(JNIEnv* env,
jobject jobj,
jlong copt_handle) {
jlong Java_org_rocksdb_AbstractComparator_createNewComparator(
JNIEnv* env, jobject jcomparator, jlong copt_handle) {
auto* copt =
reinterpret_cast<rocksdb::ComparatorJniCallbackOptions*>(copt_handle);
auto* c = new rocksdb::ComparatorJniCallback(env, jobj, copt);
auto* c = new rocksdb::ComparatorJniCallback(env, jcomparator, copt);
return reinterpret_cast<jlong>(c);
}
// </editor-fold>
// <editor-fold desc="org.rocksdb.DirectComparator>
/*
* Class: org_rocksdb_DirectComparator
* Method: createNewDirectComparator0
* Signature: ()J
* Class: org_rocksdb_AbstractComparator
* Method: usingDirectBuffers
* Signature: (J)Z
*/
jlong Java_org_rocksdb_DirectComparator_createNewDirectComparator0(
JNIEnv* env, jobject jobj, jlong copt_handle) {
auto* copt =
reinterpret_cast<rocksdb::ComparatorJniCallbackOptions*>(copt_handle);
auto* c = new rocksdb::DirectComparatorJniCallback(env, jobj, copt);
return reinterpret_cast<jlong>(c);
jboolean Java_org_rocksdb_AbstractComparator_usingDirectBuffers(
JNIEnv*, jobject, jlong jhandle) {
auto* c = reinterpret_cast<rocksdb::ComparatorJniCallback*>(jhandle);
return static_cast<jboolean>(c->m_options->direct_buffer);
}
/*
......@@ -60,4 +51,3 @@ void Java_org_rocksdb_NativeComparatorWrapper_disposeInternal(
auto* comparator = reinterpret_cast<rocksdb::Comparator*>(jcomparator_handle);
delete comparator;
}
// </editor-fold>
java/rocksjni/comparatorjnicallback.h
浏览文件 @ 7242dae7
......@@ -4,7 +4,7 @@
// (found in the LICENSE.Apache file in the root directory).
//
// This file implements the callback "bridge" between Java and C++ for
// rocksdb::Comparator and rocksdb::DirectComparator.
// rocksdb::Comparator
#ifndef JAVA_ROCKSJNI_COMPARATORJNICALLBACK_H_
#define JAVA_ROCKSJNI_COMPARATORJNICALLBACK_H_
......@@ -16,19 +16,54 @@
#include "rocksdb/comparator.h"
#include "rocksdb/slice.h"
#include "port/port.h"
#include "util/thread_local.h"
namespace rocksdb {
enum ReusedSynchronisationType {
/**
* Standard mutex.
*/
MUTEX,
/**
* Use adaptive mutex, which spins in the user space before resorting
* to kernel. This could reduce context switch when the mutex is not
* heavily contended. However, if the mutex is hot, we could end up
* wasting spin time.
*/
ADAPTIVE_MUTEX,
/**
* There is a reused buffer per-thread.
*/
THREAD_LOCAL
};
struct ComparatorJniCallbackOptions {
// Use adaptive mutex, which spins in the user space before resorting
// to kernel. This could reduce context switch when the mutex is not
// heavily contended. However, if the mutex is hot, we could end up
// wasting spin time.
// Default: false
bool use_adaptive_mutex;
ComparatorJniCallbackOptions() : use_adaptive_mutex(false) {
}
// Set the synchronisation type used to guard the reused buffers.
// Only used if max_reused_buffer_size > 0.
// Default: ADAPTIVE_MUTEX
ReusedSynchronisationType reused_synchronisation_type =
ReusedSynchronisationType::ADAPTIVE_MUTEX;
// Indicates if a direct byte buffer (i.e. outside of the normal
// garbage-collected heap) is used for the callbacks to Java,
// as opposed to a non-direct byte buffer which is a wrapper around
// an on-heap byte[].
// Default: true
bool direct_buffer = true;
// Maximum size of a buffer (in bytes) that will be reused.
// Comparators will use 5 of these buffers,
// so the retained memory size will be 5 * max_reused_buffer_size.
// When a buffer is needed for transferring data to a callback,
// if it requires less than max_reused_buffer_size, then an
// existing buffer will be reused, else a new buffer will be
// allocated just for that callback. -1 to disable.
// Default: 64 bytes
int32_t max_reused_buffer_size = 64;
};
/**
......@@ -46,47 +81,56 @@ struct ComparatorJniCallbackOptions {
* introduce independent locking in regions of each of those methods
* via the mutexs mtx_compare and mtx_findShortestSeparator respectively
*/
class BaseComparatorJniCallback : public JniCallback, public Comparator {
class ComparatorJniCallback : public JniCallback, public Comparator {
public:
BaseComparatorJniCallback(
JNIEnv* env, jobject jComparator,
const ComparatorJniCallbackOptions* copt);
ComparatorJniCallback(
JNIEnv* env, jobject jcomparator,
const ComparatorJniCallbackOptions* options);
~ComparatorJniCallback();
virtual const char* Name() const;
virtual int Compare(const Slice& a, const Slice& b) const;
virtual void FindShortestSeparator(
std::string* start, const Slice& limit) const;
virtual void FindShortSuccessor(std::string* key) const;
const ComparatorJniCallbackOptions* m_options;
private:
struct ThreadLocalBuf {
ThreadLocalBuf(JavaVM* _jvm, bool _direct_buffer, jobject _jbuf) :
jvm(_jvm), direct_buffer(_direct_buffer), jbuf(_jbuf) {}
JavaVM* jvm;
bool direct_buffer;
jobject jbuf;
};
inline void MaybeLockForReuse(const std::unique_ptr<port::Mutex>& mutex,
const bool cond) const;
inline void MaybeUnlockForReuse(const std::unique_ptr<port::Mutex>& mutex,
const bool cond) const;
jobject GetBuffer(JNIEnv* env, const Slice& src, bool reuse_buffer,
ThreadLocalPtr* tl_buf, jobject jreuse_buffer) const;
jobject ReuseBuffer(JNIEnv* env, const Slice& src,
jobject jreuse_buffer) const;
jobject NewBuffer(JNIEnv* env, const Slice& src) const;
void DeleteBuffer(JNIEnv* env, jobject jbuffer) const;
// used for synchronisation in compare method
std::unique_ptr<port::Mutex> mtx_compare;
// used for synchronisation in findShortestSeparator method
std::unique_ptr<port::Mutex> mtx_findShortestSeparator;
std::unique_ptr<port::Mutex> mtx_shortest;
// used for synchronisation in findShortSuccessor method
std::unique_ptr<port::Mutex> mtx_short;
std::unique_ptr<const char[]> m_name;
jmethodID m_jCompareMethodId;
jmethodID m_jFindShortestSeparatorMethodId;
jmethodID m_jFindShortSuccessorMethodId;
protected:
jobject m_jSliceA;
jobject m_jSliceB;
jobject m_jSliceLimit;
};
class ComparatorJniCallback : public BaseComparatorJniCallback {
public:
ComparatorJniCallback(
JNIEnv* env, jobject jComparator,
const ComparatorJniCallbackOptions* copt);
~ComparatorJniCallback();
};
class DirectComparatorJniCallback : public BaseComparatorJniCallback {
public:
DirectComparatorJniCallback(
JNIEnv* env, jobject jComparator,
const ComparatorJniCallbackOptions* copt);
~DirectComparatorJniCallback();
jclass m_abstract_comparator_jni_bridge_clazz; // TODO(AR) could we make this static somehow?
jclass m_jbytebuffer_clazz; // TODO(AR) we could cache this globally for the entire VM if we switch more APIs to use ByteBuffer // TODO(AR) could we make this static somehow?
jmethodID m_jcompare_mid; // TODO(AR) could we make this static somehow?
jmethodID m_jshortest_mid; // TODO(AR) could we make this static somehow?
jmethodID m_jshort_mid; // TODO(AR) could we make this static somehow?
jobject m_jcompare_buf_a;
jobject m_jcompare_buf_b;
jobject m_jshortest_buf_start;
jobject m_jshortest_buf_limit;
jobject m_jshort_buf_key;
ThreadLocalPtr* m_tl_buf_a;
ThreadLocalPtr* m_tl_buf_b;
};
} // namespace rocksdb
......
java/rocksjni/options.cc
浏览文件 @ 7242dae7
......@@ -173,14 +173,8 @@ void Java_org_rocksdb_Options_setComparatorHandle__JJB(
reinterpret_cast<rocksdb::ComparatorJniCallback*>(jcomparator_handle);
break;
// JAVA_DIRECT_COMPARATOR
case 0x1:
comparator = reinterpret_cast<rocksdb::DirectComparatorJniCallback*>(
jcomparator_handle);
break;
// JAVA_NATIVE_COMPARATOR_WRAPPER
case 0x2:
case 0x1:
comparator = reinterpret_cast<rocksdb::Comparator*>(jcomparator_handle);
break;
}
......@@ -3301,14 +3295,8 @@ void Java_org_rocksdb_ColumnFamilyOptions_setComparatorHandle__JJB(
reinterpret_cast<rocksdb::ComparatorJniCallback*>(jcomparator_handle);
break;
// JAVA_DIRECT_COMPARATOR
case 0x1:
comparator = reinterpret_cast<rocksdb::DirectComparatorJniCallback*>(
jcomparator_handle);
break;
// JAVA_NATIVE_COMPARATOR_WRAPPER
case 0x2:
case 0x1:
comparator = reinterpret_cast<rocksdb::Comparator*>(jcomparator_handle);
break;
}
......@@ -6869,24 +6857,75 @@ jlong Java_org_rocksdb_ComparatorOptions_newComparatorOptions(
/*
* Class: org_rocksdb_ComparatorOptions
* Method: useAdaptiveMutex
* Method: reusedSynchronisationType
* Signature: (J)B
*/
jbyte Java_org_rocksdb_ComparatorOptions_reusedSynchronisationType(
JNIEnv *, jobject, jlong jhandle) {
auto* comparator_opt =
reinterpret_cast<rocksdb::ComparatorJniCallbackOptions*>(jhandle);
return rocksdb::ReusedSynchronisationTypeJni::toJavaReusedSynchronisationType(
comparator_opt->reused_synchronisation_type);
}
/*
* Class: org_rocksdb_ComparatorOptions
* Method: setReusedSynchronisationType
* Signature: (JB)V
*/
void Java_org_rocksdb_ComparatorOptions_setReusedSynchronisationType(
JNIEnv*, jobject, jlong jhandle, jbyte jreused_synhcronisation_type) {
auto* comparator_opt =
reinterpret_cast<rocksdb::ComparatorJniCallbackOptions*>(jhandle);
comparator_opt->reused_synchronisation_type =
rocksdb::ReusedSynchronisationTypeJni::toCppReusedSynchronisationType(jreused_synhcronisation_type);
}
/*
* Class: org_rocksdb_ComparatorOptions
* Method: useDirectBuffer
* Signature: (J)Z
*/
jboolean Java_org_rocksdb_ComparatorOptions_useAdaptiveMutex(
jboolean Java_org_rocksdb_ComparatorOptions_useDirectBuffer(
JNIEnv*, jobject, jlong jhandle) {
return reinterpret_cast<rocksdb::ComparatorJniCallbackOptions*>(jhandle)
->use_adaptive_mutex;
return static_cast<jboolean>(
reinterpret_cast<rocksdb::ComparatorJniCallbackOptions*>(jhandle)
->direct_buffer);
}
/*
* Class: org_rocksdb_ComparatorOptions
* Method: setUseAdaptiveMutex
* Method: setUseDirectBuffer
* Signature: (JZ)V
*/
void Java_org_rocksdb_ComparatorOptions_setUseAdaptiveMutex(
JNIEnv*, jobject, jlong jhandle, jboolean juse_adaptive_mutex) {
void Java_org_rocksdb_ComparatorOptions_setUseDirectBuffer(
JNIEnv*, jobject, jlong jhandle, jboolean jdirect_buffer) {
reinterpret_cast<rocksdb::ComparatorJniCallbackOptions*>(jhandle)
->direct_buffer = jdirect_buffer == JNI_TRUE;
}
/*
* Class: org_rocksdb_ComparatorOptions
* Method: maxReusedBufferSize
* Signature: (J)I
*/
jint Java_org_rocksdb_ComparatorOptions_maxReusedBufferSize(
JNIEnv*, jobject, jlong jhandle) {
return static_cast<jint>(
reinterpret_cast<rocksdb::ComparatorJniCallbackOptions*>(jhandle)
->max_reused_buffer_size);
}
/*
* Class: org_rocksdb_ComparatorOptions
* Method: setMaxReusedBufferSize
* Signature: (JI)V
*/
void Java_org_rocksdb_ComparatorOptions_setMaxReusedBufferSize(
JNIEnv*, jobject, jlong jhandle, jint jmax_reused_buffer_size) {
reinterpret_cast<rocksdb::ComparatorJniCallbackOptions*>(jhandle)
->use_adaptive_mutex = static_cast<bool>(juse_adaptive_mutex);
->max_reused_buffer_size
= static_cast<int32_t>(jmax_reused_buffer_size);
}
/*
......
java/rocksjni/portal.h
浏览文件 @ 7242dae7
......@@ -1207,6 +1207,157 @@ class ByteJni : public JavaClass {
};
// The portal class for java.nio.ByteBuffer
class ByteBufferJni : public JavaClass {
public:
/**
* Get the Java Class java.nio.ByteBuffer
*
* @param env A pointer to the Java environment
*
* @return The Java Class or nullptr if one of the
* ClassFormatError, ClassCircularityError, NoClassDefFoundError,
* OutOfMemoryError or ExceptionInInitializerError exceptions is thrown
*/
static jclass getJClass(JNIEnv* env) {
return JavaClass::getJClass(env, "java/nio/ByteBuffer");
}
/**
* Get the Java Method: ByteBuffer#allocate
*
* @param env A pointer to the Java environment
* @param jbytebuffer_clazz if you have a reference to a ByteBuffer class, or nullptr
*
* @return The Java Method ID or nullptr if the class or method id could not
* be retieved
*/
static jmethodID getAllocateMethodId(JNIEnv* env,
jclass jbytebuffer_clazz = nullptr) {
const jclass jclazz =
jbytebuffer_clazz == nullptr ? getJClass(env) : jbytebuffer_clazz;
if (jclazz == nullptr) {
// exception occurred accessing class
return nullptr;
}
static jmethodID mid = env->GetStaticMethodID(
jclazz, "allocate", "(I)Ljava/nio/ByteBuffer;");
assert(mid != nullptr);
return mid;
}
/**
* Get the Java Method: ByteBuffer#array
*
* @param env A pointer to the Java environment
*
* @return The Java Method ID or nullptr if the class or method id could not
* be retieved
*/
static jmethodID getArrayMethodId(JNIEnv* env,
jclass jbytebuffer_clazz = nullptr) {
const jclass jclazz =
jbytebuffer_clazz == nullptr ? getJClass(env) : jbytebuffer_clazz;
if(jclazz == nullptr) {
// exception occurred accessing class
return nullptr;
}
static jmethodID mid = env->GetMethodID(jclazz, "array", "()[B");
assert(mid != nullptr);
return mid;
}
static jobject construct(
JNIEnv* env, const bool direct, const size_t capacity,
jclass jbytebuffer_clazz = nullptr) {
return constructWith(env, direct, nullptr, capacity, jbytebuffer_clazz);
}
static jobject constructWith(
JNIEnv* env, const bool direct, const char* buf, const size_t capacity,
jclass jbytebuffer_clazz = nullptr) {
if (direct) {
bool allocated = false;
if (buf == nullptr) {
buf = new char[capacity];
allocated = true;
}
jobject jbuf = env->NewDirectByteBuffer(const_cast<char*>(buf), static_cast<jlong>(capacity));
if (jbuf == nullptr) {
// exception occurred
if (allocated) {
delete[] static_cast<const char*>(buf);
}
return nullptr;
}
return jbuf;
} else {
const jclass jclazz =
jbytebuffer_clazz == nullptr ? getJClass(env) : jbytebuffer_clazz;
if (jclazz == nullptr) {
// exception occurred accessing class
return nullptr;
}
const jmethodID jmid_allocate = getAllocateMethodId(env, jbytebuffer_clazz);
if (jmid_allocate == nullptr) {
// exception occurred accessing class, or NoSuchMethodException or OutOfMemoryError
return nullptr;
}
const jobject jbuf = env->CallStaticObjectMethod(
jclazz, jmid_allocate, static_cast<jint>(capacity));
if (env->ExceptionCheck()) {
// exception occurred
return nullptr;
}
// set buffer data?
if (buf != nullptr) {
jbyteArray jarray = array(env, jbuf, jbytebuffer_clazz);
if (jarray == nullptr) {
// exception occurred
env->DeleteLocalRef(jbuf);
return nullptr;
}
jboolean is_copy = JNI_FALSE;
jbyte* ja = reinterpret_cast<jbyte*>(
env->GetPrimitiveArrayCritical(jarray, &is_copy));
if (ja == nullptr) {
// exception occurred
env->DeleteLocalRef(jarray);
env->DeleteLocalRef(jbuf);
return nullptr;
}
memcpy(ja, const_cast<char*>(buf), capacity);
env->ReleasePrimitiveArrayCritical(jarray, ja, 0);
env->DeleteLocalRef(jarray);
}
return jbuf;
}
}
static jbyteArray array(JNIEnv* env, const jobject& jbyte_buffer,
jclass jbytebuffer_clazz = nullptr) {
const jmethodID mid = getArrayMethodId(env, jbytebuffer_clazz);
if (mid == nullptr) {
// exception occurred accessing class, or NoSuchMethodException or OutOfMemoryError
return nullptr;
}
const jobject jarray = env->CallObjectMethod(jbyte_buffer, mid);
if (env->ExceptionCheck()) {
// exception occurred
return nullptr;
}
return static_cast<jbyteArray>(jarray);
}
};
// The portal class for java.lang.Integer
class IntegerJni : public JavaClass {
public:
......@@ -1405,7 +1556,7 @@ class JniUtil {
JNIEnv *env;
const jint env_rs = jvm->GetEnv(reinterpret_cast<void**>(&env),
JNI_VERSION_1_2);
JNI_VERSION_1_6);
if(env_rs == JNI_OK) {
// current thread is already attached, return the JNIEnv
......@@ -1423,8 +1574,8 @@ class JniUtil {
return nullptr;
}
} else if(env_rs == JNI_EVERSION) {
// error, JDK does not support JNI_VERSION_1_2+
std::cerr << "JniUtil::getJniEnv - Fatal: JDK does not support JNI_VERSION_1_2" << std::endl;
// error, JDK does not support JNI_VERSION_1_6+
std::cerr << "JniUtil::getJniEnv - Fatal: JDK does not support JNI_VERSION_1_6" << std::endl;
return nullptr;
} else {
std::cerr << "JniUtil::getJniEnv - Fatal: Unknown error: env_rs=" << env_rs << std::endl;
......@@ -3303,13 +3454,11 @@ class AbstractTransactionNotifierJni : public RocksDBNativeClass<
}
};
// The portal class for org.rocksdb.AbstractComparator
class AbstractComparatorJni : public RocksDBNativeClass<
const rocksdb::BaseComparatorJniCallback*,
AbstractComparatorJni> {
// The portal class for org.rocksdb.AbstractComparatorJniBridge
class AbstractComparatorJniBridge : public JavaClass {
public:
/**
* Get the Java Class org.rocksdb.AbstractComparator
* Get the Java Class org.rocksdb.AbstractComparatorJniBridge
*
* @param env A pointer to the Java environment
*
......@@ -3318,84 +3467,90 @@ class AbstractComparatorJni : public RocksDBNativeClass<
* OutOfMemoryError or ExceptionInInitializerError exceptions is thrown
*/
static jclass getJClass(JNIEnv* env) {
return RocksDBNativeClass::getJClass(env,
"org/rocksdb/AbstractComparator");
return JavaClass::getJClass(env,
"org/rocksdb/AbstractComparatorJniBridge");
}
/**
* Get the Java Method: Comparator#name
* Get the Java Method: Comparator#compareInternal
*
* @param env A pointer to the Java environment
* @param jclazz the AbstractComparatorJniBridge class
*
* @return The Java Method ID or nullptr if the class or method id could not
* be retieved
*/
static jmethodID getNameMethodId(JNIEnv* env) {
jclass jclazz = getJClass(env);
if(jclazz == nullptr) {
// exception occurred accessing class
return nullptr;
}
static jmethodID getCompareInternalMethodId(JNIEnv* env, jclass jclazz) {
static jmethodID mid =
env->GetMethodID(jclazz, "name", "()Ljava/lang/String;");
env->GetStaticMethodID(jclazz, "compareInternal",
"(Lorg/rocksdb/AbstractComparator;Ljava/nio/ByteBuffer;ILjava/nio/ByteBuffer;I)I");
assert(mid != nullptr);
return mid;
}
/**
* Get the Java Method: Comparator#compare
* Get the Java Method: Comparator#findShortestSeparatorInternal
*
* @param env A pointer to the Java environment
* @param jclazz the AbstractComparatorJniBridge class
*
* @return The Java Method ID or nullptr if the class or method id could not
* be retieved
*/
static jmethodID getCompareMethodId(JNIEnv* env) {
jclass jclazz = getJClass(env);
if(jclazz == nullptr) {
// exception occurred accessing class
return nullptr;
}
static jmethodID getFindShortestSeparatorInternalMethodId(JNIEnv* env, jclass jclazz) {
static jmethodID mid =
env->GetMethodID(jclazz, "compare",
"(Lorg/rocksdb/AbstractSlice;Lorg/rocksdb/AbstractSlice;)I");
env->GetStaticMethodID(jclazz, "findShortestSeparatorInternal",
"(Lorg/rocksdb/AbstractComparator;Ljava/nio/ByteBuffer;ILjava/nio/ByteBuffer;I)I");
assert(mid != nullptr);
return mid;
}
/**
* Get the Java Method: Comparator#findShortestSeparator
* Get the Java Method: Comparator#findShortSuccessorInternal
*
* @param env A pointer to the Java environment
* @param jclazz the AbstractComparatorJniBridge class
*
* @return The Java Method ID or nullptr if the class or method id could not
* be retieved
*/
static jmethodID getFindShortestSeparatorMethodId(JNIEnv* env) {
jclass jclazz = getJClass(env);
if(jclazz == nullptr) {
// exception occurred accessing class
return nullptr;
}
static jmethodID getFindShortSuccessorInternalMethodId(JNIEnv* env, jclass jclazz) {
static jmethodID mid =
env->GetMethodID(jclazz, "findShortestSeparator",
"(Ljava/lang/String;Lorg/rocksdb/AbstractSlice;)Ljava/lang/String;");
env->GetStaticMethodID(jclazz, "findShortSuccessorInternal",
"(Lorg/rocksdb/AbstractComparator;Ljava/nio/ByteBuffer;I)I");
assert(mid != nullptr);
return mid;
}
};
// The portal class for org.rocksdb.AbstractComparator
class AbstractComparatorJni : public RocksDBNativeClass<
const rocksdb::ComparatorJniCallback*,
AbstractComparatorJni> {
public:
/**
* Get the Java Method: Comparator#findShortSuccessor
* Get the Java Class org.rocksdb.AbstractComparator
*
* @param env A pointer to the Java environment
*
* @return The Java Class or nullptr if one of the
* ClassFormatError, ClassCircularityError, NoClassDefFoundError,
* OutOfMemoryError or ExceptionInInitializerError exceptions is thrown
*/
static jclass getJClass(JNIEnv* env) {
return RocksDBNativeClass::getJClass(env,
"org/rocksdb/AbstractComparator");
}
/**
* Get the Java Method: Comparator#name
*
* @param env A pointer to the Java environment
*
* @return The Java Method ID or nullptr if the class or method id could not
* be retieved
*/
static jmethodID getFindShortSuccessorMethodId(JNIEnv* env) {
static jmethodID getNameMethodId(JNIEnv* env) {
jclass jclazz = getJClass(env);
if(jclazz == nullptr) {
// exception occurred accessing class
......@@ -3403,8 +3558,7 @@ class AbstractComparatorJni : public RocksDBNativeClass<
}
static jmethodID mid =
env->GetMethodID(jclazz, "findShortSuccessor",
"(Ljava/lang/String;)Ljava/lang/String;");
env->GetMethodID(jclazz, "name", "()Ljava/lang/String;");
assert(mid != nullptr);
return mid;
}
......@@ -7109,5 +7263,42 @@ class WalProcessingOptionJni {
}
}
};
// The portal class for org.rocksdb.ReusedSynchronisationType
class ReusedSynchronisationTypeJni {
public:
// Returns the equivalent org.rocksdb.ReusedSynchronisationType for the provided
// C++ rocksdb::ReusedSynchronisationType enum
static jbyte toJavaReusedSynchronisationType(
const rocksdb::ReusedSynchronisationType& reused_synchronisation_type) {
switch(reused_synchronisation_type) {
case rocksdb::ReusedSynchronisationType::MUTEX:
return 0x0;
case rocksdb::ReusedSynchronisationType::ADAPTIVE_MUTEX:
return 0x1;
case rocksdb::ReusedSynchronisationType::THREAD_LOCAL:
return 0x2;
default:
return 0x7F; // undefined
}
}
// Returns the equivalent C++ rocksdb::ReusedSynchronisationType enum for the
// provided Java org.rocksdb.ReusedSynchronisationType
static rocksdb::ReusedSynchronisationType toCppReusedSynchronisationType(
jbyte reused_synchronisation_type) {
switch(reused_synchronisation_type) {
case 0x0:
return rocksdb::ReusedSynchronisationType::MUTEX;
case 0x1:
return rocksdb::ReusedSynchronisationType::ADAPTIVE_MUTEX;
case 0x2:
return rocksdb::ReusedSynchronisationType::THREAD_LOCAL;
default:
// undefined/default
return rocksdb::ReusedSynchronisationType::ADAPTIVE_MUTEX;
}
}
};
} // namespace rocksdb
#endif // JAVA_ROCKSJNI_PORTAL_H_
java/rocksjni/sst_file_writerjni.cc
浏览文件 @ 7242dae7
......@@ -33,14 +33,8 @@ jlong Java_org_rocksdb_SstFileWriter_newSstFileWriter__JJJB(
jcomparator_handle);
break;
// JAVA_DIRECT_COMPARATOR
case 0x1:
comparator = reinterpret_cast<rocksdb::DirectComparatorJniCallback *>(
jcomparator_handle);
break;
// JAVA_NATIVE_COMPARATOR_WRAPPER
case 0x2:
case 0x1:
comparator = reinterpret_cast<rocksdb::Comparator *>(jcomparator_handle);
break;
}
......
java/rocksjni/write_batch_with_index.cc
浏览文件 @ 7242dae7
......@@ -51,15 +51,8 @@ jlong Java_org_rocksdb_WriteBatchWithIndex_newWriteBatchWithIndex__JBIZ(
jfallback_index_comparator_handle);
break;
// JAVA_DIRECT_COMPARATOR
case 0x1:
fallback_comparator =
reinterpret_cast<rocksdb::DirectComparatorJniCallback*>(
jfallback_index_comparator_handle);
break;
// JAVA_NATIVE_COMPARATOR_WRAPPER
case 0x2:
case 0x1:
fallback_comparator = reinterpret_cast<rocksdb::Comparator*>(
jfallback_index_comparator_handle);
break;
......
java/src/main/java/org/rocksdb/AbstractComparator.java
浏览文件 @ 7242dae7
......@@ -5,19 +5,18 @@
package org.rocksdb;
import java.nio.ByteBuffer;
/**
* Comparators are used by RocksDB to determine
* the ordering of keys.
*
* This class is package private, implementers
* should extend either of the public abstract classes:
* @see org.rocksdb.Comparator
* @see org.rocksdb.DirectComparator
* Implementations of Comparators in Java should extend this class.
*/
public abstract class AbstractComparator<T extends AbstractSlice<?>>
public abstract class AbstractComparator
extends RocksCallbackObject {
protected AbstractComparator() {
AbstractComparator() {
super();
}
......@@ -25,6 +24,11 @@ public abstract class AbstractComparator<T extends AbstractSlice<?>>
super(copt.nativeHandle_);
}
@Override
protected long initializeNative(final long... nativeParameterHandles) {
return createNewComparator(nativeParameterHandles[0]);
}
/**
* Get the type of this comparator.
*
......@@ -32,7 +36,9 @@ public abstract class AbstractComparator<T extends AbstractSlice<?>>
*
* @return The type of the comparator.
*/
abstract ComparatorType getComparatorType();
ComparatorType getComparatorType() {
return ComparatorType.JAVA_COMPARATOR;
}
/**
* The name of the comparator. Used to check for comparator
......@@ -50,54 +56,69 @@ public abstract class AbstractComparator<T extends AbstractSlice<?>>
public abstract String name();
/**
* Three-way key comparison
* Three-way key comparison. Implementations should provide a
* <a href="https://en.wikipedia.org/wiki/Total_order">total order</a>
* on keys that might be passed to it.
*
* The implementation may modify the {@code ByteBuffer}s passed in, though
* it would be unconventional to modify the "limit" or any of the
* underlying bytes. As a callback, RocksJava will ensure that {@code a}
* is a different instance from {@code b}.
*
* @param a Slice access to first key
* @param b Slice access to second key
* @param a buffer containing the first key in its "remaining" elements
* @param b buffer containing the second key in its "remaining" elements
*
* @return Should return either:
* @return Should return either:
* 1) &lt; 0 if "a" &lt; "b"
* 2) == 0 if "a" == "b"
* 3) &gt; 0 if "a" &gt; "b"
*/
public abstract int compare(final T a, final T b);
public abstract int compare(final ByteBuffer a, final ByteBuffer b);
/**
* <p>Used to reduce the space requirements
* for internal data structures like index blocks.</p>
*
* <p>If start &lt; limit, you may return a new start which is a
* <p>If start &lt; limit, you may modify start which is a
* shorter string in [start, limit).</p>
*
* <p>Simple comparator implementations may return null if they
* wish to use start unchanged. i.e., an implementation of
* this method that does nothing is correct.</p>
* If you modify start, it is expected that you set the byte buffer so that
* a subsequent read of start.remaining() bytes from start.position()
* to start.limit() will obtain the new start value.
*
* @param start String
* @param limit of type T
* <p>Simple comparator implementations may return with start unchanged.
* i.e., an implementation of this method that does nothing is correct.</p>
*
* @return a shorter start, or null
* @param start the start
* @param limit the limit
*/
public String findShortestSeparator(final String start, final T limit) {
return null;
public void findShortestSeparator(final ByteBuffer start,
final ByteBuffer limit) {
// no-op
}
/**
* <p>Used to reduce the space requirements
* for internal data structures like index blocks.</p>
*
* <p>You may return a new short key (key1) where
* <p>You may change key to a shorter key (key1) where
* key1 &ge; key.</p>
*
* <p>Simple comparator implementations may return null if they
* wish to leave the key unchanged. i.e., an implementation of
* <p>Simple comparator implementations may return the key unchanged.
* i.e., an implementation of
* this method that does nothing is correct.</p>
*
* @param key String
*
* @return a shorter key, or null
* @param key the key
*/
public String findShortSuccessor(final String key) {
return null;
public void findShortSuccessor(final ByteBuffer key) {
// no-op
}
public final boolean usingDirectBuffers() {
return usingDirectBuffers(nativeHandle_);
}
private native boolean usingDirectBuffers(final long nativeHandle);
private native long createNewComparator(final long comparatorOptionsHandle);
}
java/src/main/java/org/rocksdb/AbstractComparatorJniBridge.java 0 → 100644
浏览文件 @ 7242dae7
// Copyright (c) 2011-present, Facebook, Inc. All rights reserved.
// This source code is licensed under both the GPLv2 (found in the
// COPYING file in the root directory) and Apache 2.0 License
// (found in the LICENSE.Apache file in the root directory).
package org.rocksdb;
import java.nio.ByteBuffer;
/**
* This class is intentionally private,
* it holds methods which are called
* from C++ to interact with a Comparator
* written in Java.
*
* Placing these bridge methods in this
* class keeps the API of the
* {@link org.rocksdb.AbstractComparator} clean.
*/
class AbstractComparatorJniBridge {
/**
* Only called from JNI.
*
* Simply a bridge to calling
* {@link AbstractComparator#compare(ByteBuffer, ByteBuffer)},
* which ensures that the byte buffer lengths are correct
* before and after the call.
*
* @param comparator the comparator object on which to
* call {@link AbstractComparator#compare(ByteBuffer, ByteBuffer)}
* @param a buffer access to first key
* @param aLen the length of the a key,
* may be smaller than the buffer {@code a}
* @param b buffer access to second key
* @param bLen the length of the b key,
* may be smaller than the buffer {@code b}
*
* @return the result of the comparison
*/
private static int compareInternal(
final AbstractComparator comparator,
final ByteBuffer a, final int aLen,
final ByteBuffer b, final int bLen) {
if (aLen != -1) {
a.mark();
a.limit(aLen);
}
if (bLen != -1) {
b.mark();
b.limit(bLen);
}
final int c = comparator.compare(a, b);
if (aLen != -1) {
a.reset();
}
if (bLen != -1) {
b.reset();
}
return c;
}
/**
* Only called from JNI.
*
* Simply a bridge to calling
* {@link AbstractComparator#findShortestSeparator(ByteBuffer, ByteBuffer)},
* which ensures that the byte buffer lengths are correct
* before the call.
*
* @param comparator the comparator object on which to
* call {@link AbstractComparator#findShortestSeparator(ByteBuffer, ByteBuffer)}
* @param start buffer access to the start key
* @param startLen the length of the start key,
* may be smaller than the buffer {@code start}
* @param limit buffer access to the limit key
* @param limitLen the length of the limit key,
* may be smaller than the buffer {@code limit}
*
* @return either {@code startLen} if the start key is unchanged, otherwise
* the new length of the start key
*/
private static int findShortestSeparatorInternal(
final AbstractComparator comparator,
final ByteBuffer start, final int startLen,
final ByteBuffer limit, final int limitLen) {
if (startLen != -1) {
start.limit(startLen);
}
if (limitLen != -1) {
limit.limit(limitLen);
}
comparator.findShortestSeparator(start, limit);
return start.remaining();
}
/**
* Only called from JNI.
*
* Simply a bridge to calling
* {@link AbstractComparator#findShortestSeparator(ByteBuffer, ByteBuffer)},
* which ensures that the byte buffer length is correct
* before the call.
*
* @param comparator the comparator object on which to
* call {@link AbstractComparator#findShortSuccessor(ByteBuffer)}
* @param key buffer access to the key
* @param keyLen the length of the key,
* may be smaller than the buffer {@code key}
*
* @return either keyLen if the key is unchanged, otherwise the new length of the key
*/
private static int findShortSuccessorInternal(
final AbstractComparator comparator,
final ByteBuffer key, final int keyLen) {
if (keyLen != -1) {
key.limit(keyLen);
}
comparator.findShortSuccessor(key);
return key.remaining();
}
}
java/src/main/java/org/rocksdb/ColumnFamilyOptions.java
浏览文件 @ 7242dae7
......@@ -170,7 +170,7 @@ public class ColumnFamilyOptions extends RocksObject
@Override
public ColumnFamilyOptions setComparator(
final AbstractComparator<? extends AbstractSlice<?>> comparator) {
final AbstractComparator comparator) {
assert (isOwningHandle());
setComparatorHandle(nativeHandle_, comparator.nativeHandle_,
comparator.getComparatorType().getValue());
......@@ -989,7 +989,7 @@ public class ColumnFamilyOptions extends RocksObject
// NOTE: If you add new member variables, please update the copy constructor above!
private MemTableConfig memTableConfig_;
private TableFormatConfig tableFormatConfig_;
private AbstractComparator<? extends AbstractSlice<?>> comparator_;
private AbstractComparator comparator_;
private AbstractCompactionFilter<? extends AbstractSlice<?>> compactionFilter_;
private AbstractCompactionFilterFactory<? extends AbstractCompactionFilter<?>>
compactionFilterFactory_;
......
java/src/main/java/org/rocksdb/ColumnFamilyOptionsInterface.java
浏览文件 @ 7242dae7
......@@ -121,7 +121,7 @@ public interface ColumnFamilyOptionsInterface<T extends ColumnFamilyOptionsInter
* @return the instance of the current object.
*/
T setComparator(
AbstractComparator<? extends AbstractSlice<?>> comparator);
AbstractComparator comparator);
/**
* <p>Set the merge operator to be used for merging two merge operands
......
java/src/main/java/org/rocksdb/Comparator.java 已删除 100644 → 0
浏览文件 @ 800d24dd
// Copyright (c) 2011-present, Facebook, Inc. All rights reserved.
// This source code is licensed under both the GPLv2 (found in the
// COPYING file in the root directory) and Apache 2.0 License
// (found in the LICENSE.Apache file in the root directory).
package org.rocksdb;
/**
* Base class for comparators which will receive
* byte[] based access via org.rocksdb.Slice in their
* compare method implementation.
*
* byte[] based slices perform better when small keys
* are involved. When using larger keys consider
* using @see org.rocksdb.DirectComparator
*/
public abstract class Comparator extends AbstractComparator<Slice> {
public Comparator(final ComparatorOptions copt) {
super(copt);
}
@Override
protected long initializeNative(final long... nativeParameterHandles) {
return createNewComparator0(nativeParameterHandles[0]);
}
@Override
final ComparatorType getComparatorType() {
return ComparatorType.JAVA_COMPARATOR;
}
private native long createNewComparator0(final long comparatorOptionsHandle);
}
java/src/main/java/org/rocksdb/ComparatorOptions.java
浏览文件 @ 7242dae7
// Copyright (c) Facebook, Inc. and its affiliates. All Rights Reserved.
// Copyright (c) 2011-present, Facebook, Inc. All rights reserved.
// This source code is licensed under both the GPLv2 (found in the
// COPYING file in the root directory) and Apache 2.0 License
// (found in the LICENSE.Apache file in the root directory).
package org.rocksdb;
/**
......@@ -15,38 +18,116 @@ public class ComparatorOptions extends RocksObject {
}
/**
* Use adaptive mutex, which spins in the user space before resorting
* to kernel. This could reduce context switch when the mutex is not
* heavily contended. However, if the mutex is hot, we could end up
* wasting spin time.
* Default: false
* Get the synchronisation type used to guard the reused buffers.
* Only used if {@link #maxReusedBufferSize()} &gt; 0
* Default: {@link ReusedSynchronisationType#ADAPTIVE_MUTEX}
*
* @return true if adaptive mutex is used.
* @return the synchronisation type
*/
public boolean useAdaptiveMutex() {
public ReusedSynchronisationType reusedSynchronisationType() {
assert(isOwningHandle());
return useAdaptiveMutex(nativeHandle_);
return ReusedSynchronisationType.getReusedSynchronisationType(
reusedSynchronisationType(nativeHandle_));
}
/**
* Use adaptive mutex, which spins in the user space before resorting
* to kernel. This could reduce context switch when the mutex is not
* heavily contended. However, if the mutex is hot, we could end up
* wasting spin time.
* Default: false
* Set the synchronisation type used to guard the reused buffers.
* Only used if {@link #maxReusedBufferSize()} &gt; 0
* Default: {@link ReusedSynchronisationType#ADAPTIVE_MUTEX}
*
* @param reusedSynchronisationType the synchronisation type
*
* @param useAdaptiveMutex true if adaptive mutex is used.
* @return the reference to the current comparator options.
*/
public ComparatorOptions setUseAdaptiveMutex(final boolean useAdaptiveMutex) {
public ComparatorOptions setReusedSynchronisationType(
final ReusedSynchronisationType reusedSynchronisationType) {
assert (isOwningHandle());
setUseAdaptiveMutex(nativeHandle_, useAdaptiveMutex);
setReusedSynchronisationType(nativeHandle_,
reusedSynchronisationType.getValue());
return this;
}
/**
* Indicates if a direct byte buffer (i.e. outside of the normal
* garbage-collected heap) is used, as opposed to a non-direct byte buffer
* which is a wrapper around an on-heap byte[].
*
* Default: true
*
* @return true if a direct byte buffer will be used, false otherwise
*/
public boolean useDirectBuffer() {
assert(isOwningHandle());
return useDirectBuffer(nativeHandle_);
}
/**
* Controls whether a direct byte buffer (i.e. outside of the normal
* garbage-collected heap) is used, as opposed to a non-direct byte buffer
* which is a wrapper around an on-heap byte[].
*
* Default: true
*
* @param useDirectBuffer true if a direct byte buffer should be used,
* false otherwise
* @return the reference to the current comparator options.
*/
public ComparatorOptions setUseDirectBuffer(final boolean useDirectBuffer) {
assert(isOwningHandle());
setUseDirectBuffer(nativeHandle_, useDirectBuffer);
return this;
}
/**
* Maximum size of a buffer (in bytes) that will be reused.
* Comparators will use 5 of these buffers,
* so the retained memory size will be 5 * max_reused_buffer_size.
* When a buffer is needed for transferring data to a callback,
* if it requires less than {@code maxReuseBufferSize}, then an
* existing buffer will be reused, else a new buffer will be
* allocated just for that callback.
*
* Default: 64 bytes
*
* @return the maximum size of a buffer which is reused,
* or 0 if reuse is disabled
*/
public int maxReusedBufferSize() {
assert(isOwningHandle());
return maxReusedBufferSize(nativeHandle_);
}
/**
* Sets the maximum size of a buffer (in bytes) that will be reused.
* Comparators will use 5 of these buffers,
* so the retained memory size will be 5 * max_reused_buffer_size.
* When a buffer is needed for transferring data to a callback,
* if it requires less than {@code maxReuseBufferSize}, then an
* existing buffer will be reused, else a new buffer will be
* allocated just for that callback.
*
* Default: 64 bytes
*
* @param maxReusedBufferSize the maximum size for a buffer to reuse, or 0 to
* disable reuse
*
* @return the maximum size of a buffer which is reused
*/
public ComparatorOptions setMaxReusedBufferSize(final int maxReusedBufferSize) {
assert(isOwningHandle());
setMaxReusedBufferSize(nativeHandle_, maxReusedBufferSize);
return this;
}
private native static long newComparatorOptions();
private native boolean useAdaptiveMutex(final long handle);
private native void setUseAdaptiveMutex(final long handle,
final boolean useAdaptiveMutex);
private native byte reusedSynchronisationType(final long handle);
private native void setReusedSynchronisationType(final long handle,
final byte reusedSynchronisationType);
private native boolean useDirectBuffer(final long handle);
private native void setUseDirectBuffer(final long handle,
final boolean useDirectBuffer);
private native int maxReusedBufferSize(final long handle);
private native void setMaxReusedBufferSize(final long handle,
final int maxReuseBufferSize);
@Override protected final native void disposeInternal(final long handle);
}
java/src/main/java/org/rocksdb/ComparatorType.java
浏览文件 @ 7242dae7
......@@ -7,8 +7,7 @@ package org.rocksdb;
enum ComparatorType {
JAVA_COMPARATOR((byte)0x0),
JAVA_DIRECT_COMPARATOR((byte)0x1),
JAVA_NATIVE_COMPARATOR_WRAPPER((byte)0x2);
JAVA_NATIVE_COMPARATOR_WRAPPER((byte)0x1);
private final byte value;
......
java/src/main/java/org/rocksdb/DirectComparator.java 已删除 100644 → 0
浏览文件 @ 800d24dd
// Copyright (c) 2011-present, Facebook, Inc. All rights reserved.
// This source code is licensed under both the GPLv2 (found in the
// COPYING file in the root directory) and Apache 2.0 License
// (found in the LICENSE.Apache file in the root directory).
package org.rocksdb;
/**
* Base class for comparators which will receive
* ByteBuffer based access via org.rocksdb.DirectSlice
* in their compare method implementation.
*
* ByteBuffer based slices perform better when large keys
* are involved. When using smaller keys consider
* using @see org.rocksdb.Comparator
*/
public abstract class DirectComparator extends AbstractComparator<DirectSlice> {
public DirectComparator(final ComparatorOptions copt) {
super(copt);
}
@Override
protected long initializeNative(final long... nativeParameterHandles) {
return createNewDirectComparator0(nativeParameterHandles[0]);
}
@Override
final ComparatorType getComparatorType() {
return ComparatorType.JAVA_DIRECT_COMPARATOR;
}
private native long createNewDirectComparator0(
final long comparatorOptionsHandle);
}
java/src/main/java/org/rocksdb/NativeComparatorWrapper.java
浏览文件 @ 7242dae7
......@@ -5,6 +5,8 @@
package org.rocksdb;
import java.nio.ByteBuffer;
/**
* A simple abstraction to allow a Java class to wrap a custom comparator
* implemented in C++.
......@@ -12,7 +14,7 @@ package org.rocksdb;
* The native comparator must directly extend rocksdb::Comparator.
*/
public abstract class NativeComparatorWrapper
extends AbstractComparator<Slice> {
extends AbstractComparator {
@Override
final ComparatorType getComparatorType() {
......@@ -26,26 +28,26 @@ public abstract class NativeComparatorWrapper
}
@Override
public final int compare(final Slice s1, final Slice s2) {
public final int compare(final ByteBuffer s1, final ByteBuffer s2) {
throw new IllegalStateException("This should not be called. " +
"Implementation is in Native code");
}
@Override
public final String findShortestSeparator(final String start, final Slice limit) {
public final void findShortestSeparator(final ByteBuffer start, final ByteBuffer limit) {
throw new IllegalStateException("This should not be called. " +
"Implementation is in Native code");
}
@Override
public final String findShortSuccessor(final String key) {
public final void findShortSuccessor(final ByteBuffer key) {
throw new IllegalStateException("This should not be called. " +
"Implementation is in Native code");
}
/**
* We override {@link RocksCallbackObject#disposeInternal()}
* as disposing of a native rocksd::Comparator extension requires
* as disposing of a native rocksdb::Comparator extension requires
* a slightly different approach as it is not really a RocksCallbackObject
*/
@Override
......
java/src/main/java/org/rocksdb/OptimisticTransactionOptions.java
浏览文件 @ 7242dae7
......@@ -37,7 +37,7 @@ public class OptimisticTransactionOptions extends RocksObject
* @return this OptimisticTransactionOptions instance
*/
public OptimisticTransactionOptions setComparator(
final AbstractComparator<? extends AbstractSlice<?>> comparator) {
final AbstractComparator comparator) {
assert(isOwningHandle());
setComparator(nativeHandle_, comparator.nativeHandle_);
return this;
......
java/src/main/java/org/rocksdb/Options.java
浏览文件 @ 7242dae7
......@@ -193,7 +193,7 @@ public class Options extends RocksObject
@Override
public Options setComparator(
final AbstractComparator<? extends AbstractSlice<?>> comparator) {
final AbstractComparator comparator) {
assert(isOwningHandle());
setComparatorHandle(nativeHandle_, comparator.nativeHandle_,
comparator.getComparatorType().getValue());
......@@ -2169,7 +2169,7 @@ public class Options extends RocksObject
private MemTableConfig memTableConfig_;
private TableFormatConfig tableFormatConfig_;
private RateLimiter rateLimiter_;
private AbstractComparator<? extends AbstractSlice<?>> comparator_;
private AbstractComparator comparator_;
private AbstractCompactionFilter<? extends AbstractSlice<?>> compactionFilter_;
private AbstractCompactionFilterFactory<? extends AbstractCompactionFilter<?>>
compactionFilterFactory_;
......
java/src/main/java/org/rocksdb/ReusedSynchronisationType.java 0 → 100644
浏览文件 @ 7242dae7
// Copyright (c) 2011-present, Facebook, Inc. All rights reserved.
// This source code is licensed under both the GPLv2 (found in the
// COPYING file in the root directory) and Apache 2.0 License
// (found in the LICENSE.Apache file in the root directory).
package org.rocksdb;
/**
* Determines the type of synchronisation primitive used
* in native code.
*/
public enum ReusedSynchronisationType {
/**
* Standard mutex.
*/
MUTEX((byte)0x0),
/**
* Use adaptive mutex, which spins in the user space before resorting
* to kernel. This could reduce context switch when the mutex is not
* heavily contended. However, if the mutex is hot, we could end up
* wasting spin time.
*/
ADAPTIVE_MUTEX((byte)0x1),
/**
* There is a reused buffer per-thread.
*/
THREAD_LOCAL((byte)0x2);
private final byte value;
ReusedSynchronisationType(final byte value) {
this.value = value;
}
/**
* Returns the byte value of the enumerations value
*
* @return byte representation
*/
public byte getValue() {
return value;
}
/**
* Get ReusedSynchronisationType by byte value.
*
* @param value byte representation of ReusedSynchronisationType.
*
* @return {@link org.rocksdb.ReusedSynchronisationType} instance.
* @throws java.lang.IllegalArgumentException if an invalid
* value is provided.
*/
public static ReusedSynchronisationType getReusedSynchronisationType(
final byte value) {
for (final ReusedSynchronisationType reusedSynchronisationType
: ReusedSynchronisationType.values()) {
if (reusedSynchronisationType.getValue() == value) {
return reusedSynchronisationType;
}
}
throw new IllegalArgumentException(
"Illegal value provided for ReusedSynchronisationType.");
}
}
java/src/main/java/org/rocksdb/SstFileWriter.java
浏览文件 @ 7242dae7
......@@ -28,7 +28,7 @@ public class SstFileWriter extends RocksObject {
*/
@Deprecated
public SstFileWriter(final EnvOptions envOptions, final Options options,
final AbstractComparator<? extends AbstractSlice<?>> comparator) {
final AbstractComparator comparator) {
super(newSstFileWriter(
envOptions.nativeHandle_, options.nativeHandle_, comparator.nativeHandle_,
comparator.getComparatorType().getValue()));
......
java/src/main/java/org/rocksdb/WriteBatchWithIndex.java
浏览文件 @ 7242dae7
......@@ -57,7 +57,7 @@ public class WriteBatchWithIndex extends AbstractWriteBatch {
* show two entries with the same key.
*/
public WriteBatchWithIndex(
final AbstractComparator<? extends AbstractSlice<?>>
final AbstractComparator
fallbackIndexComparator, final int reservedBytes,
final boolean overwriteKey) {
super(newWriteBatchWithIndex(fallbackIndexComparator.nativeHandle_,
......
java/src/main/java/org/rocksdb/util/ByteUtil.java 0 → 100644
浏览文件 @ 7242dae7
package org.rocksdb.util;
import java.nio.ByteBuffer;
import static java.nio.charset.StandardCharsets.UTF_8;
public class ByteUtil {
/**
* Convert a String to a UTF-8 byte array.
*
* @param str the string
*
* @return the byte array.
*/
public static byte[] bytes(final String str) {
return str.getBytes(UTF_8);
}
/**
* Compares the first {@code count} bytes of two areas of memory. Returns
* zero if they are the same, a value less than zero if {@code x} is
* lexically less than {@code y}, or a value greater than zero if {@code x}
* is lexically greater than {@code y}. Note that lexical order is determined
* as if comparing unsigned char arrays.
*
* Similar to <a href="https://github.com/gcc-mirror/gcc/blob/master/libiberty/memcmp.c">memcmp.c</a>.
*
* @param x the first value to compare with
* @param y the second value to compare against
* @param count the number of bytes to compare
*
* @return the result of the comparison
*/
public static int memcmp(final ByteBuffer x, final ByteBuffer y,
final int count) {
for (int idx = 0; idx < count; idx++) {
final int aa = x.get(idx) & 0xff;
final int bb = y.get(idx) & 0xff;
if (aa != bb) {
return aa - bb;
}
}
return 0;
}
}
java/src/main/java/org/rocksdb/util/BytewiseComparator.java
浏览文件 @ 7242dae7
......@@ -9,6 +9,8 @@ import org.rocksdb.*;
import java.nio.ByteBuffer;
import static org.rocksdb.util.ByteUtil.memcmp;
/**
* This is a Java Native implementation of the C++
* equivalent BytewiseComparatorImpl using {@link Slice}
......@@ -19,7 +21,7 @@ import java.nio.ByteBuffer;
* and you most likely instead wanted
* {@link org.rocksdb.BuiltinComparator#BYTEWISE_COMPARATOR}
*/
public class BytewiseComparator extends Comparator {
public final class BytewiseComparator extends AbstractComparator {
public BytewiseComparator(final ComparatorOptions copt) {
super(copt);
......@@ -31,61 +33,89 @@ public class BytewiseComparator extends Comparator {
}
@Override
public int compare(final Slice a, final Slice b) {
return compare(a.data(), b.data());
public int compare(final ByteBuffer a, final ByteBuffer b) {
return _compare(a, b);
}
@Override
public String findShortestSeparator(final String start,
final Slice limit) {
final byte[] startBytes = start.getBytes();
final byte[] limitBytes = limit.data();
static int _compare(final ByteBuffer a, final ByteBuffer b) {
assert(a != null && b != null);
final int minLen = a.remaining() < b.remaining() ?
a.remaining() : b.remaining();
int r = memcmp(a, b, minLen);
if (r == 0) {
if (a.remaining() < b.remaining()) {
r = -1;
} else if (a.remaining() > b.remaining()) {
r = +1;
}
}
return r;
}
@Override
public void findShortestSeparator(final ByteBuffer start,
final ByteBuffer limit) {
// Find length of common prefix
final int min_length = Math.min(startBytes.length, limit.size());
int diff_index = 0;
while ((diff_index < min_length) &&
(startBytes[diff_index] == limitBytes[diff_index])) {
diff_index++;
final int minLength = Math.min(start.remaining(), limit.remaining());
int diffIndex = 0;
while (diffIndex < minLength &&
start.get(diffIndex) == limit.get(diffIndex)) {
diffIndex++;
}
if (diff_index >= min_length) {
if (diffIndex >= minLength) {
// Do not shorten if one string is a prefix of the other
} else {
final byte diff_byte = startBytes[diff_index];
if(diff_byte < 0xff && diff_byte + 1 < limitBytes[diff_index]) {
final byte shortest[] = new byte[diff_index + 1];
System.arraycopy(startBytes, 0, shortest, 0, diff_index + 1);
shortest[diff_index]++;
assert(compare(shortest, limitBytes) < 0);
return new String(shortest);
final int startByte = start.get(diffIndex) & 0xff;
final int limitByte = limit.get(diffIndex) & 0xff;
if (startByte >= limitByte) {
// Cannot shorten since limit is smaller than start or start is
// already the shortest possible.
return;
}
}
assert(startByte < limitByte);
return null;
}
if (diffIndex < limit.remaining() - 1 || startByte + 1 < limitByte) {
start.put(diffIndex, (byte)((start.get(diffIndex) & 0xff) + 1));
start.limit(diffIndex + 1);
} else {
// v
// A A 1 A A A
// A A 2
//
// Incrementing the current byte will make start bigger than limit, we
// will skip this byte, and find the first non 0xFF byte in start and
// increment it.
diffIndex++;
private static int compare(final byte[] a, final byte[] b) {
return ByteBuffer.wrap(a).compareTo(ByteBuffer.wrap(b));
while (diffIndex < start.remaining()) {
// Keep moving until we find the first non 0xFF byte to
// increment it
if ((start.get(diffIndex) & 0xff) <
0xff) {
start.put(diffIndex, (byte)((start.get(diffIndex) & 0xff) + 1));
start.limit(diffIndex + 1);
break;
}
diffIndex++;
}
}
assert(compare(start.duplicate(), limit.duplicate()) < 0);
}
}
@Override
public String findShortSuccessor(final String key) {
final byte[] keyBytes = key.getBytes();
public void findShortSuccessor(final ByteBuffer key) {
// Find first character that can be incremented
final int n = keyBytes.length;
final int n = key.remaining();
for (int i = 0; i < n; i++) {
final byte byt = keyBytes[i];
final int byt = key.get(i) & 0xff;
if (byt != 0xff) {
final byte shortSuccessor[] = new byte[i + 1];
System.arraycopy(keyBytes, 0, shortSuccessor, 0, i + 1);
shortSuccessor[i]++;
return new String(shortSuccessor);
key.put(i, (byte)(byt + 1));
key.limit(i+1);
return;
}
}
// *key is a run of 0xffs. Leave it alone.
return null;
}
}
java/src/main/java/org/rocksdb/util/DirectBytewiseComparator.java 已删除 100644 → 0
浏览文件 @ 800d24dd
// Copyright (c) 2011-present, Facebook, Inc. All rights reserved.
// This source code is licensed under both the GPLv2 (found in the
// COPYING file in the root directory) and Apache 2.0 License
// (found in the LICENSE.Apache file in the root directory).
package org.rocksdb.util;
import org.rocksdb.ComparatorOptions;
import org.rocksdb.DirectComparator;
import org.rocksdb.DirectSlice;
import java.nio.ByteBuffer;
/**
* This is a Java Native implementation of the C++
* equivalent BytewiseComparatorImpl using {@link DirectSlice}
*
* The performance of Comparators implemented in Java is always
* less than their C++ counterparts due to the bridging overhead,
* as such you likely don't want to use this apart from benchmarking
* and you most likely instead wanted
* {@link org.rocksdb.BuiltinComparator#BYTEWISE_COMPARATOR}
*/
public class DirectBytewiseComparator extends DirectComparator {
public DirectBytewiseComparator(final ComparatorOptions copt) {
super(copt);
}
@Override
public String name() {
return "rocksdb.java.DirectBytewiseComparator";
}
@Override
public int compare(final DirectSlice a, final DirectSlice b) {
return a.data().compareTo(b.data());
}
@Override
public String findShortestSeparator(final String start,
final DirectSlice limit) {
final byte[] startBytes = start.getBytes();
// Find length of common prefix
final int min_length = Math.min(startBytes.length, limit.size());
int diff_index = 0;
while ((diff_index < min_length) &&
(startBytes[diff_index] == limit.get(diff_index))) {
diff_index++;
}
if (diff_index >= min_length) {
// Do not shorten if one string is a prefix of the other
} else {
final byte diff_byte = startBytes[diff_index];
if(diff_byte < 0xff && diff_byte + 1 < limit.get(diff_index)) {
final byte shortest[] = new byte[diff_index + 1];
System.arraycopy(startBytes, 0, shortest, 0, diff_index + 1);
shortest[diff_index]++;
assert(ByteBuffer.wrap(shortest).compareTo(limit.data()) < 0);
return new String(shortest);
}
}
return null;
}
@Override
public String findShortSuccessor(final String key) {
final byte[] keyBytes = key.getBytes();
// Find first character that can be incremented
final int n = keyBytes.length;
for (int i = 0; i < n; i++) {
final byte byt = keyBytes[i];
if (byt != 0xff) {
final byte shortSuccessor[] = new byte[i + 1];
System.arraycopy(keyBytes, 0, shortSuccessor, 0, i + 1);
shortSuccessor[i]++;
return new String(shortSuccessor);
}
}
// *key is a run of 0xffs. Leave it alone.
return null;
}
}
java/src/main/java/org/rocksdb/util/IntComparator.java 0 → 100644
浏览文件 @ 7242dae7
// Copyright (c) 2011-present, Facebook, Inc. All rights reserved.
// This source code is licensed under both the GPLv2 (found in the
// COPYING file in the root directory) and Apache 2.0 License
// (found in the LICENSE.Apache file in the root directory).
package org.rocksdb.util;
import org.rocksdb.AbstractComparator;
import org.rocksdb.ComparatorOptions;
import java.nio.ByteBuffer;
/**
* This is a Java implementation of a Comparator for Java int
* keys.
*
* This comparator assumes keys are (at least) four bytes, so
* the caller must guarantee that in accessing other APIs in
* combination with this comparator.
*
* The performance of Comparators implemented in Java is always
* less than their C++ counterparts due to the bridging overhead,
* as such you likely don't want to use this apart from benchmarking
* or testing.
*/
public final class IntComparator extends AbstractComparator {
public IntComparator(final ComparatorOptions copt) {
super(copt);
}
@Override
public String name() {
return "rocksdb.java.IntComparator";
}
@Override
public int compare(final ByteBuffer a, final ByteBuffer b) {
return compareIntKeys(a, b);
}
/**
* Compares integer keys
* so that they are in ascending order
*
* @param a 4-bytes representing an integer key
* @param b 4-bytes representing an integer key
*
* @return negative if a &lt; b, 0 if a == b, positive otherwise
*/
private final int compareIntKeys(final ByteBuffer a, final ByteBuffer b) {
final int iA = a.getInt();
final int iB = b.getInt();
// protect against int key calculation overflow
final long diff = (long)iA - iB;
final int result;
if (diff < Integer.MIN_VALUE) {
result = Integer.MIN_VALUE;
} else if(diff > Integer.MAX_VALUE) {
result = Integer.MAX_VALUE;
} else {
result = (int)diff;
}
return result;
}
}
java/src/main/java/org/rocksdb/util/ReverseBytewiseComparator.java
浏览文件 @ 7242dae7
......@@ -5,10 +5,13 @@
package org.rocksdb.util;
import org.rocksdb.AbstractComparator;
import org.rocksdb.BuiltinComparator;
import org.rocksdb.ComparatorOptions;
import org.rocksdb.Slice;
import java.nio.ByteBuffer;
/**
* This is a Java Native implementation of the C++
* equivalent ReverseBytewiseComparatorImpl using {@link Slice}
......@@ -19,7 +22,7 @@ import org.rocksdb.Slice;
* and you most likely instead wanted
* {@link BuiltinComparator#REVERSE_BYTEWISE_COMPARATOR}
*/
public class ReverseBytewiseComparator extends BytewiseComparator {
public final class ReverseBytewiseComparator extends AbstractComparator {
public ReverseBytewiseComparator(final ComparatorOptions copt) {
super(copt);
......@@ -31,7 +34,55 @@ public class ReverseBytewiseComparator extends BytewiseComparator {
}
@Override
public int compare(final Slice a, final Slice b) {
return -super.compare(a, b);
public int compare(final ByteBuffer a, final ByteBuffer b) {
return -BytewiseComparator._compare(a, b);
}
@Override
public void findShortestSeparator(final ByteBuffer start,
final ByteBuffer limit) {
// Find length of common prefix
final int minLength = Math.min(start.remaining(), limit.remaining());
int diffIndex = 0;
while (diffIndex < minLength &&
start.get(diffIndex) == limit.get(diffIndex)) {
diffIndex++;
}
assert(diffIndex <= minLength);
if (diffIndex == minLength) {
// Do not shorten if one string is a prefix of the other
//
// We could handle cases like:
// V
// A A 2 X Y
// A A 2
// in a similar way as BytewiseComparator::FindShortestSeparator().
// We keep it simple by not implementing it. We can come back to it
// later when needed.
} else {
final int startByte = start.get(diffIndex) & 0xff;
final int limitByte = limit.get(diffIndex) & 0xff;
if (startByte > limitByte && diffIndex < start.remaining() - 1) {
// Case like
// V
// A A 3 A A
// A A 1 B B
//
// or
// v
// A A 2 A A
// A A 1 B B
// In this case "AA2" will be good.
//#ifndef NDEBUG
// std::string old_start = *start;
//#endif
start.limit(diffIndex + 1);
//#ifndef NDEBUG
// assert(old_start >= *start);
//#endif
assert(BytewiseComparator._compare(start.duplicate(), limit.duplicate()) > 0);
}
}
}
}
java/src/test/java/org/rocksdb/AbstractComparatorTest.java 已删除 100644 → 0
浏览文件 @ 800d24dd
// Copyright (c) 2011-present, Facebook, Inc. All rights reserved.
// This source code is licensed under both the GPLv2 (found in the
// COPYING file in the root directory) and Apache 2.0 License
// (found in the LICENSE.Apache file in the root directory).
package org.rocksdb;
import java.io.IOException;
import java.nio.file.*;
import java.util.ArrayList;
import java.util.Arrays;
import java.util.List;
import java.util.Random;
import static org.assertj.core.api.Assertions.assertThat;
import static org.rocksdb.Types.byteToInt;
import static org.rocksdb.Types.intToByte;
/**
* Abstract tests for both Comparator and DirectComparator
*/
public abstract class AbstractComparatorTest<T extends AbstractSlice<?>> {
/**
* Get a comparator which will expect Integer keys
* and determine an ascending order
*
* @return An integer ascending order key comparator
*/
public abstract AbstractComparator<T> getAscendingIntKeyComparator();
/**
* Test which stores random keys into the database
* using an @see getAscendingIntKeyComparator
* it then checks that these keys are read back in
* ascending order
*
* @param db_path A path where we can store database
* files temporarily
*
* @throws java.io.IOException if IO error happens.
*/
public void testRoundtrip(final Path db_path) throws IOException,
RocksDBException {
try (final AbstractComparator<T> comparator = getAscendingIntKeyComparator();
final Options opt = new Options()
.setCreateIfMissing(true)
.setComparator(comparator)) {
// store 10,000 random integer keys
final int ITERATIONS = 10000;
try (final RocksDB db = RocksDB.open(opt, db_path.toString())) {
final Random random = new Random();
for (int i = 0; i < ITERATIONS; i++) {
final byte[] key = intToByte(random.nextInt());
// does key already exist (avoid duplicates)
if (i > 0 && db.get(key) != null) {
i--; // generate a different key
} else {
db.put(key, "value".getBytes());
}
}
}
// re-open db and read from start to end
// integer keys should be in ascending
// order as defined by SimpleIntComparator
try (final RocksDB db = RocksDB.open(opt, db_path.toString());
final RocksIterator it = db.newIterator()) {
it.seekToFirst();
int lastKey = Integer.MIN_VALUE;
int count = 0;
for (it.seekToFirst(); it.isValid(); it.next()) {
final int thisKey = byteToInt(it.key());
assertThat(thisKey).isGreaterThan(lastKey);
lastKey = thisKey;
count++;
}
assertThat(count).isEqualTo(ITERATIONS);
}
}
}
/**
* Test which stores random keys into a column family
* in the database
* using an @see getAscendingIntKeyComparator
* it then checks that these keys are read back in
* ascending order
*
* @param db_path A path where we can store database
* files temporarily
*
* @throws java.io.IOException if IO error happens.
*/
public void testRoundtripCf(final Path db_path) throws IOException,
RocksDBException {
try(final AbstractComparator<T> comparator = getAscendingIntKeyComparator()) {
final List<ColumnFamilyDescriptor> cfDescriptors = Arrays.asList(
new ColumnFamilyDescriptor(RocksDB.DEFAULT_COLUMN_FAMILY),
new ColumnFamilyDescriptor("new_cf".getBytes(),
new ColumnFamilyOptions().setComparator(comparator))
);
final List<ColumnFamilyHandle> cfHandles = new ArrayList<>();
try (final DBOptions opt = new DBOptions().
setCreateIfMissing(true).
setCreateMissingColumnFamilies(true)) {
// store 10,000 random integer keys
final int ITERATIONS = 10000;
try (final RocksDB db = RocksDB.open(opt, db_path.toString(),
cfDescriptors, cfHandles)) {
try {
assertThat(cfDescriptors.size()).isEqualTo(2);
assertThat(cfHandles.size()).isEqualTo(2);
final Random random = new Random();
for (int i = 0; i < ITERATIONS; i++) {
final byte key[] = intToByte(random.nextInt());
if (i > 0 && db.get(cfHandles.get(1), key) != null) {
// does key already exist (avoid duplicates)
i--; // generate a different key
} else {
db.put(cfHandles.get(1), key, "value".getBytes());
}
}
} finally {
for (final ColumnFamilyHandle handle : cfHandles) {
handle.close();
}
}
cfHandles.clear();
}
// re-open db and read from start to end
// integer keys should be in ascending
// order as defined by SimpleIntComparator
try (final RocksDB db = RocksDB.open(opt, db_path.toString(),
cfDescriptors, cfHandles);
final RocksIterator it = db.newIterator(cfHandles.get(1))) {
try {
assertThat(cfDescriptors.size()).isEqualTo(2);
assertThat(cfHandles.size()).isEqualTo(2);
it.seekToFirst();
int lastKey = Integer.MIN_VALUE;
int count = 0;
for (it.seekToFirst(); it.isValid(); it.next()) {
final int thisKey = byteToInt(it.key());
assertThat(thisKey).isGreaterThan(lastKey);
lastKey = thisKey;
count++;
}
assertThat(count).isEqualTo(ITERATIONS);
} finally {
for (final ColumnFamilyHandle handle : cfHandles) {
handle.close();
}
}
cfHandles.clear();
}
}
}
}
/**
* Compares integer keys
* so that they are in ascending order
*
* @param a 4-bytes representing an integer key
* @param b 4-bytes representing an integer key
*
* @return negative if a &lt; b, 0 if a == b, positive otherwise
*/
protected final int compareIntKeys(final byte[] a, final byte[] b) {
final int iA = byteToInt(a);
final int iB = byteToInt(b);
// protect against int key calculation overflow
final double diff = (double)iA - iB;
final int result;
if (diff < Integer.MIN_VALUE) {
result = Integer.MIN_VALUE;
} else if(diff > Integer.MAX_VALUE) {
result = Integer.MAX_VALUE;
} else {
result = (int)diff;
}
return result;
}
}
java/src/test/java/org/rocksdb/ComparatorTest.java → java/src/test/java/org/rocksdb/BuiltinComparatorTest.java
浏览文件 @ 7242dae7
......@@ -10,12 +10,9 @@ import org.junit.Rule;
import org.junit.Test;
import org.junit.rules.TemporaryFolder;
import java.io.IOException;
import java.nio.file.FileSystems;
import static org.assertj.core.api.Assertions.assertThat;
public class ComparatorTest {
public class BuiltinComparatorTest {
@ClassRule
public static final RocksNativeLibraryResource ROCKS_NATIVE_LIBRARY_RESOURCE =
......@@ -24,64 +21,12 @@ public class ComparatorTest {
@Rule
public TemporaryFolder dbFolder = new TemporaryFolder();
@Test
public void javaComparator() throws IOException, RocksDBException {
final AbstractComparatorTest<Slice> comparatorTest = new AbstractComparatorTest<Slice>() {
@Override
public AbstractComparator<Slice> getAscendingIntKeyComparator() {
return new Comparator(new ComparatorOptions()) {
@Override
public String name() {
return "test.AscendingIntKeyComparator";
}
@Override
public int compare(final Slice a, final Slice b) {
return compareIntKeys(a.data(), b.data());
}
};
}
};
// test the round-tripability of keys written and read with the Comparator
comparatorTest.testRoundtrip(FileSystems.getDefault().getPath(
dbFolder.getRoot().getAbsolutePath()));
}
@Test
public void javaComparatorCf() throws IOException, RocksDBException {
final AbstractComparatorTest<Slice> comparatorTest = new AbstractComparatorTest<Slice>() {
@Override
public AbstractComparator<Slice> getAscendingIntKeyComparator() {
return new Comparator(new ComparatorOptions()) {
@Override
public String name() {
return "test.AscendingIntKeyComparator";
}
@Override
public int compare(final Slice a, final Slice b) {
return compareIntKeys(a.data(), b.data());
}
};
}
};
// test the round-tripability of keys written and read with the Comparator
comparatorTest.testRoundtripCf(FileSystems.getDefault().getPath(
dbFolder.getRoot().getAbsolutePath()));
}
@Test
public void builtinForwardComparator()
throws RocksDBException {
try (final Options options = new Options()
.setCreateIfMissing(true)
.setComparator(BuiltinComparator.BYTEWISE_COMPARATOR);
.setCreateIfMissing(true)
.setComparator(BuiltinComparator.BYTEWISE_COMPARATOR);
final RocksDB rocksDb = RocksDB.open(options,
dbFolder.getRoot().getAbsolutePath())
) {
......@@ -133,8 +78,8 @@ public class ComparatorTest {
public void builtinReverseComparator()
throws RocksDBException {
try (final Options options = new Options()
.setCreateIfMissing(true)
.setComparator(BuiltinComparator.REVERSE_BYTEWISE_COMPARATOR);
.setCreateIfMissing(true)
.setComparator(BuiltinComparator.REVERSE_BYTEWISE_COMPARATOR);
final RocksDB rocksDb = RocksDB.open(options,
dbFolder.getRoot().getAbsolutePath())
) {
......
java/src/test/java/org/rocksdb/ComparatorOptionsTest.java
浏览文件 @ 7242dae7
......@@ -17,16 +17,42 @@ public class ComparatorOptionsTest {
new RocksNativeLibraryResource();
@Test
public void comparatorOptions() {
public void reusedSynchronisationType() {
try(final ComparatorOptions copt = new ComparatorOptions()) {
assertThat(copt).isNotNull();
// UseAdaptiveMutex test
copt.setUseAdaptiveMutex(true);
assertThat(copt.useAdaptiveMutex()).isTrue();
copt.setReusedSynchronisationType(ReusedSynchronisationType.MUTEX);
assertThat(copt.reusedSynchronisationType())
.isEqualTo(ReusedSynchronisationType.MUTEX);
copt.setUseAdaptiveMutex(false);
assertThat(copt.useAdaptiveMutex()).isFalse();
copt.setReusedSynchronisationType(ReusedSynchronisationType.ADAPTIVE_MUTEX);
assertThat(copt.reusedSynchronisationType())
.isEqualTo(ReusedSynchronisationType.ADAPTIVE_MUTEX);
copt.setReusedSynchronisationType(ReusedSynchronisationType.THREAD_LOCAL);
assertThat(copt.reusedSynchronisationType())
.isEqualTo(ReusedSynchronisationType.THREAD_LOCAL);
}
}
@Test
public void useDirectBuffer() {
try(final ComparatorOptions copt = new ComparatorOptions()) {
copt.setUseDirectBuffer(true);
assertThat(copt.useDirectBuffer()).isTrue();
copt.setUseDirectBuffer(false);
assertThat(copt.useDirectBuffer()).isFalse();
}
}
@Test
public void maxReusedBufferSize() {
try(final ComparatorOptions copt = new ComparatorOptions()) {
copt.setMaxReusedBufferSize(12345);
assertThat(copt.maxReusedBufferSize()).isEqualTo(12345);
copt.setMaxReusedBufferSize(-1);
assertThat(copt.maxReusedBufferSize()).isEqualTo(-1);
}
}
}
java/src/test/java/org/rocksdb/DirectComparatorTest.java 已删除 100644 → 0
浏览文件 @ 800d24dd
// Copyright (c) 2011-present, Facebook, Inc. All rights reserved.
// This source code is licensed under both the GPLv2 (found in the
// COPYING file in the root directory) and Apache 2.0 License
// (found in the LICENSE.Apache file in the root directory).
package org.rocksdb;
import org.junit.ClassRule;
import org.junit.Rule;
import org.junit.Test;
import org.junit.rules.TemporaryFolder;
import java.io.IOException;
import java.nio.file.FileSystems;
public class DirectComparatorTest {
@ClassRule
public static final RocksNativeLibraryResource ROCKS_NATIVE_LIBRARY_RESOURCE =
new RocksNativeLibraryResource();
@Rule
public TemporaryFolder dbFolder = new TemporaryFolder();
@Test
public void directComparator() throws IOException, RocksDBException {
final AbstractComparatorTest<DirectSlice> comparatorTest = new AbstractComparatorTest<DirectSlice>() {
@Override
public AbstractComparator<DirectSlice> getAscendingIntKeyComparator() {
return new DirectComparator(new ComparatorOptions()) {
@Override
public String name() {
return "test.AscendingIntKeyDirectComparator";
}
@Override
public int compare(final DirectSlice a, final DirectSlice b) {
final byte ax[] = new byte[4], bx[] = new byte[4];
a.data().get(ax);
b.data().get(bx);
return compareIntKeys(ax, bx);
}
};
}
};
// test the round-tripability of keys written and read with the DirectComparator
comparatorTest.testRoundtrip(FileSystems.getDefault().getPath(
dbFolder.getRoot().getAbsolutePath()));
}
}
java/src/test/java/org/rocksdb/OptimisticTransactionOptionsTest.java
浏览文件 @ 7242dae7
......@@ -6,7 +6,7 @@
package org.rocksdb;
import org.junit.Test;
import org.rocksdb.util.DirectBytewiseComparator;
import org.rocksdb.util.BytewiseComparator;
import java.util.Random;
......@@ -29,8 +29,9 @@ public class OptimisticTransactionOptionsTest {
@Test
public void comparator() {
try (final OptimisticTransactionOptions opt = new OptimisticTransactionOptions();
final ComparatorOptions copt = new ComparatorOptions();
final DirectComparator comparator = new DirectBytewiseComparator(copt)) {
final ComparatorOptions copt = new ComparatorOptions()
.setUseDirectBuffer(true);
final AbstractComparator comparator = new BytewiseComparator(copt)) {
opt.setComparator(comparator);
}
}
......
java/src/test/java/org/rocksdb/SstFileWriterTest.java
浏览文件 @ 7242dae7
......@@ -65,7 +65,7 @@ public class SstFileWriterTest {
ComparatorOptions comparatorOptions = null;
BytewiseComparator comparator = null;
if (useJavaBytewiseComparator) {
comparatorOptions = new ComparatorOptions();
comparatorOptions = new ComparatorOptions().setUseDirectBuffer(false);
comparator = new BytewiseComparator(comparatorOptions);
options.setComparator(comparator);
sstFileWriter = new SstFileWriter(envOptions, options);
......
java/src/test/java/org/rocksdb/WalFilterTest.java
浏览文件 @ 7242dae7
......@@ -16,6 +16,7 @@ import java.util.List;
import java.util.Map;
import static org.assertj.core.api.Assertions.assertThat;
import static org.rocksdb.util.ByteUtil.bytes;
import static org.rocksdb.util.TestUtil.*;
public class WalFilterTest {
......@@ -32,23 +33,23 @@ public class WalFilterTest {
// Create 3 batches with two keys each
final byte[][][] batchKeys = {
new byte[][] {
u("key1"),
u("key2")
bytes("key1"),
bytes("key2")
},
new byte[][] {
u("key3"),
u("key4")
bytes("key3"),
bytes("key4")
},
new byte[][] {
u("key5"),
u("key6")
bytes("key5"),
bytes("key6")
}
};
final List<ColumnFamilyDescriptor> cfDescriptors = Arrays.asList(
new ColumnFamilyDescriptor(RocksDB.DEFAULT_COLUMN_FAMILY),
new ColumnFamilyDescriptor(u("pikachu"))
new ColumnFamilyDescriptor(bytes("pikachu"))
);
final List<ColumnFamilyHandle> cfHandles = new ArrayList<>();
......
java/src/test/java/org/rocksdb/util/BytewiseComparatorIntTest.java 0 → 100644
浏览文件 @ 7242dae7
// Copyright (c) 2011-present, Facebook, Inc. All rights reserved.
// This source code is licensed under both the GPLv2 (found in the
// COPYING file in the root directory) and Apache 2.0 License
// (found in the LICENSE.Apache file in the root directory).
package org.rocksdb.util;
import org.junit.BeforeClass;
import org.junit.ClassRule;
import org.junit.Rule;
import org.junit.Test;
import org.junit.rules.TemporaryFolder;
import org.junit.runner.RunWith;
import org.junit.runners.Parameterized;
import org.junit.runners.Parameterized.Parameter;
import org.junit.runners.Parameterized.Parameters;
import org.rocksdb.*;
import java.nio.ByteBuffer;
import java.nio.file.FileSystems;
import java.nio.file.Path;
import java.util.ArrayList;
import java.util.Arrays;
import java.util.List;
import java.util.Random;
import static java.nio.charset.StandardCharsets.UTF_8;
import static org.assertj.core.api.Assertions.assertThat;
/**
* Similar to {@link IntComparatorTest}, but uses {@link BytewiseComparator}
* which ensures the correct ordering of positive integers.
*/
@RunWith(Parameterized.class)
public class BytewiseComparatorIntTest {
// test with 500 random positive integer keys
private static final int TOTAL_KEYS = 500;
private static final byte[][] keys = new byte[TOTAL_KEYS][4];
@BeforeClass
public static void prepareKeys() {
final ByteBuffer buf = ByteBuffer.allocate(4);
final Random random = new Random();
for (int i = 0; i < TOTAL_KEYS; i++) {
final int ri = random.nextInt() & Integer.MAX_VALUE; // the & ensures positive integer
buf.putInt(ri);
buf.flip();
final byte[] key = buf.array();
// does key already exist (avoid duplicates)
if (keyExists(key, i)) {
i--; // loop round and generate a different key
} else {
System.arraycopy(key, 0, keys[i], 0, 4);
}
}
}
private static boolean keyExists(final byte[] key, final int limit) {
for (int j = 0; j < limit; j++) {
if (Arrays.equals(key, keys[j])) {
return true;
}
}
return false;
}
@Parameters(name = "{0}")
public static Iterable<Object[]> parameters() {
return Arrays.asList(new Object[][] {
{ "non-direct_reused64_mutex", false, 64, ReusedSynchronisationType.MUTEX },
{ "direct_reused64_mutex", true, 64, ReusedSynchronisationType.MUTEX },
{ "non-direct_reused64_adaptive-mutex", false, 64, ReusedSynchronisationType.ADAPTIVE_MUTEX },
{ "direct_reused64_adaptive-mutex", true, 64, ReusedSynchronisationType.ADAPTIVE_MUTEX },
{ "non-direct_reused64_thread-local", false, 64, ReusedSynchronisationType.THREAD_LOCAL },
{ "direct_reused64_thread-local", true, 64, ReusedSynchronisationType.THREAD_LOCAL },
{ "non-direct_noreuse", false, -1, null },
{ "direct_noreuse", true, -1, null }
});
}
@Parameter(0)
public String name;
@Parameter(1)
public boolean useDirectBuffer;
@Parameter(2)
public int maxReusedBufferSize;
@Parameter(3)
public ReusedSynchronisationType reusedSynchronisationType;
@ClassRule
public static final RocksNativeLibraryResource ROCKS_NATIVE_LIBRARY_RESOURCE =
new RocksNativeLibraryResource();
@Rule
public TemporaryFolder dbFolder = new TemporaryFolder();
@Test
public void javaComparatorDefaultCf() throws RocksDBException {
try (final ComparatorOptions options = new ComparatorOptions()
.setUseDirectBuffer(useDirectBuffer)
.setMaxReusedBufferSize(maxReusedBufferSize)
// if reusedSynchronisationType == null we assume that maxReusedBufferSize <= 0 and so we just set ADAPTIVE_MUTEX, even though it won't be used
.setReusedSynchronisationType(reusedSynchronisationType == null ? ReusedSynchronisationType.ADAPTIVE_MUTEX : reusedSynchronisationType);
final BytewiseComparator comparator = new BytewiseComparator(options)) {
// test the round-tripability of keys written and read with the Comparator
testRoundtrip(FileSystems.getDefault().getPath(
dbFolder.getRoot().getAbsolutePath()), comparator);
}
}
@Test
public void javaComparatorNamedCf() throws RocksDBException {
try (final ComparatorOptions options = new ComparatorOptions()
.setUseDirectBuffer(useDirectBuffer)
.setMaxReusedBufferSize(maxReusedBufferSize)
// if reusedSynchronisationType == null we assume that maxReusedBufferSize <= 0 and so we just set ADAPTIVE_MUTEX, even though it won't be used
.setReusedSynchronisationType(reusedSynchronisationType == null ? ReusedSynchronisationType.ADAPTIVE_MUTEX : reusedSynchronisationType);
final BytewiseComparator comparator = new BytewiseComparator(options)) {
// test the round-tripability of keys written and read with the Comparator
testRoundtripCf(FileSystems.getDefault().getPath(
dbFolder.getRoot().getAbsolutePath()), comparator);
}
}
/**
* Test which stores random keys into the database
* using an {@link IntComparator}
* it then checks that these keys are read back in
* ascending order
*
* @param db_path A path where we can store database
* files temporarily
*
* @param comparator the comparator
*
* @throws RocksDBException if a database error happens.
*/
private void testRoundtrip(final Path db_path,
final AbstractComparator comparator) throws RocksDBException {
try (final Options opt = new Options()
.setCreateIfMissing(true)
.setComparator(comparator)) {
// store TOTAL_KEYS into the db
try (final RocksDB db = RocksDB.open(opt, db_path.toString())) {
for (int i = 0; i < TOTAL_KEYS; i++) {
db.put(keys[i], "value".getBytes(UTF_8));
}
}
// re-open db and read from start to end
// integer keys should be in ascending
// order as defined by IntComparator
final ByteBuffer key = ByteBuffer.allocate(4);
try (final RocksDB db = RocksDB.open(opt, db_path.toString());
final RocksIterator it = db.newIterator()) {
it.seekToFirst();
int lastKey = Integer.MIN_VALUE;
int count = 0;
for (it.seekToFirst(); it.isValid(); it.next()) {
key.put(it.key());
key.flip();
final int thisKey = key.getInt();
key.clear();
assertThat(thisKey).isGreaterThan(lastKey);
lastKey = thisKey;
count++;
}
assertThat(count).isEqualTo(TOTAL_KEYS);
}
}
}
/**
* Test which stores random keys into a column family
* in the database
* using an {@link IntComparator}
* it then checks that these keys are read back in
* ascending order
*
* @param db_path A path where we can store database
* files temporarily
*
* @param comparator the comparator
*
* @throws RocksDBException if a database error happens.
*/
private void testRoundtripCf(final Path db_path,
final AbstractComparator comparator) throws RocksDBException {
final List<ColumnFamilyDescriptor> cfDescriptors = Arrays.asList(
new ColumnFamilyDescriptor(RocksDB.DEFAULT_COLUMN_FAMILY),
new ColumnFamilyDescriptor("new_cf".getBytes(),
new ColumnFamilyOptions()
.setComparator(comparator))
);
final List<ColumnFamilyHandle> cfHandles = new ArrayList<>();
try (final DBOptions opt = new DBOptions()
.setCreateIfMissing(true)
.setCreateMissingColumnFamilies(true)) {
try (final RocksDB db = RocksDB.open(opt, db_path.toString(),
cfDescriptors, cfHandles)) {
try {
assertThat(cfDescriptors.size()).isEqualTo(2);
assertThat(cfHandles.size()).isEqualTo(2);
for (int i = 0; i < TOTAL_KEYS; i++) {
db.put(cfHandles.get(1), keys[i], "value".getBytes(UTF_8));
}
} finally {
for (final ColumnFamilyHandle cfHandle : cfHandles) {
cfHandle.close();
}
cfHandles.clear();
}
}
// re-open db and read from start to end
// integer keys should be in ascending
// order as defined by SimpleIntComparator
final ByteBuffer key = ByteBuffer.allocate(4);
try (final RocksDB db = RocksDB.open(opt, db_path.toString(),
cfDescriptors, cfHandles);
final RocksIterator it = db.newIterator(cfHandles.get(1))) {
try {
assertThat(cfDescriptors.size()).isEqualTo(2);
assertThat(cfHandles.size()).isEqualTo(2);
it.seekToFirst();
int lastKey = Integer.MIN_VALUE;
int count = 0;
for (it.seekToFirst(); it.isValid(); it.next()) {
key.put(it.key());
key.flip();
final int thisKey = key.getInt();
key.clear();
assertThat(thisKey).isGreaterThan(lastKey);
lastKey = thisKey;
count++;
}
assertThat(count).isEqualTo(TOTAL_KEYS);
} finally {
for (final ColumnFamilyHandle cfHandle : cfHandles) {
cfHandle.close();
}
cfHandles.clear();
for (final ColumnFamilyDescriptor cfDescriptor : cfDescriptors) {
cfDescriptor.getOptions().close();
}
}
}
}
}
}
java/src/test/java/org/rocksdb/util/BytewiseComparatorTest.java
浏览文件 @ 7242dae7
......@@ -5,20 +5,20 @@
package org.rocksdb.util;
import org.junit.ClassRule;
import org.junit.Rule;
import org.junit.Test;
import org.junit.rules.TemporaryFolder;
import org.rocksdb.*;
import org.rocksdb.Comparator;
import java.io.IOException;
import java.nio.charset.StandardCharsets;
import java.nio.file.FileVisitResult;
import java.nio.file.Files;
import java.nio.file.Path;
import java.nio.file.SimpleFileVisitor;
import java.nio.file.attribute.BasicFileAttributes;
import java.nio.ByteBuffer;
import java.nio.file.*;
import java.util.*;
import static java.nio.charset.StandardCharsets.UTF_8;
import static org.junit.Assert.*;
import static org.rocksdb.util.ByteUtil.bytes;
/**
* This is a direct port of various C++
......@@ -27,6 +27,13 @@ import static org.junit.Assert.*;
*/
public class BytewiseComparatorTest {
@ClassRule
public static final RocksNativeLibraryResource ROCKS_NATIVE_LIBRARY_RESOURCE =
new RocksNativeLibraryResource();
@Rule
public TemporaryFolder dbFolder = new TemporaryFolder();
private List<String> source_strings = Arrays.asList("b", "d", "f", "h", "j", "l");
private List<String> interleaving_strings = Arrays.asList("a", "b", "c", "d", "e", "f", "g", "h", "i", "j", "k", "l", "m");
......@@ -38,13 +45,15 @@ public class BytewiseComparatorTest {
public void java_vs_cpp_bytewiseComparator()
throws IOException, RocksDBException {
for(int rand_seed = 301; rand_seed < 306; rand_seed++) {
final Path dbDir = Files.createTempDirectory("comparator_db_test");
final Path dbDir =
FileSystems.getDefault().getPath(dbFolder.newFolder().getAbsolutePath());
try(final RocksDB db = openDatabase(dbDir,
BuiltinComparator.BYTEWISE_COMPARATOR)) {
final Random rnd = new Random(rand_seed);
try(final ComparatorOptions copt2 = new ComparatorOptions();
final Comparator comparator2 = new BytewiseComparator(copt2)) {
try(final ComparatorOptions copt2 = new ComparatorOptions()
.setUseDirectBuffer(false);
final AbstractComparator comparator2 = new BytewiseComparator(copt2)) {
final java.util.Comparator<String> jComparator = toJavaComparator(comparator2);
doRandomIterationTest(
db,
......@@ -53,8 +62,6 @@ public class BytewiseComparatorTest {
8, 100, 3
);
}
} finally {
removeData(dbDir);
}
}
}
......@@ -67,14 +74,17 @@ public class BytewiseComparatorTest {
public void java_vs_java_bytewiseComparator()
throws IOException, RocksDBException {
for(int rand_seed = 301; rand_seed < 306; rand_seed++) {
final Path dbDir = Files.createTempDirectory("comparator_db_test");
try(final ComparatorOptions copt = new ComparatorOptions();
final Comparator comparator = new BytewiseComparator(copt);
final Path dbDir =
FileSystems.getDefault().getPath(dbFolder.newFolder().getAbsolutePath());
try(final ComparatorOptions copt = new ComparatorOptions()
.setUseDirectBuffer(false);
final AbstractComparator comparator = new BytewiseComparator(copt);
final RocksDB db = openDatabase(dbDir, comparator)) {
final Random rnd = new Random(rand_seed);
try(final ComparatorOptions copt2 = new ComparatorOptions();
final Comparator comparator2 = new BytewiseComparator(copt2)) {
try(final ComparatorOptions copt2 = new ComparatorOptions()
.setUseDirectBuffer(false);
final AbstractComparator comparator2 = new BytewiseComparator(copt2)) {
final java.util.Comparator<String> jComparator = toJavaComparator(comparator2);
doRandomIterationTest(
db,
......@@ -83,8 +93,6 @@ public class BytewiseComparatorTest {
8, 100, 3
);
}
} finally {
removeData(dbDir);
}
}
}
......@@ -97,13 +105,15 @@ public class BytewiseComparatorTest {
public void java_vs_cpp_directBytewiseComparator()
throws IOException, RocksDBException {
for(int rand_seed = 301; rand_seed < 306; rand_seed++) {
final Path dbDir = Files.createTempDirectory("comparator_db_test");
final Path dbDir =
FileSystems.getDefault().getPath(dbFolder.newFolder().getAbsolutePath());
try(final RocksDB db = openDatabase(dbDir,
BuiltinComparator.BYTEWISE_COMPARATOR)) {
final Random rnd = new Random(rand_seed);
try(final ComparatorOptions copt2 = new ComparatorOptions();
final DirectComparator comparator2 = new DirectBytewiseComparator(copt2)) {
try(final ComparatorOptions copt2 = new ComparatorOptions()
.setUseDirectBuffer(true);
final AbstractComparator comparator2 = new BytewiseComparator(copt2)) {
final java.util.Comparator<String> jComparator = toJavaComparator(comparator2);
doRandomIterationTest(
db,
......@@ -112,8 +122,6 @@ public class BytewiseComparatorTest {
8, 100, 3
);
}
} finally {
removeData(dbDir);
}
}
}
......@@ -126,14 +134,17 @@ public class BytewiseComparatorTest {
public void java_vs_java_directBytewiseComparator()
throws IOException, RocksDBException {
for(int rand_seed = 301; rand_seed < 306; rand_seed++) {
final Path dbDir = Files.createTempDirectory("comparator_db_test");
try (final ComparatorOptions copt = new ComparatorOptions();
final DirectComparator comparator = new DirectBytewiseComparator(copt);
final Path dbDir =
FileSystems.getDefault().getPath(dbFolder.newFolder().getAbsolutePath());
try (final ComparatorOptions copt = new ComparatorOptions()
.setUseDirectBuffer(true);
final AbstractComparator comparator = new BytewiseComparator(copt);
final RocksDB db = openDatabase(dbDir, comparator)) {
final Random rnd = new Random(rand_seed);
try(final ComparatorOptions copt2 = new ComparatorOptions();
final DirectComparator comparator2 = new DirectBytewiseComparator(copt2)) {
try(final ComparatorOptions copt2 = new ComparatorOptions()
.setUseDirectBuffer(true);
final AbstractComparator comparator2 = new BytewiseComparator(copt2)) {
final java.util.Comparator<String> jComparator = toJavaComparator(comparator2);
doRandomIterationTest(
db,
......@@ -142,8 +153,6 @@ public class BytewiseComparatorTest {
8, 100, 3
);
}
} finally {
removeData(dbDir);
}
}
}
......@@ -156,13 +165,15 @@ public class BytewiseComparatorTest {
public void java_vs_cpp_reverseBytewiseComparator()
throws IOException, RocksDBException {
for(int rand_seed = 301; rand_seed < 306; rand_seed++) {
final Path dbDir = Files.createTempDirectory("comparator_db_test");
final Path dbDir =
FileSystems.getDefault().getPath(dbFolder.newFolder().getAbsolutePath());
try(final RocksDB db = openDatabase(dbDir,
BuiltinComparator.REVERSE_BYTEWISE_COMPARATOR)) {
final Random rnd = new Random(rand_seed);
try(final ComparatorOptions copt2 = new ComparatorOptions();
final Comparator comparator2 = new ReverseBytewiseComparator(copt2)) {
try(final ComparatorOptions copt2 = new ComparatorOptions()
.setUseDirectBuffer(false);
final AbstractComparator comparator2 = new ReverseBytewiseComparator(copt2)) {
final java.util.Comparator<String> jComparator = toJavaComparator(comparator2);
doRandomIterationTest(
db,
......@@ -171,8 +182,6 @@ public class BytewiseComparatorTest {
8, 100, 3
);
}
} finally {
removeData(dbDir);
}
}
}
......@@ -185,14 +194,17 @@ public class BytewiseComparatorTest {
public void java_vs_java_reverseBytewiseComparator()
throws IOException, RocksDBException {
for(int rand_seed = 301; rand_seed < 306; rand_seed++) {
final Path dbDir = Files.createTempDirectory("comparator_db_test");
try (final ComparatorOptions copt = new ComparatorOptions();
final Comparator comparator = new ReverseBytewiseComparator(copt);
final Path dbDir =
FileSystems.getDefault().getPath(dbFolder.newFolder().getAbsolutePath());
try (final ComparatorOptions copt = new ComparatorOptions()
.setUseDirectBuffer(false);
final AbstractComparator comparator = new ReverseBytewiseComparator(copt);
final RocksDB db = openDatabase(dbDir, comparator)) {
final Random rnd = new Random(rand_seed);
try(final ComparatorOptions copt2 = new ComparatorOptions();
final Comparator comparator2 = new ReverseBytewiseComparator(copt2)) {
try(final ComparatorOptions copt2 = new ComparatorOptions()
.setUseDirectBuffer(false);
final AbstractComparator comparator2 = new ReverseBytewiseComparator(copt2)) {
final java.util.Comparator<String> jComparator = toJavaComparator(comparator2);
doRandomIterationTest(
db,
......@@ -201,8 +213,6 @@ public class BytewiseComparatorTest {
8, 100, 3
);
}
} finally {
removeData(dbDir);
}
}
}
......@@ -215,34 +225,38 @@ public class BytewiseComparatorTest {
final TreeMap<String, String> map = new TreeMap<>(javaComparator);
for (int i = 0; i < num_writes; i++) {
if (num_trigger_flush > 0 && i != 0 && i % num_trigger_flush == 0) {
db.flush(new FlushOptions());
}
try (final FlushOptions flushOptions = new FlushOptions();
final WriteOptions writeOptions = new WriteOptions()) {
for (int i = 0; i < num_writes; i++) {
if (num_trigger_flush > 0 && i != 0 && i % num_trigger_flush == 0) {
db.flush(flushOptions);
}
final int type = rnd.nextInt(2);
final int index = rnd.nextInt(source_strings.size());
final String key = source_strings.get(index);
switch (type) {
case 0:
// put
map.put(key, key);
db.put(new WriteOptions(), bytes(key), bytes(key));
break;
case 1:
// delete
if (map.containsKey(key)) {
map.remove(key);
}
db.delete(new WriteOptions(), bytes(key));
break;
final int type = rnd.nextInt(2);
final int index = rnd.nextInt(source_strings.size());
final String key = source_strings.get(index);
switch (type) {
case 0:
// put
map.put(key, key);
db.put(writeOptions, bytes(key), bytes(key));
break;
case 1:
// delete
if (map.containsKey(key)) {
map.remove(key);
}
db.delete(writeOptions, bytes(key));
break;
default:
fail("Should not be able to generate random outside range 1..2");
default:
fail("Should not be able to generate random outside range 1..2");
}
}
}
try(final RocksIterator iter = db.newIterator(new ReadOptions())) {
try (final ReadOptions readOptions = new ReadOptions();
final RocksIterator iter = db.newIterator(readOptions)) {
final KVIter<String, String> result_iter = new KVIter<>(map);
boolean is_valid = false;
......@@ -300,7 +314,7 @@ public class BytewiseComparatorTest {
assert (type == 6);
final int key_idx = rnd.nextInt(source_strings.size());
final String key = source_strings.get(key_idx);
final byte[] result = db.get(new ReadOptions(), bytes(key));
final byte[] result = db.get(readOptions, bytes(key));
if (!map.containsKey(key)) {
assertNull(result);
} else {
......@@ -342,7 +356,7 @@ public class BytewiseComparatorTest {
*/
private RocksDB openDatabase(
final Path dbDir,
final AbstractComparator<? extends AbstractSlice<?>> javaComparator)
final AbstractComparator javaComparator)
throws IOException, RocksDBException {
final Options options = new Options()
.setCreateIfMissing(true)
......@@ -350,50 +364,25 @@ public class BytewiseComparatorTest {
return RocksDB.open(options, dbDir.toAbsolutePath().toString());
}
private void closeDatabase(final RocksDB db) {
db.close();
}
private void removeData(final Path dbDir) throws IOException {
Files.walkFileTree(dbDir, new SimpleFileVisitor<Path>() {
@Override
public FileVisitResult visitFile(
final Path file, final BasicFileAttributes attrs)
throws IOException {
Files.delete(file);
return FileVisitResult.CONTINUE;
}
@Override
public FileVisitResult postVisitDirectory(
final Path dir, final IOException exc) throws IOException {
Files.delete(dir);
return FileVisitResult.CONTINUE;
}
});
}
private byte[] bytes(final String s) {
return s.getBytes(StandardCharsets.UTF_8);
}
private java.util.Comparator<String> toJavaComparator(
final Comparator rocksComparator) {
return new java.util.Comparator<String>() {
@Override
public int compare(final String s1, final String s2) {
return rocksComparator.compare(new Slice(s1), new Slice(s2));
}
};
}
private java.util.Comparator<String> toJavaComparator(
final DirectComparator rocksComparator) {
final AbstractComparator rocksComparator) {
return new java.util.Comparator<String>() {
@Override
public int compare(final String s1, final String s2) {
return rocksComparator.compare(new DirectSlice(s1),
new DirectSlice(s2));
final ByteBuffer bufS1;
final ByteBuffer bufS2;
if (rocksComparator.usingDirectBuffers()) {
bufS1 = ByteBuffer.allocateDirect(s1.length());
bufS2 = ByteBuffer.allocateDirect(s2.length());
} else {
bufS1 = ByteBuffer.allocate(s1.length());
bufS2 = ByteBuffer.allocate(s2.length());
}
bufS1.put(bytes(s1));
bufS1.flip();
bufS2.put(bytes(s2));
bufS2.flip();
return rocksComparator.compare(bufS1, bufS2);
}
};
}
......@@ -434,7 +423,7 @@ public class BytewiseComparatorTest {
public void seek(final byte[] target) {
for(offset = 0; offset < entries.size(); offset++) {
if(comparator.compare(entries.get(offset).getKey(),
(K)new String(target, StandardCharsets.UTF_8)) >= 0) {
(K)new String(target, UTF_8)) >= 0) {
return;
}
}
......@@ -445,7 +434,7 @@ public class BytewiseComparatorTest {
public void seekForPrev(final byte[] target) {
for(offset = entries.size()-1; offset >= 0; offset--) {
if(comparator.compare(entries.get(offset).getKey(),
(K)new String(target, StandardCharsets.UTF_8)) <= 0) {
(K)new String(target, UTF_8)) <= 0) {
return;
}
}
......
java/src/test/java/org/rocksdb/util/IntComparatorTest.java 0 → 100644
浏览文件 @ 7242dae7
// Copyright (c) 2011-present, Facebook, Inc. All rights reserved.
// This source code is licensed under both the GPLv2 (found in the
// COPYING file in the root directory) and Apache 2.0 License
// (found in the LICENSE.Apache file in the root directory).
package org.rocksdb.util;
import org.junit.BeforeClass;
import org.junit.ClassRule;
import org.junit.Rule;
import org.junit.Test;
import org.junit.rules.TemporaryFolder;
import org.junit.runner.RunWith;
import org.junit.runners.Parameterized;
import org.junit.runners.Parameterized.Parameter;
import org.junit.runners.Parameterized.Parameters;
import org.rocksdb.*;
import java.nio.ByteBuffer;
import java.nio.file.*;
import java.util.ArrayList;
import java.util.Arrays;
import java.util.List;
import java.util.Random;
import static java.nio.charset.StandardCharsets.UTF_8;
import static org.assertj.core.api.Assertions.assertThat;
/**
* Tests for IntComparator, but more generally
* also for rocksdb::ComparatorJniCallback implementation.
*/
@RunWith(Parameterized.class)
public class IntComparatorTest {
// test with 500 random integer keys
private static final int TOTAL_KEYS = 500;
private static final byte[][] keys = new byte[TOTAL_KEYS][4];
@BeforeClass
public static void prepareKeys() {
final ByteBuffer buf = ByteBuffer.allocate(4);
final Random random = new Random();
for (int i = 0; i < TOTAL_KEYS; i++) {
final int ri = random.nextInt();
buf.putInt(ri);
buf.flip();
final byte[] key = buf.array();
// does key already exist (avoid duplicates)
if (keyExists(key, i)) {
i--; // loop round and generate a different key
} else {
System.arraycopy(key, 0, keys[i], 0, 4);
}
}
}
private static boolean keyExists(final byte[] key, final int limit) {
for (int j = 0; j < limit; j++) {
if (Arrays.equals(key, keys[j])) {
return true;
}
}
return false;
}
@Parameters(name = "{0}")
public static Iterable<Object[]> parameters() {
return Arrays.asList(new Object[][] {
{ "non-direct_reused64_mutex", false, 64, ReusedSynchronisationType.MUTEX },
{ "direct_reused64_mutex", true, 64, ReusedSynchronisationType.MUTEX },
{ "non-direct_reused64_adaptive-mutex", false, 64, ReusedSynchronisationType.ADAPTIVE_MUTEX },
{ "direct_reused64_adaptive-mutex", true, 64, ReusedSynchronisationType.ADAPTIVE_MUTEX },
{ "non-direct_reused64_thread-local", false, 64, ReusedSynchronisationType.THREAD_LOCAL },
{ "direct_reused64_thread-local", true, 64, ReusedSynchronisationType.THREAD_LOCAL },
{ "non-direct_noreuse", false, -1, null },
{ "direct_noreuse", true, -1, null }
});
}
@Parameter(0)
public String name;
@Parameter(1)
public boolean useDirectBuffer;
@Parameter(2)
public int maxReusedBufferSize;
@Parameter(3)
public ReusedSynchronisationType reusedSynchronisationType;
@ClassRule
public static final RocksNativeLibraryResource ROCKS_NATIVE_LIBRARY_RESOURCE =
new RocksNativeLibraryResource();
@Rule
public TemporaryFolder dbFolder = new TemporaryFolder();
@Test
public void javaComparatorDefaultCf() throws RocksDBException {
try (final ComparatorOptions options = new ComparatorOptions()
.setUseDirectBuffer(useDirectBuffer)
.setMaxReusedBufferSize(maxReusedBufferSize)
// if reusedSynchronisationType == null we assume that maxReusedBufferSize <= 0 and so we just set ADAPTIVE_MUTEX, even though it won't be used
.setReusedSynchronisationType(reusedSynchronisationType == null ? ReusedSynchronisationType.ADAPTIVE_MUTEX : reusedSynchronisationType);
final IntComparator comparator = new IntComparator(options)) {
// test the round-tripability of keys written and read with the Comparator
testRoundtrip(FileSystems.getDefault().getPath(
dbFolder.getRoot().getAbsolutePath()), comparator);
}
}
@Test
public void javaComparatorNamedCf() throws RocksDBException {
try (final ComparatorOptions options = new ComparatorOptions()
.setUseDirectBuffer(useDirectBuffer)
.setMaxReusedBufferSize(maxReusedBufferSize)
// if reusedSynchronisationType == null we assume that maxReusedBufferSize <= 0 and so we just set ADAPTIVE_MUTEX, even though it won't be used
.setReusedSynchronisationType(reusedSynchronisationType == null ? ReusedSynchronisationType.ADAPTIVE_MUTEX : reusedSynchronisationType);
final IntComparator comparator = new IntComparator(options)) {
// test the round-tripability of keys written and read with the Comparator
testRoundtripCf(FileSystems.getDefault().getPath(
dbFolder.getRoot().getAbsolutePath()), comparator);
}
}
/**
* Test which stores random keys into the database
* using an {@link IntComparator}
* it then checks that these keys are read back in
* ascending order
*
* @param db_path A path where we can store database
* files temporarily
*
* @param comparator the comparator
*
* @throws RocksDBException if a database error happens.
*/
private void testRoundtrip(final Path db_path,
final AbstractComparator comparator) throws RocksDBException {
try (final Options opt = new Options()
.setCreateIfMissing(true)
.setComparator(comparator)) {
// store TOTAL_KEYS into the db
try (final RocksDB db = RocksDB.open(opt, db_path.toString())) {
for (int i = 0; i < TOTAL_KEYS; i++) {
db.put(keys[i], "value".getBytes(UTF_8));
}
}
// re-open db and read from start to end
// integer keys should be in ascending
// order as defined by IntComparator
final ByteBuffer key = ByteBuffer.allocate(4);
try (final RocksDB db = RocksDB.open(opt, db_path.toString());
final RocksIterator it = db.newIterator()) {
it.seekToFirst();
int lastKey = Integer.MIN_VALUE;
int count = 0;
for (it.seekToFirst(); it.isValid(); it.next()) {
key.put(it.key());
key.flip();
final int thisKey = key.getInt();
key.clear();
assertThat(thisKey).isGreaterThan(lastKey);
lastKey = thisKey;
count++;
}
assertThat(count).isEqualTo(TOTAL_KEYS);
}
}
}
/**
* Test which stores random keys into a column family
* in the database
* using an {@link IntComparator}
* it then checks that these keys are read back in
* ascending order
*
* @param db_path A path where we can store database
* files temporarily
*
* @param comparator the comparator
*
* @throws RocksDBException if a database error happens.
*/
private void testRoundtripCf(final Path db_path,
final AbstractComparator comparator) throws RocksDBException {
final List<ColumnFamilyDescriptor> cfDescriptors = Arrays.asList(
new ColumnFamilyDescriptor(RocksDB.DEFAULT_COLUMN_FAMILY),
new ColumnFamilyDescriptor("new_cf".getBytes(),
new ColumnFamilyOptions()
.setComparator(comparator))
);
final List<ColumnFamilyHandle> cfHandles = new ArrayList<>();
try (final DBOptions opt = new DBOptions()
.setCreateIfMissing(true)
.setCreateMissingColumnFamilies(true)) {
try (final RocksDB db = RocksDB.open(opt, db_path.toString(),
cfDescriptors, cfHandles)) {
try {
assertThat(cfDescriptors.size()).isEqualTo(2);
assertThat(cfHandles.size()).isEqualTo(2);
for (int i = 0; i < TOTAL_KEYS; i++) {
db.put(cfHandles.get(1), keys[i], "value".getBytes(UTF_8));
}
} finally {
for (final ColumnFamilyHandle cfHandle : cfHandles) {
cfHandle.close();
}
cfHandles.clear();
}
}
// re-open db and read from start to end
// integer keys should be in ascending
// order as defined by SimpleIntComparator
final ByteBuffer key = ByteBuffer.allocate(4);
try (final RocksDB db = RocksDB.open(opt, db_path.toString(),
cfDescriptors, cfHandles);
final RocksIterator it = db.newIterator(cfHandles.get(1))) {
try {
assertThat(cfDescriptors.size()).isEqualTo(2);
assertThat(cfHandles.size()).isEqualTo(2);
it.seekToFirst();
int lastKey = Integer.MIN_VALUE;
int count = 0;
for (it.seekToFirst(); it.isValid(); it.next()) {
key.put(it.key());
key.flip();
final int thisKey = key.getInt();
key.clear();
assertThat(thisKey).isGreaterThan(lastKey);
lastKey = thisKey;
count++;
}
assertThat(count).isEqualTo(TOTAL_KEYS);
} finally {
for (final ColumnFamilyHandle cfHandle : cfHandles) {
cfHandle.close();
}
cfHandles.clear();
for (final ColumnFamilyDescriptor cfDescriptor : cfDescriptors) {
cfDescriptor.getOptions().close();
}
}
}
}
}
}
java/src/test/java/org/rocksdb/util/JNIComparatorTest.java 0 → 100644
浏览文件 @ 7242dae7
package org.rocksdb.util;
import org.junit.ClassRule;
import org.junit.Rule;
import org.junit.Test;
import org.junit.rules.TemporaryFolder;
import org.junit.runner.RunWith;
import org.junit.runners.Parameterized;
import org.junit.runners.Parameterized.Parameter;
import org.junit.runners.Parameterized.Parameters;
import org.rocksdb.*;
import java.io.IOException;
import java.nio.ByteBuffer;
import java.nio.file.*;
import java.util.Arrays;
import static org.assertj.core.api.Assertions.assertThat;
@RunWith(Parameterized.class)
public class JNIComparatorTest {
@Parameters(name = "{0}")
public static Iterable<Object[]> parameters() {
return Arrays.asList(new Object[][] {
{ "bytewise_non-direct", BuiltinComparator.BYTEWISE_COMPARATOR, false },
{ "bytewise_direct", BuiltinComparator.BYTEWISE_COMPARATOR, true },
{ "reverse-bytewise_non-direct", BuiltinComparator.REVERSE_BYTEWISE_COMPARATOR, false },
{ "reverse-bytewise_direct", BuiltinComparator.REVERSE_BYTEWISE_COMPARATOR, true },
});
}
@Parameter(0)
public String name;
@Parameter(1)
public BuiltinComparator builtinComparator;
@Parameter(2)
public boolean useDirectBuffer;
@ClassRule
public static final RocksNativeLibraryResource ROCKS_NATIVE_LIBRARY_RESOURCE =
new RocksNativeLibraryResource();
@Rule
public TemporaryFolder dbFolder = new TemporaryFolder();
private static final int MIN = Short.MIN_VALUE - 1;
private static final int MAX = Short.MAX_VALUE + 1;
@Test
public void java_comparator_equals_cpp_comparator() throws RocksDBException, IOException {
final int[] javaKeys;
try (final ComparatorOptions comparatorOptions = new ComparatorOptions();
final AbstractComparator comparator = builtinComparator == BuiltinComparator.BYTEWISE_COMPARATOR
? new BytewiseComparator(comparatorOptions)
: new ReverseBytewiseComparator(comparatorOptions)) {
final Path javaDbDir =
FileSystems.getDefault().getPath(dbFolder.newFolder().getAbsolutePath());
storeWithJavaComparator(javaDbDir, comparator);
javaKeys = readAllWithJavaComparator(javaDbDir, comparator);
}
final Path cppDbDir =
FileSystems.getDefault().getPath(dbFolder.newFolder().getAbsolutePath());
storeWithCppComparator(cppDbDir, builtinComparator);
final int[] cppKeys =
readAllWithCppComparator(cppDbDir, builtinComparator);
assertThat(javaKeys).isEqualTo(cppKeys);
}
private void storeWithJavaComparator(final Path dir,
final AbstractComparator comparator) throws RocksDBException {
final ByteBuffer buf = ByteBuffer.allocate(4);
try (final Options options = new Options()
.setCreateIfMissing(true)
.setComparator(comparator);
final RocksDB db =
RocksDB.open(options, dir.toAbsolutePath().toString())) {
for (int i = MIN; i < MAX; i++) {
buf.putInt(i);
buf.flip();
db.put(buf.array(), buf.array());
buf.clear();
}
}
}
private void storeWithCppComparator(final Path dir,
final BuiltinComparator builtinComparator) throws RocksDBException {
try (final Options options = new Options()
.setCreateIfMissing(true)
.setComparator(builtinComparator);
final RocksDB db =
RocksDB.open(options, dir.toAbsolutePath().toString())) {
final ByteBuffer buf = ByteBuffer.allocate(4);
for (int i = MIN; i < MAX; i++) {
buf.putInt(i);
buf.flip();
db.put(buf.array(), buf.array());
buf.clear();
}
}
}
private int[] readAllWithJavaComparator(final Path dir,
final AbstractComparator comparator) throws RocksDBException {
try (final Options options = new Options()
.setCreateIfMissing(true)
.setComparator(comparator);
final RocksDB db =
RocksDB.open(options, dir.toAbsolutePath().toString())) {
try (final RocksIterator it = db.newIterator()) {
it.seekToFirst();
final ByteBuffer buf = ByteBuffer.allocate(4);
final int[] keys = new int[MAX - MIN];
int idx = 0;
while (it.isValid()) {
buf.put(it.key());
buf.flip();
final int thisKey = buf.getInt();
keys[idx++] = thisKey;
buf.clear();
it.next();
}
return keys;
}
}
}
private int[] readAllWithCppComparator(final Path dir,
final BuiltinComparator comparator) throws RocksDBException {
try (final Options options = new Options()
.setCreateIfMissing(true)
.setComparator(comparator);
final RocksDB db =
RocksDB.open(options, dir.toAbsolutePath().toString())) {
try (final RocksIterator it = db.newIterator()) {
it.seekToFirst();
final ByteBuffer buf = ByteBuffer.allocate(4);
final int[] keys = new int[MAX - MIN];
int idx = 0;
while (it.isValid()) {
buf.put(it.key());
buf.flip();
final int thisKey = buf.getInt();
keys[idx++] = thisKey;
buf.clear();
it.next();
}
return keys;
}
}
}
}
java/src/test/java/org/rocksdb/util/ReverseBytewiseComparatorIntTest.java 0 → 100644
浏览文件 @ 7242dae7
// Copyright (c) 2011-present, Facebook, Inc. All rights reserved.
// This source code is licensed under both the GPLv2 (found in the
// COPYING file in the root directory) and Apache 2.0 License
// (found in the LICENSE.Apache file in the root directory).
package org.rocksdb.util;
import org.junit.BeforeClass;
import org.junit.ClassRule;
import org.junit.Rule;
import org.junit.Test;
import org.junit.rules.TemporaryFolder;
import org.junit.runner.RunWith;
import org.junit.runners.Parameterized;
import org.junit.runners.Parameterized.Parameter;
import org.junit.runners.Parameterized.Parameters;
import org.rocksdb.*;
import java.nio.ByteBuffer;
import java.nio.file.FileSystems;
import java.nio.file.Path;
import java.util.ArrayList;
import java.util.Arrays;
import java.util.List;
import java.util.Random;
import static java.nio.charset.StandardCharsets.UTF_8;
import static org.assertj.core.api.Assertions.assertThat;
/**
* Similar to {@link IntComparatorTest}, but uses
* {@link ReverseBytewiseComparator} which ensures the correct reverse
* ordering of positive integers.
*/
@RunWith(Parameterized.class)
public class ReverseBytewiseComparatorIntTest {
// test with 500 random positive integer keys
private static final int TOTAL_KEYS = 500;
private static final byte[][] keys = new byte[TOTAL_KEYS][4];
@BeforeClass
public static void prepareKeys() {
final ByteBuffer buf = ByteBuffer.allocate(4);
final Random random = new Random();
for (int i = 0; i < TOTAL_KEYS; i++) {
final int ri = random.nextInt() & Integer.MAX_VALUE; // the & ensures positive integer
buf.putInt(ri);
buf.flip();
final byte[] key = buf.array();
// does key already exist (avoid duplicates)
if (keyExists(key, i)) {
i--; // loop round and generate a different key
} else {
System.arraycopy(key, 0, keys[i], 0, 4);
}
}
}
private static boolean keyExists(final byte[] key, final int limit) {
for (int j = 0; j < limit; j++) {
if (Arrays.equals(key, keys[j])) {
return true;
}
}
return false;
}
@Parameters(name = "{0}")
public static Iterable<Object[]> parameters() {
return Arrays.asList(new Object[][] {
{ "non-direct_reused64_mutex", false, 64, ReusedSynchronisationType.MUTEX },
{ "direct_reused64_adaptive-mutex", true, 64, ReusedSynchronisationType.MUTEX },
{ "non-direct_reused64_adaptive-mutex", false, 64, ReusedSynchronisationType.ADAPTIVE_MUTEX },
{ "direct_reused64_adaptive-mutex", true, 64, ReusedSynchronisationType.ADAPTIVE_MUTEX },
{ "non-direct_reused64_adaptive-mutex", false, 64, ReusedSynchronisationType.THREAD_LOCAL },
{ "direct_reused64_adaptive-mutex", true, 64, ReusedSynchronisationType.THREAD_LOCAL },
{ "non-direct_noreuse", false, -1, null },
{ "direct_noreuse", true, -1, null }
});
}
@Parameter(0)
public String name;
@Parameter(1)
public boolean useDirectBuffer;
@Parameter(2)
public int maxReusedBufferSize;
@Parameter(3)
public ReusedSynchronisationType reusedSynchronisationType;
@ClassRule
public static final RocksNativeLibraryResource ROCKS_NATIVE_LIBRARY_RESOURCE =
new RocksNativeLibraryResource();
@Rule
public TemporaryFolder dbFolder = new TemporaryFolder();
@Test
public void javaComparatorDefaultCf() throws RocksDBException {
try (final ComparatorOptions options = new ComparatorOptions()
.setUseDirectBuffer(useDirectBuffer)
.setMaxReusedBufferSize(maxReusedBufferSize)
// if reusedSynchronisationType == null we assume that maxReusedBufferSize <= 0 and so we just set ADAPTIVE_MUTEX, even though it won't be used
.setReusedSynchronisationType(reusedSynchronisationType == null ? ReusedSynchronisationType.ADAPTIVE_MUTEX : reusedSynchronisationType);
final ReverseBytewiseComparator comparator =
new ReverseBytewiseComparator(options)) {
// test the round-tripability of keys written and read with the Comparator
testRoundtrip(FileSystems.getDefault().getPath(
dbFolder.getRoot().getAbsolutePath()), comparator);
}
}
@Test
public void javaComparatorNamedCf() throws RocksDBException {
try (final ComparatorOptions options = new ComparatorOptions()
.setUseDirectBuffer(useDirectBuffer)
.setMaxReusedBufferSize(maxReusedBufferSize)
// if reusedSynchronisationType == null we assume that maxReusedBufferSize <= 0 and so we just set ADAPTIVE_MUTEX, even though it won't be used
.setReusedSynchronisationType(reusedSynchronisationType == null ? ReusedSynchronisationType.ADAPTIVE_MUTEX : reusedSynchronisationType);
final ReverseBytewiseComparator comparator
= new ReverseBytewiseComparator(options)) {
// test the round-tripability of keys written and read with the Comparator
testRoundtripCf(FileSystems.getDefault().getPath(
dbFolder.getRoot().getAbsolutePath()), comparator);
}
}
/**
* Test which stores random keys into the database
* using an {@link IntComparator}
* it then checks that these keys are read back in
* ascending order
*
* @param db_path A path where we can store database
* files temporarily
*
* @param comparator the comparator
*
* @throws RocksDBException if a database error happens.
*/
private void testRoundtrip(final Path db_path,
final AbstractComparator comparator) throws RocksDBException {
try (final Options opt = new Options()
.setCreateIfMissing(true)
.setComparator(comparator)) {
// store TOTAL_KEYS into the db
try (final RocksDB db = RocksDB.open(opt, db_path.toString())) {
for (int i = 0; i < TOTAL_KEYS; i++) {
db.put(keys[i], "value".getBytes(UTF_8));
}
}
// re-open db and read from start to end
// integer keys should be in descending
// order
final ByteBuffer key = ByteBuffer.allocate(4);
try (final RocksDB db = RocksDB.open(opt, db_path.toString());
final RocksIterator it = db.newIterator()) {
it.seekToFirst();
int lastKey = Integer.MAX_VALUE;
int count = 0;
for (it.seekToFirst(); it.isValid(); it.next()) {
key.put(it.key());
key.flip();
final int thisKey = key.getInt();
key.clear();
assertThat(thisKey).isLessThan(lastKey);
lastKey = thisKey;
count++;
}
assertThat(count).isEqualTo(TOTAL_KEYS);
}
}
}
/**
* Test which stores random keys into a column family
* in the database
* using an {@link IntComparator}
* it then checks that these keys are read back in
* ascending order
*
* @param db_path A path where we can store database
* files temporarily
*
* @param comparator the comparator
*
* @throws RocksDBException if a database error happens.
*/
private void testRoundtripCf(final Path db_path,
final AbstractComparator comparator) throws RocksDBException {
final List<ColumnFamilyDescriptor> cfDescriptors = Arrays.asList(
new ColumnFamilyDescriptor(RocksDB.DEFAULT_COLUMN_FAMILY),
new ColumnFamilyDescriptor("new_cf".getBytes(),
new ColumnFamilyOptions()
.setComparator(comparator))
);
final List<ColumnFamilyHandle> cfHandles = new ArrayList<>();
try (final DBOptions opt = new DBOptions()
.setCreateIfMissing(true)
.setCreateMissingColumnFamilies(true)) {
try (final RocksDB db = RocksDB.open(opt, db_path.toString(),
cfDescriptors, cfHandles)) {
try {
assertThat(cfDescriptors.size()).isEqualTo(2);
assertThat(cfHandles.size()).isEqualTo(2);
for (int i = 0; i < TOTAL_KEYS; i++) {
db.put(cfHandles.get(1), keys[i], "value".getBytes(UTF_8));
}
} finally {
for (final ColumnFamilyHandle cfHandle : cfHandles) {
cfHandle.close();
}
cfHandles.clear();
}
}
// re-open db and read from start to end
// integer keys should be in descending
// order
final ByteBuffer key = ByteBuffer.allocate(4);
try (final RocksDB db = RocksDB.open(opt, db_path.toString(),
cfDescriptors, cfHandles);
final RocksIterator it = db.newIterator(cfHandles.get(1))) {
try {
assertThat(cfDescriptors.size()).isEqualTo(2);
assertThat(cfHandles.size()).isEqualTo(2);
it.seekToFirst();
int lastKey = Integer.MAX_VALUE;
int count = 0;
for (it.seekToFirst(); it.isValid(); it.next()) {
key.put(it.key());
key.flip();
final int thisKey = key.getInt();
key.clear();
assertThat(thisKey).isLessThan(lastKey);
lastKey = thisKey;
count++;
}
assertThat(count).isEqualTo(TOTAL_KEYS);
} finally {
for (final ColumnFamilyHandle cfHandle : cfHandles) {
cfHandle.close();
}
cfHandles.clear();
for (final ColumnFamilyDescriptor cfDescriptor : cfDescriptors) {
cfDescriptor.getOptions().close();
}
}
}
}
}
}
java/src/test/java/org/rocksdb/util/TestUtil.java
浏览文件 @ 7242dae7
......@@ -58,15 +58,4 @@ public class TestUtil {
random.nextBytes(str);
return str;
}
/**
* Convert a UTF-8 String to a byte array.
*
* @param str the string
*
* @return the byte array.
*/
public static byte[] u(final String str) {
return str.getBytes(UTF_8);
}
}
Markdown is supported
0% 或 .
You are about to add 0 people to the discussion. Proceed with caution.
先完成此消息的编辑！
想要评论请 注册 或