未验证 提交 4284a575 编写于 作者: K Kirs 提交者: GitHub

Add MicroBench module to make it easier for developers to write JMH Test (#7985)

上级 10bb68a8
......@@ -46,6 +46,7 @@ Release Notes.
* Optimize metrics of minute dimensionality persistence. The value of metrics, which has declaration of the default
value and current value equals the default value logically, the whole row wouldn't be pushed into database.
* Fix `max` function in OAL doesn't support negative long.
* Add `MicroBench` module to make it easier for developers to write JMH test.
#### UI
......
......@@ -44,6 +44,20 @@ and if you would like to run all the ITs, simply run `./mvnw -Pall,CI-with-IT cl
Please be advised that if you're writing integration tests, name it with the pattern `IT*` so they would only run with the `CI-with-IT` profile.
### Java Microbenchmark Harness (JMH)
JMH is a Java harness for building, running, and analysing nano/micro/milli/macro benchmarks written in Java and other languages targeting the JVM.
We have a module called `microbench` which performs a series of micro-benchmark tests for JMH testing.
Make new JMH tests extend the `org.apache.skywalking.oap.server.microbench.base.AbstractMicrobenchmark`
to customize runtime conditions (Measurement, Fork, Warmup, etc.).
JMH tests could run as a normal unit test. And they could run as an independent uber jar via `java -jar benchmark.jar` for all benchmarks,
or via `java -jar /benchmarks.jar exampleClassName` for a specific test.
Output test results in JSON format, you can add `-rf json` like `java -jar benchmarks.jar -rf json`, if you run through the IDE, you can configure the `-DperfReportDir=savePath` parameter to set the JMH report result save path, a report results in JSON format will be generated when the run ends.
More information about JMH can be found here: [jmh docs](https://openjdk.java.net/projects/code-tools/jmh/).
### End to End Tests (E2E)
Since version 6.3.0, we have introduced more automatic tests to perform software quality assurance. E2E is an integral part of it.
......
<?xml version="1.0" encoding="UTF-8"?>
<!--
~ Licensed to the Apache Software Foundation (ASF) under one or more
~ contributor license agreements. See the NOTICE file distributed with
~ this work for additional information regarding copyright ownership.
~ The ASF licenses this file to You under the Apache License, Version 2.0
~ (the "License"); you may not use this file except in compliance with
~ the License. You may obtain a copy of the License at
~
~ http://www.apache.org/licenses/LICENSE-2.0
~
~ Unless required by applicable law or agreed to in writing, software
~ distributed under the License is distributed on an "AS IS" BASIS,
~ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
~ See the License for the specific language governing permissions and
~ limitations under the License.
~
-->
<project xmlns="http://maven.apache.org/POM/4.0.0"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
<parent>
<artifactId>oap-server</artifactId>
<groupId>org.apache.skywalking</groupId>
<version>8.9.0-SNAPSHOT</version>
</parent>
<modelVersion>4.0.0</modelVersion>
<artifactId>microbench</artifactId>
<properties>
<jmh.version>1.25</jmh.version>
<slf4j.version>1.7.30</slf4j.version>
<uberjar.name>benchmarks</uberjar.name>
<maven-shade-plugin.version>3.1.1</maven-shade-plugin.version>
</properties>
<dependencies>
<dependency>
<groupId>org.apache.skywalking</groupId>
<artifactId>server-core</artifactId>
<version>${project.version}</version>
</dependency>
<dependency>
<groupId>org.apache.skywalking</groupId>
<artifactId>library-util</artifactId>
<version>${project.version}</version>
</dependency>
<dependency>
<groupId>org.apache.skywalking</groupId>
<artifactId>library-datacarrier-queue</artifactId>
<version>${project.version}</version>
</dependency>
<!--JMH-->
<dependency>
<groupId>org.openjdk.jmh</groupId>
<artifactId>jmh-core</artifactId>
<version>${jmh.version}</version>
</dependency>
<dependency>
<groupId>org.openjdk.jmh</groupId>
<artifactId>jmh-generator-annprocess</artifactId>
<version>${jmh.version}</version>
<scope>provided</scope>
</dependency>
<!--SLF4j-->
<dependency>
<groupId>org.slf4j</groupId>
<artifactId>slf4j-api</artifactId>
<version>${slf4j.version}</version>
</dependency>
<!--JUNIT-->
<dependency>
<groupId>junit</groupId>
<artifactId>junit</artifactId>
<version>${junit.version}</version>
<scope>compile</scope>
</dependency>
</dependencies>
<build>
<plugins>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-shade-plugin</artifactId>
<version>${maven-shade-plugin.version}</version>
<executions>
<execution>
<phase>package</phase>
<goals>
<goal>shade</goal>
</goals>
<configuration>
<finalName>${uberjar.name}</finalName>
<transformers>
<transformer
implementation="org.apache.maven.plugins.shade.resource.ManifestResourceTransformer">
<mainClass>org.openjdk.jmh.Main</mainClass>
</transformer>
</transformers>
<filters>
<filter>
<artifact>*:*</artifact>
<excludes>
<exclude>**/Log4j2Plugins.dat</exclude>
</excludes>
</filter>
</filters>
</configuration>
</execution>
</executions>
</plugin>
</plugins>
</build>
</project>
/*
* Licensed to the Apache Software Foundation (ASF) under one or more
* contributor license agreements. See the NOTICE file distributed with
* this work for additional information regarding copyright ownership.
* The ASF licenses this file to You under the Apache License, Version 2.0
* (the "License"); you may not use this file except in compliance with
* the License. You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*
*/
package org.apache.skywalking.oap.server.microbench.base;
import java.io.File;
import java.io.IOException;
import java.util.concurrent.Executors;
import java.util.concurrent.LinkedBlockingQueue;
import java.util.concurrent.ThreadPoolExecutor;
import java.util.concurrent.TimeUnit;
import org.junit.Test;
import org.openjdk.jmh.annotations.Fork;
import org.openjdk.jmh.annotations.Measurement;
import org.openjdk.jmh.annotations.Scope;
import org.openjdk.jmh.annotations.State;
import org.openjdk.jmh.annotations.Warmup;
import org.openjdk.jmh.profile.GCProfiler;
import org.openjdk.jmh.results.format.ResultFormatType;
import org.openjdk.jmh.runner.Runner;
import org.openjdk.jmh.runner.options.ChainedOptionsBuilder;
import org.openjdk.jmh.runner.options.OptionsBuilder;
import lombok.extern.slf4j.Slf4j;
/**
* All JMH tests need to extend this class to make it easier for you to complete JMHTest, you can also choose to
* customize runtime conditions (Measurement, Fork, Warmup, etc.)
* <p>
* You can run any of the JMH tests as a normal UT, or you can package it and get all the reported results via `java
* -jar benchmark.jar`, or get the results of a particular Test via `java -jar /benchmarks.jar exampleClassName`.
*/
@Warmup(iterations = AbstractMicrobenchmark.DEFAULT_WARMUP_ITERATIONS)
@Measurement(iterations = AbstractMicrobenchmark.DEFAULT_MEASURE_ITERATIONS)
@Fork(AbstractMicrobenchmark.DEFAULT_FORKS)
@State(Scope.Thread)
@Slf4j
public abstract class AbstractMicrobenchmark {
static final int DEFAULT_WARMUP_ITERATIONS = 10;
static final int DEFAULT_MEASURE_ITERATIONS = 10;
static final int DEFAULT_FORKS = 2;
public static class JmhThreadExecutor extends ThreadPoolExecutor {
public JmhThreadExecutor(int size, String name) {
super(size, size, 10, TimeUnit.SECONDS, new LinkedBlockingQueue<>(), Executors.defaultThreadFactory());
}
}
private ChainedOptionsBuilder newOptionsBuilder() {
String className = getClass().getSimpleName();
ChainedOptionsBuilder optBuilder = new OptionsBuilder()
// set benchmark class name
.include(".*" + className + ".*")
// add GC profiler
.addProfiler(GCProfiler.class)
//set jvm args
.jvmArgsAppend("-Xmx512m", "-Xms512m", "-XX:MaxDirectMemorySize=512m",
"-XX:BiasedLockingStartupDelay=0",
"-Djmh.executor=CUSTOM",
"-Djmh.executor.class=org.apache.skywalking.oap.server.microbench.base.AbstractMicrobenchmark$JmhThreadExecutor"
);
String output = getReportDir();
if (output != null) {
boolean writeFileStatus;
String filePath = getReportDir() + className + ".json";
File file = new File(filePath);
if (file.exists()) {
writeFileStatus = file.delete();
} else {
writeFileStatus = file.getParentFile().mkdirs();
try {
writeFileStatus = file.createNewFile();
} catch (IOException e) {
log.warn("jmh test create file error", e);
}
}
if (writeFileStatus) {
optBuilder.resultFormat(ResultFormatType.JSON)
.result(filePath);
}
}
return optBuilder;
}
@Test
public void run() throws Exception {
new Runner(newOptionsBuilder().build()).run();
}
private static String getReportDir() {
return System.getProperty("perfReportDir");
}
}
......@@ -16,116 +16,82 @@
*
*/
package org.apache.skywalking.oap.server.core.config.group.openapi;
package org.apache.skywalking.oap.server.microbench.core.config.group.openapi;
import org.apache.skywalking.oap.server.core.config.group.openapi.EndpointGroupingRule4Openapi;
import org.apache.skywalking.oap.server.core.config.group.openapi.EndpointGroupingRuleReader4Openapi;
import org.apache.skywalking.oap.server.library.util.StringFormatGroup.FormatResult;
import org.apache.skywalking.oap.server.microbench.base.AbstractMicrobenchmark;
import java.util.Collections;
import java.util.Map;
import java.io.FileNotFoundException;
import lombok.SneakyThrows;
import org.openjdk.jmh.annotations.Benchmark;
import org.openjdk.jmh.annotations.BenchmarkMode;
import org.openjdk.jmh.annotations.Mode;
import org.openjdk.jmh.annotations.Scope;
import org.openjdk.jmh.annotations.State;
import org.openjdk.jmh.annotations.Threads;
import org.openjdk.jmh.profile.GCProfiler;
import org.openjdk.jmh.runner.Runner;
import org.openjdk.jmh.runner.RunnerException;
import org.openjdk.jmh.runner.options.Options;
import org.openjdk.jmh.runner.options.OptionsBuilder;
import org.openjdk.jmh.infra.Blackhole;
@BenchmarkMode({Mode.Throughput})
@Threads(4)
public class EndpointGroupingBenchmark4Openapi {
public class EndpointGrouping4OpenapiBenchmark extends AbstractMicrobenchmark {
private static final String APT_TEST_DATA = " /products1/{id}/%d:\n" + " get:\n" + " post:\n"
+ " /products2/{id}/%d:\n" + " get:\n" + " post:\n"
+ " /products3/{id}/%d:\n" + " get:\n";
private static Map<String, String> createTestFile(int size) {
StringBuilder stringBuilder = new StringBuilder();
stringBuilder.append("paths:\n");
for (int i = 0; i <= size; i++) {
stringBuilder.append(String.format(APT_TEST_DATA, i, i, i));
}
return Collections.singletonMap("whatever", stringBuilder.toString());
}
@State(Scope.Benchmark)
public static class FormatClassPaths20 {
private EndpointGroupingRule4Openapi rule;
@SneakyThrows
public FormatClassPaths20() {
rule = new EndpointGroupingRule4Openapi();
for (int i = 0; i <= 3; i++) {
rule.addGroupedRule("serviceA", "GET:/products1/{id}/" + i, "GET:/products1/([^/]+)/" + i);
rule.addGroupedRule("serviceA", "POST:/products1/{id}/" + i, "POST:/products1/([^/]+)/" + i);
rule.addGroupedRule("serviceA", "GET:/products2/{id}/" + i, "GET:/products2/([^/]+)/" + i);
rule.addGroupedRule("serviceA", "POST:/products3/{id}/" + i, "POST:/products3/([^/]+)/" + i);
rule.addGroupedRule("serviceA", "GET:/products3/{id}/" + i, "GET:/products3/([^/]+)/" + i);
}
}
private final EndpointGroupingRule4Openapi rule = new EndpointGroupingRuleReader4Openapi(createTestFile(3)).read();
public void format(String serviceName, String endpointName) {
rule.format(serviceName, endpointName);
public FormatResult format(String serviceName, String endpointName) {
return rule.format(serviceName, endpointName);
}
}
@State(Scope.Benchmark)
public static class FormatClassPaths50 {
private EndpointGroupingRule4Openapi rule;
@SneakyThrows
public FormatClassPaths50() {
rule = new EndpointGroupingRule4Openapi();
for (int i = 0; i <= 9; i++) {
rule.addGroupedRule("serviceA", "GET:/products1/{id}/" + i, "GET:/products1/([^/]+)/" + i);
rule.addGroupedRule("serviceA", "POST:/products1/{id}/" + i, "POST:/products1/([^/]+)/" + i);
rule.addGroupedRule("serviceA", "GET:/products2/{id}/" + i, "GET:/products2/([^/]+)/" + i);
rule.addGroupedRule("serviceA", "POST:/products3/{id}/" + i, "POST:/products3/([^/]+)/" + i);
rule.addGroupedRule("serviceA", "GET:/products3/{id}/" + i, "GET:/products3/([^/]+)/" + i);
}
}
private final EndpointGroupingRule4Openapi rule = new EndpointGroupingRuleReader4Openapi(createTestFile(9)).read();
public void format(String serviceName, String endpointName) {
rule.format(serviceName, endpointName);
public FormatResult format(String serviceName, String endpointName) {
return rule.format(serviceName, endpointName);
}
}
@State(Scope.Benchmark)
public static class FormatClassPaths200 {
private EndpointGroupingRule4Openapi rule;
@SneakyThrows
public FormatClassPaths200() {
rule = new EndpointGroupingRule4Openapi();
for (int i = 0; i <= 39; i++) {
rule.addGroupedRule("serviceA", "GET:/products1/{id}/" + i, "GET:/products1/([^/]+)/" + i);
rule.addGroupedRule("serviceA", "POST:/products1/{id}/" + i, "POST:/products1/([^/]+)/" + i);
rule.addGroupedRule("serviceA", "GET:/products2/{id}/" + i, "GET:/products2/([^/]+)/" + i);
rule.addGroupedRule("serviceA", "POST:/products3/{id}/" + i, "POST:/products3/([^/]+)/" + i);
rule.addGroupedRule("serviceA", "GET:/products3/{id}/" + i, "GET:/products3/([^/]+)/" + i);
}
}
private final EndpointGroupingRule4Openapi rule = new EndpointGroupingRuleReader4Openapi(createTestFile(39)).read();
public void format(String serviceName, String endpointName) {
rule.format(serviceName, endpointName);
public FormatResult format(String serviceName, String endpointName) {
return rule.format(serviceName, endpointName);
}
}
@Benchmark
public void formatEndpointNameMatchedPaths20(FormatClassPaths20 formatClass) {
formatClass.format("serviceA", "GET:/products1/123");
public void formatEndpointNameMatchedPaths20(Blackhole bh, FormatClassPaths20 formatClass) {
bh.consume(formatClass.format("serviceA", "GET:/products1/123"));
}
@Benchmark
public void formatEndpointNameMatchedPaths50(FormatClassPaths50 formatClass) {
formatClass.format("serviceA", "GET:/products1/123");
public void formatEndpointNameMatchedPaths50(Blackhole bh, FormatClassPaths50 formatClass) {
bh.consume(formatClass.format("serviceA", "GET:/products1/123"));
}
@Benchmark
public void formatEndpointNameMatchedPaths200(FormatClassPaths200 formatClass) {
formatClass.format("serviceA", "GET:/products1/123");
public void formatEndpointNameMatchedPaths200(Blackhole bh, FormatClassPaths200 formatClass) {
bh.consume(formatClass.format("serviceA", "GET:/products1/123"));
}
public static void main(String[] args) throws RunnerException, FileNotFoundException {
Options opt = new OptionsBuilder()
.include(EndpointGroupingBenchmark4Openapi.class.getName())
.addProfiler(GCProfiler.class)
.jvmArgsAppend("-Xmx512m", "-Xms512m")
.forks(1)
.build();
new Runner(opt).run();
}
}
/*
......@@ -169,4 +135,4 @@ EndpointGroupingBenchmark4Openapi.formatEndpointNameMatchedPaths50:·gc.churn.PS
EndpointGroupingBenchmark4Openapi.formatEndpointNameMatchedPaths50:·gc.churn.PS_Survivor_Space.norm thrpt 5 0.231 ± 0.066 B/op
EndpointGroupingBenchmark4Openapi.formatEndpointNameMatchedPaths50:·gc.count thrpt 5 1405.000 counts
EndpointGroupingBenchmark4Openapi.formatEndpointNameMatchedPaths50:·gc.time thrpt 5 841.000 ms
*/
\ No newline at end of file
*/
......@@ -16,16 +16,10 @@
*
*/
package org.apache.skywalking.oap.server.library.datacarrier;
package org.apache.skywalking.oap.server.microbench.library.datacarrier;
import org.apache.skywalking.oap.server.microbench.base.AbstractMicrobenchmark;
import org.openjdk.jmh.annotations.Benchmark;
import org.openjdk.jmh.annotations.BenchmarkMode;
import org.openjdk.jmh.annotations.Mode;
import org.openjdk.jmh.profile.GCProfiler;
import org.openjdk.jmh.runner.Runner;
import org.openjdk.jmh.runner.RunnerException;
import org.openjdk.jmh.runner.options.Options;
import org.openjdk.jmh.runner.options.OptionsBuilder;
import java.util.ArrayList;
import java.util.LinkedList;
......@@ -34,8 +28,7 @@ import java.util.List;
/**
* ISSUE-3064
*/
@BenchmarkMode({Mode.Throughput})
public class LinkedArrayBenchmark {
public class LinkedArrayBenchmark extends AbstractMicrobenchmark {
@Benchmark
public void testArrayCap1000() {
......@@ -167,14 +160,17 @@ public class LinkedArrayBenchmark {
}
}
public static void main(String[] args) throws RunnerException {
Options opt = new OptionsBuilder().include(LinkedArrayBenchmark.class.getName())
.addProfiler(GCProfiler.class)
.jvmArgsAppend("-Xmx512m", "-Xms512m")
.forks(1)
.build();
new Runner(opt).run();
/**
* Test Data
*/
public class SampleData {
private int intValue;
private String name;
}
/*
Environment:
......
......@@ -16,17 +16,15 @@
*
*/
package org.apache.skywalking.oap.server.library.datacarrier.common;
package org.apache.skywalking.oap.server.microbench.library.datacarrier.common;
import org.apache.skywalking.oap.server.library.datacarrier.common.AtomicRangeInteger;
import org.apache.skywalking.oap.server.microbench.base.AbstractMicrobenchmark;
import org.junit.Assert;
import org.junit.Test;
import org.openjdk.jmh.annotations.Benchmark;
import org.openjdk.jmh.runner.Runner;
import org.openjdk.jmh.runner.RunnerException;
import org.openjdk.jmh.runner.options.Options;
import org.openjdk.jmh.runner.options.OptionsBuilder;
public class AtomicRangeIntegerTest {
public class AtomicRangeIntegerBenchmark extends AbstractMicrobenchmark {
private static AtomicRangeInteger ATOMIC_V3 = new AtomicRangeInteger(0, 100);
private static AtomicRangeIntegerV1 ATOMIC_V1 = new AtomicRangeIntegerV1(0, 100);
......@@ -64,19 +62,6 @@ public class AtomicRangeIntegerTest {
ATOMIC_V3.getAndIncrement();
}
public static void main(String[] args) throws RunnerException {
Options opt = new OptionsBuilder().include(AtomicRangeIntegerTest.class.getSimpleName())
.forks(1)
.warmupIterations(3)
.threads(128)
.syncIterations(false)
.output("/tmp/jmh.log")
.measurementIterations(5)
.build();
new Runner(opt).run();
}
/**
* # JMH version: 1.21
* # VM version: JDK 1.8.0_111, Java HotSpot(TM) 64-Bit Server VM, 25.111-b14
......
......@@ -16,7 +16,7 @@
*
*/
package org.apache.skywalking.oap.server.library.datacarrier.common;
package org.apache.skywalking.oap.server.microbench.library.datacarrier.common;
import java.io.Serializable;
import java.util.concurrent.atomic.AtomicInteger;
......
......@@ -16,7 +16,7 @@
*
*/
package org.apache.skywalking.oap.server.library.datacarrier.common;
package org.apache.skywalking.oap.server.microbench.library.datacarrier.common;
import java.io.Serializable;
import java.util.concurrent.atomic.AtomicInteger;
......
......@@ -16,7 +16,10 @@
*
*/
package org.apache.skywalking.oap.server.library.util;
package org.apache.skywalking.oap.server.microbench.library.util;
import org.apache.skywalking.oap.server.library.util.StringFormatGroup;
import org.apache.skywalking.oap.server.microbench.base.AbstractMicrobenchmark;
import java.util.concurrent.TimeUnit;
......@@ -26,12 +29,10 @@ import org.openjdk.jmh.annotations.Benchmark;
import org.openjdk.jmh.annotations.BenchmarkMode;
import org.openjdk.jmh.annotations.Mode;
import org.openjdk.jmh.annotations.OutputTimeUnit;
import org.openjdk.jmh.runner.Runner;
import org.openjdk.jmh.runner.RunnerException;
import org.openjdk.jmh.runner.options.Options;
import org.openjdk.jmh.runner.options.OptionsBuilder;
public class StringFormatGroupTest {
@BenchmarkMode(Mode.AverageTime)
@OutputTimeUnit(TimeUnit.MICROSECONDS)
public class StringFormatGroupBenchmark extends AbstractMicrobenchmark {
@Benchmark
@Test
public void testMatch() {
......@@ -55,21 +56,6 @@ public class StringFormatGroupTest {
Assert.assertEquals("/name/*/add/{orderId}", group.format("/name/test/add/12323").getName());
}
/**
* The report below shows this pattern match performance is much about rule numbers. This is a single thread test.
*/
@BenchmarkMode(Mode.AverageTime)
@OutputTimeUnit(TimeUnit.MICROSECONDS)
public void performanceBenchmark() throws RunnerException {
Options opt = new OptionsBuilder().include(StringFormatGroupTest.class.getSimpleName())
.forks(1)
.warmupIterations(0)
.measurementIterations(5)
.build();
new Runner(opt).run();
}
/*********************************
* # JMH version: 1.21
* # VM version: JDK 1.8.0_91, Java HotSpot(TM) 64-Bit Server VM, 25.91-b14
......
......@@ -46,6 +46,7 @@
<module>server-tools</module>
<module>server-fetcher-plugin</module>
<module>server-health-checker</module>
<module>microbench</module>
</modules>
<properties>
......
......@@ -93,11 +93,6 @@
<artifactId>grpc-testing</artifactId>
<scope>test</scope>
</dependency>
<dependency>
<groupId>org.openjdk.jmh</groupId>
<artifactId>jmh-generator-annprocess</artifactId>
<scope>test</scope>
</dependency>
</dependencies>
<build>
......
......@@ -27,12 +27,4 @@
<artifactId>library-datacarrier-queue</artifactId>
<dependencies>
<dependency>
<groupId>org.openjdk.jmh</groupId>
<artifactId>jmh-generator-annprocess</artifactId>
<scope>test</scope>
</dependency>
</dependencies>
</project>
......@@ -35,4 +35,4 @@
<scope>compile</scope>
</dependency>
</dependencies>
</project>
\ No newline at end of file
</project>
......@@ -38,11 +38,6 @@
<artifactId>meter-analyzer</artifactId>
<version>${project.version}</version>
</dependency>
<dependency>
<groupId>org.openjdk.jmh</groupId>
<artifactId>jmh-generator-annprocess</artifactId>
<scope>test</scope>
</dependency>
</dependencies>
</project>
\ No newline at end of file
</project>
......@@ -35,4 +35,4 @@
<version>${project.version}</version>
</dependency>
</dependencies>
</project>
\ No newline at end of file
</project>
Markdown is supported
0% .
You are about to add 0 people to the discussion. Proceed with caution.
先完成此消息的编辑!
想要评论请 注册