提交 933ae20b 编写于 作者: wu-sheng's avatar wu-sheng 提交者: 彭勇升 pengys

Alarm module code core ready (#1644)

* Support alarm message.

* Fix document words.

* Support alarm hook.

* Fix javadoc issue.

* Sync submodule

* Sync submodule
上级 53b9e7f8
## Support custom trace ignore
Here is an optional plugin `apm-trace-ignore-plugin`
## Introduce
- The purpose of this plugin is to filter custom services which are expected to ignore from the tracing system.
- You can set up multiple URL path patterns, The services matches these patterns means won't be traced and analysis by agent and collector.
- The current matching rule follows `Ant Path` match style , like `/path/*`, `/path/**`, `/path/?`.
- Copy `apm-trace-ignore-plugin-x.jar` to `agent/plugins`, restarting the `agent` can effect the plugin.
## How to configure
There are two ways to configure ignore patterns. Settings through system env has higher priority.
1. Set through the system environment variable,you need to add `skywalking.trace.ignore_path` to the system variables, the value is the path that you need to ignore, multiple paths should be separated by `,`
2. Copy`/agent/optional-plugins/apm-trace-ignore-plugin/apm-trace-ignore-plugin.config` to `/agent/config/` dir, and add rules to filter traces
```
trace.ignore_path=/your/path/1/**,/your/path/2/**
```
## 个性化服务过滤
提供了一个可选插件 `apm-trace-ignore-plugin`
## 介绍
- 这个插件的作用是对追踪的个性化服务过滤.
- 你可以设置多个需要忽略的URL路径, 意味着包含这些路径的`追踪信息`不会被`agent`发送到 `collector`.
- 当前的路径匹配规则是 `Ant Path`匹配风格 , 例如 `/path/*`, `/path/**`, `/path/?`.
-`apm-trace-ignore-plugin-x.jar`拷贝到`agent/plugins`后,重启探针即可生效
- [Skywalking-使用可选插件 apm-trace-ignore-plugin](https://blog.csdn.net/u013095337/article/details/80452088) 有详细使用介绍
## 如何配置路径
有两种配置方式,可使用任意一种,配置生效的优先级从高到低
1. 在系统环境变量中配置,你需要在系统变量中添加`skywalking.trace.ignore_path`, 值是你需要忽略的路径,多个以`,`号分隔
2.`/agent/optional-plugins/apm-trace-ignore-plugin/apm-trace-ignore-plugin.config` 复制或剪切到 `/agent/config/` 目录下,加上配置
```
trace.ignore_path=/your/path/1/**,/your/path/2/**
```
......@@ -30,33 +30,4 @@
<name>apm-trace-ignore-plugin</name>
<url>http://maven.apache.org</url>
<build>
<plugins>
<plugin>
<artifactId>maven-antrun-plugin</artifactId>
<executions>
<execution>
<phase>package</phase>
<goals>
<goal>run</goal>
</goals>
<configuration combine.self="override">
<tasks>
<taskdef resource="net/sf/antcontrib/antcontrib.properties" classpathref="maven.runtime.classpath" />
<mkdir dir="${optional.plugin.dest.dir}/${project.artifactId}" />
<!-- copy jar -->
<copy file="${project.build.directory}/${project.artifactId}-${project.version}.jar" tofile="${optional.plugin.dest.dir}/${project.artifactId}/${project.artifactId}-${project.version}.jar" overwrite="true" />
<!-- copy config file -->
<copy file="${project.basedir}/${project.name}.config" tofile="${optional.plugin.dest.dir}/${project.name}/${project.name}.config" overwrite="true" />
<!-- copy introduction -->
<copy file="${project.basedir}/README.md" tofile="${optional.plugin.dest.dir}/${project.name}/README.md" overwrite="true" />
<copy file="${project.basedir}/README_CN.md" tofile="${optional.plugin.dest.dir}/${project.name}/README_CN.md" overwrite="true" />
</tasks>
</configuration>
</execution>
</executions>
</plugin>
</plugins>
</build>
</project>
......@@ -53,7 +53,7 @@ For using these plugins, you need to compile source codes by yourself, or copy t
Now, we have the following known plugins.
* [Trace Spring beans](agent-optional-plugins/Spring-bean-plugins.md)
* [Trace Oracle and Resin](agent-optional-plugins/Oracle-Resin-plugins.md)
* [Filter traces through custom services](agent-optional-plugins/trace-ignore-plugin.md)
* [Filter traces through specified endpoint name patterns](agent-optional-plugins/trace-ignore-plugin.md)
## Advanced Features
* Set the settings through system properties for config file override. Read [setting override](Setting-override.md).
......
......@@ -2,9 +2,9 @@
Here is an optional plugin `apm-trace-ignore-plugin`
## Introduce
- The purpose of this plugin is to filter custom services which are expected to ignore from the tracing system.
- You can set up multiple URL path patterns, The services matches these patterns means won't be traced and analysis by agent and collector.
- The current matching rule follows `Ant Path` match style , like `/path/*`, `/path/**`, `/path/?`.
- The purpose of this plugin is to filter endpoint which are expected to be ignored by the tracing system.
- You can setup multiple URL path patterns, The endpoints match these patterns wouldn't be traced.
- The current matching rules follow `Ant Path` match style , like `/path/*`, `/path/**`, `/path/?`.
- Copy `apm-trace-ignore-plugin-x.jar` to `agent/plugins`, restarting the `agent` can effect the plugin.
## How to configure
......
/*
* Licensed to the Apache Software Foundation (ASF) under one or more
* contributor license agreements. See the NOTICE file distributed with
* this work for additional information regarding copyright ownership.
* The ASF licenses this file to You under the Apache License, Version 2.0
* (the "License"); you may not use this file except in compliance with
* the License. You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*
*/
package org.apache.skywalking.oap.server.core.alarm.provider;
import java.util.ArrayList;
import java.util.List;
import org.apache.skywalking.oap.server.core.alarm.MetaInAlarm;
/**
* This is a formatter especially for alarm message.
*
* Format string in alarm-settings.yml, such as:
*
* - Successful rate of endpoint {name} is lower than 75%
*
* @author wusheng
*/
public class AlarmMessageFormatter {
private List<String> formatSegments;
private List<ValueFrom> valueFroms;
public AlarmMessageFormatter(String format) {
if (format == null) {
format = "";
}
formatSegments = new ArrayList<>();
this.valueFroms = new ArrayList<>();
boolean match = false;
int idx = 0;
do {
match = false;
int start = format.indexOf("{", idx);
if (start > -1) {
int end = format.indexOf("}", start);
if (end > -1) {
String name = format.substring(start + 1, end);
switch (name) {
case "id":
valueFroms.add(ValueFrom.ID);
break;
case "name":
valueFroms.add(ValueFrom.NAME);
break;
default:
throw new IllegalArgumentException("Var [" + name + "] in alarm message [" + format + "] is illegal");
}
formatSegments.add(format.substring(idx, start));
idx = end + 1;
match = true;
}
}
if (!match) {
formatSegments.add(format.substring(idx));
}
}
while (match);
}
public String format(MetaInAlarm meta) {
StringBuilder message = new StringBuilder();
for (int i = 0; i < formatSegments.size(); i++) {
message.append(formatSegments.get(i));
if (i != formatSegments.size() - 1) {
switch (valueFroms.get(i)) {
case ID:
message.append(meta.getId0());
break;
case NAME:
message.append(meta.getName());
}
}
}
return message.toString();
}
private enum ValueFrom {
ID, NAME
}
}
......@@ -38,4 +38,5 @@ public class AlarmRule {
private int period;
private int count;
private int silencePeriod;
private String message;
}
......@@ -66,6 +66,7 @@ public class RulesReader {
alarmRule.setPeriod((Integer)settings.getOrDefault("period", 1));
alarmRule.setCount((Integer)settings.getOrDefault("count", 1));
alarmRule.setSilencePeriod((Integer)settings.getOrDefault("silence-period", alarmRule.getPeriod()));
alarmRule.setMessage((String)settings.getOrDefault("message", "Alarm caused by Rule " + alarmRule.getAlarmRuleName()));
rules.getRules().add(alarmRule);
}
......
......@@ -58,6 +58,7 @@ public class RunningRule {
private volatile IndicatorValueType valueType;
private Scope targetScope;
private List<String> includeNames;
private AlarmMessageFormatter formatter;
public RunningRule(AlarmRule alarmRule) {
indicatorName = alarmRule.getIndicatorName();
......@@ -75,6 +76,7 @@ public class RunningRule {
this.silencePeriod = alarmRule.getSilencePeriod();
this.includeNames = alarmRule.getIncludeNames();
this.formatter = new AlarmMessageFormatter(alarmRule.getMessage());
}
/**
......@@ -136,9 +138,16 @@ public class RunningRule {
public List<AlarmMessage> check() {
List<AlarmMessage> alarmMessageList = new ArrayList<>(30);
windows.values().forEach(window -> {
windows.entrySet().forEach(entry -> {
MetaInAlarm meta = entry.getKey();
Window window = entry.getValue();
AlarmMessage alarmMessage = window.checkAlarm();
if (alarmMessage != AlarmMessage.NONE) {
alarmMessage.setScope(meta.getScope());
alarmMessage.setName(meta.getName());
alarmMessage.setId0(meta.getId0());
alarmMessage.setId1(meta.getId1());
alarmMessage.setAlarmMessage(formatter.format(meta));
alarmMessageList.add(alarmMessage);
}
});
......@@ -238,7 +247,7 @@ public class RunningRule {
if (counter >= countThreshold && silenceCountdown < 1) {
silenceCountdown = silencePeriod;
//TODO
// set empty message, but new message
AlarmMessage message = new AlarmMessage();
return message;
} else {
......
......@@ -18,21 +18,82 @@
package org.apache.skywalking.oap.server.core.alarm.provider;
import com.google.gson.Gson;
import java.io.IOException;
import java.io.UnsupportedEncodingException;
import java.util.List;
import org.apache.http.StatusLine;
import org.apache.http.client.ClientProtocolException;
import org.apache.http.client.config.RequestConfig;
import org.apache.http.client.methods.CloseableHttpResponse;
import org.apache.http.client.methods.HttpPost;
import org.apache.http.entity.StringEntity;
import org.apache.http.impl.client.CloseableHttpClient;
import org.apache.http.impl.client.HttpClients;
import org.apache.skywalking.oap.server.core.alarm.AlarmCallback;
import org.apache.skywalking.oap.server.core.alarm.AlarmMessage;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
/**
* Use SkyWalking alarm webhook API call a remote endpoints.
*
* @author wusheng
*/
public class WebhookCallback implements AlarmCallback {
private static final Logger logger = LoggerFactory.getLogger(WebhookCallback.class);
private static final int HTTP_CONNECT_TIMEOUT = 1000;
private static final int HTTP_CONNECTION_REQUEST_TIMEOUT = 1000;
private static final int HTTP_SOCKET_TIMEOUT = 10000;
private List<String> remoteEndpoints;
private RequestConfig requestConfig;
private Gson gson = new Gson();
public WebhookCallback(List<String> remoteEndpoints) {
this.remoteEndpoints = remoteEndpoints;
requestConfig = RequestConfig.custom()
.setConnectTimeout(HTTP_CONNECT_TIMEOUT)
.setConnectionRequestTimeout(HTTP_CONNECTION_REQUEST_TIMEOUT)
.setSocketTimeout(HTTP_SOCKET_TIMEOUT).build();
}
@Override public void doAlarm(List<AlarmMessage> alarmMessage) {
if (remoteEndpoints.size() == 0) {
return;
}
CloseableHttpClient httpClient = HttpClients.custom().build();
try {
remoteEndpoints.forEach(url -> {
HttpPost post = new HttpPost(url);
post.setConfig(requestConfig);
post.setHeader("Accept", "application/json");
post.setHeader("Content-type", "application/json");
StringEntity entity = null;
try {
entity = new StringEntity(gson.toJson(alarmMessage));
post.setEntity(entity);
CloseableHttpResponse httpResponse = httpClient.execute(post);
StatusLine statusLine = httpResponse.getStatusLine();
if (statusLine != null && statusLine.getStatusCode() != 200) {
logger.error("send alarm to " + url + " failure. Response code: " + statusLine.getStatusCode());
}
} catch (UnsupportedEncodingException e) {
logger.error("Alarm to JSON error, " + e.getMessage(), e);
} catch (ClientProtocolException e) {
logger.error("send alarm to " + url + " failure.", e);
} catch (IOException e) {
logger.error("send alarm to " + url + " failure.", e);
}
});
} finally {
try {
httpClient.close();
} catch (IOException e) {
logger.error(e.getMessage(), e);
}
}
}
}
/*
* Licensed to the Apache Software Foundation (ASF) under one or more
* contributor license agreements. See the NOTICE file distributed with
* this work for additional information regarding copyright ownership.
* The ASF licenses this file to You under the Apache License, Version 2.0
* (the "License"); you may not use this file except in compliance with
* the License. You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*
*/
package org.apache.skywalking.oap.server.core.alarm.provider;
import org.apache.skywalking.oap.server.core.alarm.MetaInAlarm;
import org.apache.skywalking.oap.server.core.source.Scope;
import org.junit.Assert;
import org.junit.Test;
public class AlarmMessageFormatterTest {
@Test
public void testStringFormatWithNoArg() {
AlarmMessageFormatter formatter = new AlarmMessageFormatter("abc words {sdf");
String message = formatter.format(new MetaInAlarm() {
@Override public Scope getScope() {
return null;
}
@Override public String getName() {
return null;
}
@Override public String getIndicatorName() {
return null;
}
@Override public int getId0() {
return 0;
}
@Override public int getId1() {
return 0;
}
});
Assert.assertEquals("abc words {sdf", message);
}
@Test
public void testStringFormatWithArg() {
AlarmMessageFormatter formatter = new AlarmMessageFormatter("abc} words {name} - {id} .. {");
String message = formatter.format(new MetaInAlarm() {
@Override public Scope getScope() {
return null;
}
@Override public String getName() {
return "service";
}
@Override public String getIndicatorName() {
return null;
}
@Override public int getId0() {
return 1290;
}
@Override public int getId1() {
return 0;
}
});
Assert.assertEquals("abc} words service - 1290 .. {", message);
}
}
......@@ -34,11 +34,14 @@ public class AlarmRuleInitTest {
Assert.assertEquals("85", ruleList.get(1).getThreshold());
Assert.assertEquals("endpoint_percent_rule", ruleList.get(0).getAlarmRuleName());
Assert.assertEquals(0, ruleList.get(0).getIncludeNames().size());
Assert.assertEquals("Successful rate of endpoint {name} is lower than 75%", ruleList.get(0).getMessage());
Assert.assertEquals("service_b", ruleList.get(1).getIncludeNames().get(1));
Assert.assertEquals("Alarm caused by Rule service_percent_rule", ruleList.get(1).getMessage());
List<String> rulesWebhooks = rules.getWebhooks();
Assert.assertEquals(2, rulesWebhooks.size());
Assert.assertEquals("http://127.0.0.1/go-wechat/", rulesWebhooks.get(1));
}
}
......@@ -82,6 +82,7 @@ public class RunningRuleTest {
alarmRule.setThreshold("75");
alarmRule.setCount(3);
alarmRule.setPeriod(15);
alarmRule.setMessage("Successful rate of endpoint {name} is lower than 75%");
RunningRule runningRule = new RunningRule(alarmRule);
LocalDateTime startTime = TIME_BUCKET_FORMATTER.parseLocalDateTime("201808301440");
......@@ -95,14 +96,17 @@ public class RunningRuleTest {
runningRule.in(getMetaInAlarm(123), getIndicator(timeInPeriod3, 74));
// check at 201808301440
Assert.assertEquals(0, runningRule.check().size());
List<AlarmMessage> alarmMessages = runningRule.check();
Assert.assertEquals(0, alarmMessages.size());
runningRule.moveTo(TIME_BUCKET_FORMATTER.parseLocalDateTime("201808301441"));
// check at 201808301441
Assert.assertEquals(0, runningRule.check().size());
alarmMessages = runningRule.check();
Assert.assertEquals(0, alarmMessages.size());
runningRule.moveTo(TIME_BUCKET_FORMATTER.parseLocalDateTime("201808301442"));
// check at 201808301442
Assert.assertEquals(1, runningRule.check().size());
alarmMessages = runningRule.check();
Assert.assertEquals(1, alarmMessages.size());
Assert.assertEquals("Successful rate of endpoint Service_123 is lower than 75%", alarmMessages.get(0).getAlarmMessage());
}
@Test
......
/*
* Licensed to the Apache Software Foundation (ASF) under one or more
* contributor license agreements. See the NOTICE file distributed with
* this work for additional information regarding copyright ownership.
* The ASF licenses this file to You under the Apache License, Version 2.0
* (the "License"); you may not use this file except in compliance with
* the License. You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*
*/
package org.apache.skywalking.oap.server.core.alarm.provider;
import com.google.gson.Gson;
import com.google.gson.JsonArray;
import java.io.ByteArrayOutputStream;
import java.io.IOException;
import java.io.InputStream;
import java.net.InetSocketAddress;
import java.util.ArrayList;
import java.util.List;
import javax.servlet.Servlet;
import javax.servlet.ServletConfig;
import javax.servlet.ServletException;
import javax.servlet.ServletRequest;
import javax.servlet.ServletResponse;
import javax.servlet.http.HttpServletRequest;
import javax.servlet.http.HttpServletResponse;
import org.apache.skywalking.oap.server.core.alarm.AlarmMessage;
import org.eclipse.jetty.server.Server;
import org.eclipse.jetty.servlet.ServletContextHandler;
import org.eclipse.jetty.servlet.ServletHolder;
import org.junit.After;
import org.junit.Assert;
import org.junit.Before;
import org.junit.Test;
public class WebhookCallbackTest implements Servlet {
private Server server;
private volatile boolean isSuccess = false;
@Before
public void init() throws Exception {
server = new Server(new InetSocketAddress("127.0.0.1", 8778));
ServletContextHandler servletContextHandler = new ServletContextHandler(ServletContextHandler.NO_SESSIONS);
servletContextHandler.setContextPath("/webhook");
server.setHandler(servletContextHandler);
ServletHolder servletHolder = new ServletHolder();
servletHolder.setServlet(this);
servletContextHandler.addServlet(servletHolder, "/receiveAlarm");
server.start();
}
@After
public void stop() throws Exception {
server.stop();
}
@Test
public void testWebhook() {
List<String> remoteEndpoints = new ArrayList<>();
remoteEndpoints.add("http://127.0.0.1:8778/webhook/receiveAlarm");
WebhookCallback webhookCallback = new WebhookCallback(remoteEndpoints);
List<AlarmMessage> alarmMessages = new ArrayList<>(2);
alarmMessages.add(new AlarmMessage());
alarmMessages.add(new AlarmMessage());
webhookCallback.doAlarm(alarmMessages);
Assert.assertTrue(isSuccess);
}
@Override public void init(ServletConfig config) throws ServletException {
}
@Override public ServletConfig getServletConfig() {
return null;
}
@Override
public void service(ServletRequest request, ServletResponse response) throws ServletException, IOException {
HttpServletRequest httpServletRequest = (HttpServletRequest)request;
if (httpServletRequest.getContentType().equals("application/json")) {
InputStream inputStream = request.getInputStream();
ByteArrayOutputStream out = new ByteArrayOutputStream();
byte[] buffer = new byte[2048];
int readCntOnce;
while ((readCntOnce = inputStream.read(buffer)) >= 0) {
out.write(buffer, 0, readCntOnce);
}
JsonArray elements = new Gson().fromJson(new String(out.toByteArray()), JsonArray.class);
if (elements.size() == 2) {
((HttpServletResponse)response).setStatus(200);
isSuccess = true;
return;
}
((HttpServletResponse)response).setStatus(500);
}
}
@Override public String getServletInfo() {
return null;
}
@Override public void destroy() {
}
}
......@@ -27,6 +27,7 @@ rules:
count: 3
# How many times of checks, the alarm keeps silence after alarm triggered, default as same as period.
silence-period: 10
message: Successful rate of endpoint {name} is lower than 75%
service_percent_rule:
indicator-name: service_percent
# [Optional] Default, match all services in this indicator
......
......@@ -21,6 +21,7 @@ package org.apache.skywalking.oap.server.core.alarm;
import lombok.AccessLevel;
import lombok.Getter;
import lombok.Setter;
import org.apache.skywalking.oap.server.core.source.Scope;
/**
* Alarm message represents the details of each alarm.
......@@ -32,6 +33,12 @@ import lombok.Setter;
public class AlarmMessage {
public static AlarmMessage NONE = new NoAlarm();
private Scope scope;
private String name;
private int id0;
private int id1;
private String alarmMessage;
private static class NoAlarm extends AlarmMessage {
}
......
Markdown is supported
0% .
You are about to add 0 people to the discussion. Proceed with caution.
先完成此消息的编辑!
想要评论请 注册