Skip to content
体验新版
项目
组织
正在加载...
登录
切换导航
打开侧边栏
killuaz丶x
SkyWalking
提交
982001c6
S
SkyWalking
项目概览
killuaz丶x
/
SkyWalking
与 Fork 源项目一致
Fork自
apache / SkyWalking
通知
1
Star
0
Fork
0
代码
文件
提交
分支
Tags
贡献者
分支图
Diff
Issue
0
列表
看板
标记
里程碑
合并请求
0
Wiki
0
Wiki
分析
仓库
DevOps
项目成员
Pages
S
SkyWalking
项目概览
项目概览
详情
发布
仓库
仓库
文件
提交
分支
标签
贡献者
分支图
比较
Issue
0
Issue
0
列表
看板
标记
里程碑
合并请求
0
合并请求
0
Pages
分析
分析
仓库分析
DevOps
Wiki
0
Wiki
成员
成员
收起侧边栏
关闭侧边栏
动态
分支图
创建新Issue
提交
Issue看板
体验新版 GitCode,发现更多精彩内容 >>
未验证
提交
982001c6
编写于
7月 13, 2021
作者:
K
kezhenxu94
提交者:
GitHub
7月 13, 2021
浏览文件
操作
浏览文件
下载
电子邮件补丁
差异文件
perf: optimize Envoy access log persistence performance (#7290)
上级
6228d450
变更
18
隐藏空白更改
内联
并排
Showing
18 changed file
with
117 addition
and
66 deletion
+117
-66
CHANGES.md
CHANGES.md
+1
-0
oap-server/analyzer/log-analyzer/src/main/java/org/apache/skywalking/oap/log/analyzer/dsl/Binding.java
...a/org/apache/skywalking/oap/log/analyzer/dsl/Binding.java
+42
-5
oap-server/analyzer/log-analyzer/src/main/java/org/apache/skywalking/oap/log/analyzer/dsl/spec/filter/FilterSpec.java
...ywalking/oap/log/analyzer/dsl/spec/filter/FilterSpec.java
+3
-1
oap-server/analyzer/log-analyzer/src/main/java/org/apache/skywalking/oap/log/analyzer/provider/log/ILogAnalyzerService.java
...ng/oap/log/analyzer/provider/log/ILogAnalyzerService.java
+4
-3
oap-server/analyzer/log-analyzer/src/main/java/org/apache/skywalking/oap/log/analyzer/provider/log/LogAnalyzer.java
...skywalking/oap/log/analyzer/provider/log/LogAnalyzer.java
+5
-4
oap-server/analyzer/log-analyzer/src/main/java/org/apache/skywalking/oap/log/analyzer/provider/log/LogAnalyzerServiceImpl.java
...oap/log/analyzer/provider/log/LogAnalyzerServiceImpl.java
+3
-2
oap-server/analyzer/log-analyzer/src/main/java/org/apache/skywalking/oap/log/analyzer/provider/log/analyzer/LogAnalyzerFactory.java
...og/analyzer/provider/log/analyzer/LogAnalyzerFactory.java
+22
-0
oap-server/analyzer/log-analyzer/src/main/java/org/apache/skywalking/oap/log/analyzer/provider/log/listener/LogAnalysisListener.java
...g/analyzer/provider/log/listener/LogAnalysisListener.java
+2
-1
oap-server/analyzer/log-analyzer/src/main/java/org/apache/skywalking/oap/log/analyzer/provider/log/listener/LogFilterListener.java
...log/analyzer/provider/log/listener/LogFilterListener.java
+5
-2
oap-server/analyzer/log-analyzer/src/main/java/org/apache/skywalking/oap/log/analyzer/provider/log/listener/RecordAnalysisListener.java
...nalyzer/provider/log/listener/RecordAnalysisListener.java
+10
-1
oap-server/analyzer/log-analyzer/src/main/java/org/apache/skywalking/oap/log/analyzer/provider/log/listener/TrafficAnalysisListener.java
...alyzer/provider/log/listener/TrafficAnalysisListener.java
+3
-1
oap-server/server-bootstrap/src/main/resources/lal/envoy-als.yaml
...er/server-bootstrap/src/main/resources/lal/envoy-als.yaml
+5
-7
oap-server/server-core/src/main/java/org/apache/skywalking/oap/server/core/analysis/manual/log/AbstractLogRecord.java
...ap/server/core/analysis/manual/log/AbstractLogRecord.java
+1
-0
oap-server/server-fetcher-plugin/kafka-fetcher-plugin/src/main/java/org/apache/skywalking/oap/server/analyzer/agent/kafka/provider/handler/LogHandler.java
...ver/analyzer/agent/kafka/provider/handler/LogHandler.java
+1
-1
oap-server/server-receiver-plugin/envoy-metrics-receiver-plugin/src/main/java/org/apache/skywalking/oap/server/receiver/envoy/persistence/LogsPersistence.java
...ap/server/receiver/envoy/persistence/LogsPersistence.java
+4
-18
oap-server/server-receiver-plugin/envoy-metrics-receiver-plugin/src/main/java/org/apache/skywalking/oap/server/receiver/envoy/persistence/TCPLogsPersistence.java
...server/receiver/envoy/persistence/TCPLogsPersistence.java
+4
-18
oap-server/server-receiver-plugin/skywalking-log-recevier-plugin/src/main/java/org/apache/skywalking/oap/server/recevier/log/provider/handler/grpc/LogReportServiceHandler.java
...er/log/provider/handler/grpc/LogReportServiceHandler.java
+1
-1
oap-server/server-receiver-plugin/skywalking-log-recevier-plugin/src/main/java/org/apache/skywalking/oap/server/recevier/log/provider/handler/rest/LogReportServiceRestHandler.java
...og/provider/handler/rest/LogReportServiceRestHandler.java
+1
-1
未找到文件。
CHANGES.md
浏览文件 @
982001c6
...
@@ -80,6 +80,7 @@ Release Notes.
...
@@ -80,6 +80,7 @@ Release Notes.
*
Add thread state metric and class loaded info metric to JVMMetric.
*
Add thread state metric and class loaded info metric to JVMMetric.
*
Performance: compile LAL DSL statically and run with type checked.
*
Performance: compile LAL DSL statically and run with type checked.
*
Add pagination to event query protocol.
*
Add pagination to event query protocol.
*
Performance: optimize Envoy error logs persistence performance.
#### UI
#### UI
...
...
oap-server/analyzer/log-analyzer/src/main/java/org/apache/skywalking/oap/log/analyzer/dsl/Binding.java
浏览文件 @
982001c6
...
@@ -18,6 +18,10 @@
...
@@ -18,6 +18,10 @@
package
org.apache.skywalking.oap.log.analyzer.dsl
;
package
org.apache.skywalking.oap.log.analyzer.dsl
;
import
com.google.protobuf.Message
;
import
groovy.lang.Closure
;
import
groovy.lang.GroovyObjectSupport
;
import
groovy.lang.MissingPropertyException
;
import
java.util.Map
;
import
java.util.Map
;
import
java.util.regex.Matcher
;
import
java.util.regex.Matcher
;
import
lombok.Getter
;
import
lombok.Getter
;
...
@@ -44,6 +48,7 @@ public class Binding extends groovy.lang.Binding {
...
@@ -44,6 +48,7 @@ public class Binding extends groovy.lang.Binding {
setProperty
(
KEY_LOG
,
log
);
setProperty
(
KEY_LOG
,
log
);
setProperty
(
KEY_SAVE
,
true
);
setProperty
(
KEY_SAVE
,
true
);
setProperty
(
KEY_ABORT
,
false
);
setProperty
(
KEY_ABORT
,
false
);
parsed
().
log
=
log
;
return
this
;
return
this
;
}
}
...
@@ -55,6 +60,15 @@ public class Binding extends groovy.lang.Binding {
...
@@ -55,6 +60,15 @@ public class Binding extends groovy.lang.Binding {
return
(
LogData
.
Builder
)
getProperty
(
KEY_LOG
);
return
(
LogData
.
Builder
)
getProperty
(
KEY_LOG
);
}
}
public
Binding
extraLog
(
final
Message
extraLog
)
{
parsed
().
extraLog
=
extraLog
;
return
this
;
}
public
Message
extraLog
()
{
return
parsed
().
getExtraLog
();
}
public
Binding
parsed
(
final
Matcher
parsed
)
{
public
Binding
parsed
(
final
Matcher
parsed
)
{
parsed
().
matcher
=
parsed
;
parsed
().
matcher
=
parsed
;
return
this
;
return
this
;
...
@@ -92,19 +106,32 @@ public class Binding extends groovy.lang.Binding {
...
@@ -92,19 +106,32 @@ public class Binding extends groovy.lang.Binding {
return
(
boolean
)
getProperty
(
KEY_ABORT
);
return
(
boolean
)
getProperty
(
KEY_ABORT
);
}
}
public
static
class
Parsed
{
public
static
class
Parsed
extends
GroovyObjectSupport
{
@Getter
@Getter
private
Matcher
matcher
;
private
Matcher
matcher
;
@Getter
@Getter
private
Map
<
String
,
Object
>
map
;
private
Map
<
String
,
Object
>
map
;
@Getter
private
Message
.
Builder
log
;
@Getter
private
Message
extraLog
;
public
Object
getAt
(
final
String
key
)
{
public
Object
getAt
(
final
String
key
)
{
if
(
matcher
!=
null
)
{
Object
result
;
return
matcher
.
group
(
key
);
if
(
matcher
!=
null
&&
(
result
=
matcher
.
group
(
key
))
!=
null
)
{
return
result
;
}
if
(
map
!=
null
&&
(
result
=
map
.
get
(
key
))
!=
null
)
{
return
result
;
}
}
if
(
map
!=
null
)
{
if
(
extraLog
!=
null
&&
(
result
=
getField
(
extraLog
,
key
))
!=
null
)
{
return
map
.
get
(
key
);
return
result
;
}
if
(
log
!=
null
&&
(
result
=
getField
(
log
,
key
))
!=
null
)
{
return
result
;
}
}
return
null
;
return
null
;
}
}
...
@@ -113,5 +140,15 @@ public class Binding extends groovy.lang.Binding {
...
@@ -113,5 +140,15 @@ public class Binding extends groovy.lang.Binding {
public
Object
propertyMissing
(
final
String
name
)
{
public
Object
propertyMissing
(
final
String
name
)
{
return
getAt
(
name
);
return
getAt
(
name
);
}
}
static
Object
getField
(
Object
obj
,
String
name
)
{
try
{
Closure
<?>
c
=
new
Closure
<
Object
>(
obj
,
obj
)
{
};
return
c
.
getProperty
(
name
);
}
catch
(
MissingPropertyException
ignored
)
{
}
return
null
;
}
}
}
}
}
oap-server/analyzer/log-analyzer/src/main/java/org/apache/skywalking/oap/log/analyzer/dsl/spec/filter/FilterSpec.java
浏览文件 @
982001c6
...
@@ -19,6 +19,7 @@
...
@@ -19,6 +19,7 @@
package
org.apache.skywalking.oap.log.analyzer.dsl.spec.filter
;
package
org.apache.skywalking.oap.log.analyzer.dsl.spec.filter
;
import
com.fasterxml.jackson.core.type.TypeReference
;
import
com.fasterxml.jackson.core.type.TypeReference
;
import
com.google.protobuf.Message
;
import
com.google.protobuf.TextFormat
;
import
com.google.protobuf.TextFormat
;
import
groovy.lang.Closure
;
import
groovy.lang.Closure
;
import
groovy.lang.DelegatesTo
;
import
groovy.lang.DelegatesTo
;
...
@@ -153,6 +154,7 @@ public class FilterSpec extends AbstractSpec {
...
@@ -153,6 +154,7 @@ public class FilterSpec extends AbstractSpec {
final
Binding
b
=
BINDING
.
get
();
final
Binding
b
=
BINDING
.
get
();
final
LogData
.
Builder
logData
=
b
.
log
();
final
LogData
.
Builder
logData
=
b
.
log
();
final
Message
extraLog
=
b
.
extraLog
();
if
(!
b
.
shouldSave
())
{
if
(!
b
.
shouldSave
())
{
if
(
LOGGER
.
isDebugEnabled
())
{
if
(
LOGGER
.
isDebugEnabled
())
{
...
@@ -163,7 +165,7 @@ public class FilterSpec extends AbstractSpec {
...
@@ -163,7 +165,7 @@ public class FilterSpec extends AbstractSpec {
factories
.
stream
()
factories
.
stream
()
.
map
(
LogAnalysisListenerFactory:
:
create
)
.
map
(
LogAnalysisListenerFactory:
:
create
)
.
forEach
(
it
->
it
.
parse
(
logData
).
build
());
.
forEach
(
it
->
it
.
parse
(
logData
,
extraLog
).
build
());
}
}
@SuppressWarnings
(
"unused"
)
@SuppressWarnings
(
"unused"
)
...
...
oap-server/analyzer/log-analyzer/src/main/java/org/apache/skywalking/oap/log/analyzer/provider/log/ILogAnalyzerService.java
浏览文件 @
982001c6
...
@@ -17,6 +17,7 @@
...
@@ -17,6 +17,7 @@
package
org.apache.skywalking.oap.log.analyzer.provider.log
;
package
org.apache.skywalking.oap.log.analyzer.provider.log
;
import
com.google.protobuf.Message
;
import
org.apache.skywalking.apm.network.logging.v3.LogData
;
import
org.apache.skywalking.apm.network.logging.v3.LogData
;
import
org.apache.skywalking.oap.server.library.module.Service
;
import
org.apache.skywalking.oap.server.library.module.Service
;
...
@@ -25,10 +26,10 @@ import org.apache.skywalking.oap.server.library.module.Service;
...
@@ -25,10 +26,10 @@ import org.apache.skywalking.oap.server.library.module.Service;
*/
*/
public
interface
ILogAnalyzerService
extends
Service
{
public
interface
ILogAnalyzerService
extends
Service
{
void
doAnalysis
(
LogData
.
Builder
log
);
void
doAnalysis
(
LogData
.
Builder
log
,
Message
extraLog
);
default
void
doAnalysis
(
LogData
logData
)
{
default
void
doAnalysis
(
LogData
logData
,
Message
extraLog
)
{
doAnalysis
(
logData
.
toBuilder
());
doAnalysis
(
logData
.
toBuilder
()
,
extraLog
);
}
}
}
}
oap-server/analyzer/log-analyzer/src/main/java/org/apache/skywalking/oap/log/analyzer/provider/log/LogAnalyzer.java
浏览文件 @
982001c6
...
@@ -17,6 +17,7 @@
...
@@ -17,6 +17,7 @@
package
org.apache.skywalking.oap.log.analyzer.provider.log
;
package
org.apache.skywalking.oap.log.analyzer.provider.log
;
import
com.google.protobuf.Message
;
import
java.util.ArrayList
;
import
java.util.ArrayList
;
import
java.util.List
;
import
java.util.List
;
import
lombok.RequiredArgsConstructor
;
import
lombok.RequiredArgsConstructor
;
...
@@ -39,7 +40,7 @@ public class LogAnalyzer {
...
@@ -39,7 +40,7 @@ public class LogAnalyzer {
private
final
List
<
LogAnalysisListener
>
listeners
=
new
ArrayList
<>();
private
final
List
<
LogAnalysisListener
>
listeners
=
new
ArrayList
<>();
public
void
doAnalysis
(
LogData
.
Builder
builder
)
{
public
void
doAnalysis
(
LogData
.
Builder
builder
,
Message
extraLog
)
{
if
(
StringUtil
.
isEmpty
(
builder
.
getService
()))
{
if
(
StringUtil
.
isEmpty
(
builder
.
getService
()))
{
// If the service name is empty, the log will be ignored.
// If the service name is empty, the log will be ignored.
log
.
debug
(
"The log is ignored because the Service name is empty"
);
log
.
debug
(
"The log is ignored because the Service name is empty"
);
...
@@ -51,12 +52,12 @@ public class LogAnalyzer {
...
@@ -51,12 +52,12 @@ public class LogAnalyzer {
builder
.
setTimestamp
(
System
.
currentTimeMillis
());
builder
.
setTimestamp
(
System
.
currentTimeMillis
());
}
}
notifyListener
(
builder
);
notifyListener
(
builder
,
extraLog
);
notifyListenerToBuild
();
notifyListenerToBuild
();
}
}
private
void
notifyListener
(
LogData
.
Builder
builder
)
{
private
void
notifyListener
(
LogData
.
Builder
builder
,
final
Message
extraLog
)
{
listeners
.
forEach
(
listener
->
listener
.
parse
(
builder
));
listeners
.
forEach
(
listener
->
listener
.
parse
(
builder
,
extraLog
));
}
}
private
void
notifyListenerToBuild
()
{
private
void
notifyListenerToBuild
()
{
...
...
oap-server/analyzer/log-analyzer/src/main/java/org/apache/skywalking/oap/log/analyzer/provider/log/LogAnalyzerServiceImpl.java
浏览文件 @
982001c6
...
@@ -17,6 +17,7 @@
...
@@ -17,6 +17,7 @@
package
org.apache.skywalking.oap.log.analyzer.provider.log
;
package
org.apache.skywalking.oap.log.analyzer.provider.log
;
import
com.google.protobuf.Message
;
import
java.util.ArrayList
;
import
java.util.ArrayList
;
import
java.util.List
;
import
java.util.List
;
import
lombok.RequiredArgsConstructor
;
import
lombok.RequiredArgsConstructor
;
...
@@ -32,9 +33,9 @@ public class LogAnalyzerServiceImpl implements ILogAnalyzerService, ILogAnalysis
...
@@ -32,9 +33,9 @@ public class LogAnalyzerServiceImpl implements ILogAnalyzerService, ILogAnalysis
private
final
List
<
LogAnalysisListenerFactory
>
factories
=
new
ArrayList
<>();
private
final
List
<
LogAnalysisListenerFactory
>
factories
=
new
ArrayList
<>();
@Override
@Override
public
void
doAnalysis
(
final
LogData
.
Builder
log
)
{
public
void
doAnalysis
(
final
LogData
.
Builder
log
,
Message
extraLog
)
{
LogAnalyzer
analyzer
=
new
LogAnalyzer
(
moduleManager
,
moduleConfig
,
this
);
LogAnalyzer
analyzer
=
new
LogAnalyzer
(
moduleManager
,
moduleConfig
,
this
);
analyzer
.
doAnalysis
(
log
);
analyzer
.
doAnalysis
(
log
,
extraLog
);
}
}
@Override
@Override
...
...
oap-server/analyzer/log-analyzer/src/main/java/org/apache/skywalking/oap/log/analyzer/provider/log/analyzer/LogAnalyzerFactory.java
0 → 100644
浏览文件 @
982001c6
/*
* Licensed to the Apache Software Foundation (ASF) under one or more
* contributor license agreements. See the NOTICE file distributed with
* this work for additional information regarding copyright ownership.
* The ASF licenses this file to You under the Apache License, Version 2.0
* (the "License"); you may not use this file except in compliance with
* the License. You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package
org.apache.skywalking.oap.log.analyzer.provider.log.analyzer
;
public
class
LogAnalyzerFactory
{
}
oap-server/analyzer/log-analyzer/src/main/java/org/apache/skywalking/oap/log/analyzer/provider/log/listener/LogAnalysisListener.java
浏览文件 @
982001c6
...
@@ -17,6 +17,7 @@
...
@@ -17,6 +17,7 @@
package
org.apache.skywalking.oap.log.analyzer.provider.log.listener
;
package
org.apache.skywalking.oap.log.analyzer.provider.log.listener
;
import
com.google.protobuf.Message
;
import
org.apache.skywalking.apm.network.logging.v3.LogData
;
import
org.apache.skywalking.apm.network.logging.v3.LogData
;
/**
/**
...
@@ -33,5 +34,5 @@ public interface LogAnalysisListener {
...
@@ -33,5 +34,5 @@ public interface LogAnalysisListener {
* Parse the raw data from the probe.
* Parse the raw data from the probe.
* @return {@code this} for chaining.
* @return {@code this} for chaining.
*/
*/
LogAnalysisListener
parse
(
LogData
.
Builder
logData
);
LogAnalysisListener
parse
(
LogData
.
Builder
logData
,
final
Message
extraLog
);
}
}
oap-server/analyzer/log-analyzer/src/main/java/org/apache/skywalking/oap/log/analyzer/provider/log/listener/LogFilterListener.java
浏览文件 @
982001c6
...
@@ -18,6 +18,7 @@
...
@@ -18,6 +18,7 @@
package
org.apache.skywalking.oap.log.analyzer.provider.log.listener
;
package
org.apache.skywalking.oap.log.analyzer.provider.log.listener
;
import
com.google.protobuf.Message
;
import
java.util.ArrayList
;
import
java.util.ArrayList
;
import
java.util.List
;
import
java.util.List
;
import
java.util.stream.Collectors
;
import
java.util.stream.Collectors
;
...
@@ -48,8 +49,10 @@ public class LogFilterListener implements LogAnalysisListener {
...
@@ -48,8 +49,10 @@ public class LogFilterListener implements LogAnalysisListener {
}
}
@Override
@Override
public
LogAnalysisListener
parse
(
final
LogData
.
Builder
logData
)
{
public
LogAnalysisListener
parse
(
final
LogData
.
Builder
logData
,
dsls
.
forEach
(
dsl
->
dsl
.
bind
(
new
Binding
().
log
(
logData
.
build
())));
final
Message
extraLog
)
{
dsls
.
forEach
(
dsl
->
dsl
.
bind
(
new
Binding
().
log
(
logData
.
build
())
.
extraLog
(
extraLog
)));
return
this
;
return
this
;
}
}
...
...
oap-server/analyzer/log-analyzer/src/main/java/org/apache/skywalking/oap/log/analyzer/provider/log/listener/RecordAnalysisListener.java
浏览文件 @
982001c6
...
@@ -17,12 +17,14 @@
...
@@ -17,12 +17,14 @@
package
org.apache.skywalking.oap.log.analyzer.provider.log.listener
;
package
org.apache.skywalking.oap.log.analyzer.provider.log.listener
;
import
com.google.protobuf.Message
;
import
java.util.Arrays
;
import
java.util.Arrays
;
import
java.util.Collection
;
import
java.util.Collection
;
import
java.util.HashSet
;
import
java.util.HashSet
;
import
java.util.List
;
import
java.util.List
;
import
java.util.UUID
;
import
java.util.UUID
;
import
lombok.RequiredArgsConstructor
;
import
lombok.RequiredArgsConstructor
;
import
lombok.SneakyThrows
;
import
org.apache.skywalking.apm.network.logging.v3.LogData
;
import
org.apache.skywalking.apm.network.logging.v3.LogData
;
import
org.apache.skywalking.apm.network.logging.v3.LogDataBody
;
import
org.apache.skywalking.apm.network.logging.v3.LogDataBody
;
import
org.apache.skywalking.apm.network.logging.v3.TraceContext
;
import
org.apache.skywalking.apm.network.logging.v3.TraceContext
;
...
@@ -41,6 +43,8 @@ import org.apache.skywalking.oap.server.core.source.Log;
...
@@ -41,6 +43,8 @@ import org.apache.skywalking.oap.server.core.source.Log;
import
org.apache.skywalking.oap.server.core.source.SourceReceiver
;
import
org.apache.skywalking.oap.server.core.source.SourceReceiver
;
import
org.apache.skywalking.oap.server.library.module.ModuleManager
;
import
org.apache.skywalking.oap.server.library.module.ModuleManager
;
import
static
org
.
apache
.
skywalking
.
oap
.
server
.
library
.
util
.
ProtoBufJsonUtils
.
toJSON
;
/**
/**
* RecordAnalysisListener forwards the log data to the persistence layer with the query required conditions.
* RecordAnalysisListener forwards the log data to the persistence layer with the query required conditions.
*/
*/
...
@@ -57,7 +61,9 @@ public class RecordAnalysisListener implements LogAnalysisListener {
...
@@ -57,7 +61,9 @@ public class RecordAnalysisListener implements LogAnalysisListener {
}
}
@Override
@Override
public
LogAnalysisListener
parse
(
final
LogData
.
Builder
logData
)
{
@SneakyThrows
public
LogAnalysisListener
parse
(
final
LogData
.
Builder
logData
,
final
Message
extraLog
)
{
LogDataBody
body
=
logData
.
getBody
();
LogDataBody
body
=
logData
.
getBody
();
log
.
setUniqueId
(
UUID
.
randomUUID
().
toString
().
replace
(
"-"
,
""
));
log
.
setUniqueId
(
UUID
.
randomUUID
().
toString
().
replace
(
"-"
,
""
));
// timestamp
// timestamp
...
@@ -100,6 +106,9 @@ public class RecordAnalysisListener implements LogAnalysisListener {
...
@@ -100,6 +106,9 @@ public class RecordAnalysisListener implements LogAnalysisListener {
}
else
if
(
body
.
hasJson
())
{
}
else
if
(
body
.
hasJson
())
{
log
.
setContentType
(
ContentType
.
JSON
);
log
.
setContentType
(
ContentType
.
JSON
);
log
.
setContent
(
body
.
getJson
().
getJson
());
log
.
setContent
(
body
.
getJson
().
getJson
());
}
else
if
(
extraLog
!=
null
)
{
log
.
setContentType
(
ContentType
.
JSON
);
log
.
setContent
(
toJSON
(
extraLog
));
}
}
if
(
logData
.
getTags
().
getDataCount
()
>
0
)
{
if
(
logData
.
getTags
().
getDataCount
()
>
0
)
{
log
.
setTagsRawData
(
logData
.
getTags
().
toByteArray
());
log
.
setTagsRawData
(
logData
.
getTags
().
toByteArray
());
...
...
oap-server/analyzer/log-analyzer/src/main/java/org/apache/skywalking/oap/log/analyzer/provider/log/listener/TrafficAnalysisListener.java
浏览文件 @
982001c6
...
@@ -17,6 +17,7 @@
...
@@ -17,6 +17,7 @@
package
org.apache.skywalking.oap.log.analyzer.provider.log.listener
;
package
org.apache.skywalking.oap.log.analyzer.provider.log.listener
;
import
com.google.protobuf.Message
;
import
lombok.RequiredArgsConstructor
;
import
lombok.RequiredArgsConstructor
;
import
org.apache.skywalking.apm.network.logging.v3.LogData
;
import
org.apache.skywalking.apm.network.logging.v3.LogData
;
import
org.apache.skywalking.apm.util.StringUtil
;
import
org.apache.skywalking.apm.util.StringUtil
;
...
@@ -61,7 +62,8 @@ public class TrafficAnalysisListener implements LogAnalysisListener {
...
@@ -61,7 +62,8 @@ public class TrafficAnalysisListener implements LogAnalysisListener {
}
}
@Override
@Override
public
LogAnalysisListener
parse
(
final
LogData
.
Builder
logData
)
{
public
LogAnalysisListener
parse
(
final
LogData
.
Builder
logData
,
final
Message
extraLog
)
{
final
long
timeBucket
=
TimeBucket
.
getTimeBucket
(
System
.
currentTimeMillis
(),
DownSampling
.
Minute
);
final
long
timeBucket
=
TimeBucket
.
getTimeBucket
(
System
.
currentTimeMillis
(),
DownSampling
.
Minute
);
// to service traffic
// to service traffic
String
serviceName
=
namingControl
.
formatServiceName
(
logData
.
getService
());
String
serviceName
=
namingControl
.
formatServiceName
(
logData
.
getService
());
...
...
oap-server/server-bootstrap/src/main/resources/lal/envoy-als.yaml
浏览文件 @
982001c6
...
@@ -17,24 +17,22 @@ rules:
...
@@ -17,24 +17,22 @@ rules:
-
name
:
envoy-als
-
name
:
envoy-als
dsl
:
|
dsl
:
|
filter {
filter {
json {
}
// only collect abnormal logs (http status code >= 300, or commonProperties?.responseFlags is not empty)
// only collect abnormal logs (http status code >= 300, or commonProperties?.responseFlags is not empty)
if (parsed?.response?.responseCode
as Integer < 400 && !parsed?.commonProperties?.responseFlags
) {
if (parsed?.response?.responseCode
?.value as Integer < 400 && !parsed?.commonProperties?.responseFlags?.toString()?.trim()
) {
abort {}
abort {}
}
}
extractor {
extractor {
if (parsed?.response?.responseCode) {
if (parsed?.response?.responseCode) {
tag 'status.code': parsed?.response?.responseCode
as int
tag 'status.code': parsed?.response?.responseCode
?.value
}
}
tag 'response.flag':
(parsed?.commonProperties?.responseFlags as Map)?.keySet()
tag 'response.flag':
parsed?.commonProperties?.responseFlags
}
}
sink {
sink {
sampler {
sampler {
if (parsed?.commonProperties?.responseFlags) {
if (parsed?.commonProperties?.responseFlags
?.toString()
) {
// use service:errorCode as sampler id so that each service:errorCode has its own sampler,
// use service:errorCode as sampler id so that each service:errorCode has its own sampler,
// e.g. checkoutservice:[upstreamConnectionFailure], checkoutservice:[upstreamRetryLimitExceeded]
// e.g. checkoutservice:[upstreamConnectionFailure], checkoutservice:[upstreamRetryLimitExceeded]
rateLimit("${log.service}:${
(parsed?.commonProperties?.responseFlags as Map)?.keySet
()}") {
rateLimit("${log.service}:${
parsed?.commonProperties?.responseFlags?.toString
()}") {
qps 100
qps 100
}
}
} else {
} else {
...
...
oap-server/server-core/src/main/java/org/apache/skywalking/oap/server/core/analysis/manual/log/AbstractLogRecord.java
浏览文件 @
982001c6
...
@@ -44,6 +44,7 @@ public abstract class AbstractLogRecord extends Record {
...
@@ -44,6 +44,7 @@ public abstract class AbstractLogRecord extends Record {
public
static
final
String
SPAN_ID
=
"span_id"
;
public
static
final
String
SPAN_ID
=
"span_id"
;
public
static
final
String
CONTENT_TYPE
=
"content_type"
;
public
static
final
String
CONTENT_TYPE
=
"content_type"
;
public
static
final
String
CONTENT
=
"content"
;
public
static
final
String
CONTENT
=
"content"
;
public
static
final
String
CONTENT_TYPE_CLASS
=
"content_type_class"
;
public
static
final
String
TAGS_RAW_DATA
=
"tags_raw_data"
;
public
static
final
String
TAGS_RAW_DATA
=
"tags_raw_data"
;
public
static
final
String
TIMESTAMP
=
"timestamp"
;
public
static
final
String
TIMESTAMP
=
"timestamp"
;
public
static
final
String
TAGS
=
"tags"
;
public
static
final
String
TAGS
=
"tags"
;
...
...
oap-server/server-fetcher-plugin/kafka-fetcher-plugin/src/main/java/org/apache/skywalking/oap/server/analyzer/agent/kafka/provider/handler/LogHandler.java
浏览文件 @
982001c6
...
@@ -77,7 +77,7 @@ public class LogHandler implements KafkaHandler {
...
@@ -77,7 +77,7 @@ public class LogHandler implements KafkaHandler {
public
void
handle
(
final
ConsumerRecord
<
String
,
Bytes
>
record
)
{
public
void
handle
(
final
ConsumerRecord
<
String
,
Bytes
>
record
)
{
try
(
HistogramMetrics
.
Timer
ignore
=
histogram
.
createTimer
())
{
try
(
HistogramMetrics
.
Timer
ignore
=
histogram
.
createTimer
())
{
LogData
logData
=
parseConsumerRecord
(
record
);
LogData
logData
=
parseConsumerRecord
(
record
);
logAnalyzerService
.
doAnalysis
(
logData
);
logAnalyzerService
.
doAnalysis
(
logData
,
null
);
}
catch
(
Exception
e
)
{
}
catch
(
Exception
e
)
{
errorCounter
.
inc
();
errorCounter
.
inc
();
log
.
error
(
e
.
getMessage
(),
e
);
log
.
error
(
e
.
getMessage
(),
e
);
...
...
oap-server/server-receiver-plugin/envoy-metrics-receiver-plugin/src/main/java/org/apache/skywalking/oap/server/receiver/envoy/persistence/LogsPersistence.java
浏览文件 @
982001c6
...
@@ -21,9 +21,7 @@ package org.apache.skywalking.oap.server.receiver.envoy.persistence;
...
@@ -21,9 +21,7 @@ package org.apache.skywalking.oap.server.receiver.envoy.persistence;
import
io.envoyproxy.envoy.data.accesslog.v3.HTTPAccessLogEntry
;
import
io.envoyproxy.envoy.data.accesslog.v3.HTTPAccessLogEntry
;
import
io.envoyproxy.envoy.service.accesslog.v3.StreamAccessLogsMessage
;
import
io.envoyproxy.envoy.service.accesslog.v3.StreamAccessLogsMessage
;
import
lombok.extern.slf4j.Slf4j
;
import
lombok.extern.slf4j.Slf4j
;
import
org.apache.skywalking.apm.network.logging.v3.JSONLog
;
import
org.apache.skywalking.apm.network.logging.v3.LogData
;
import
org.apache.skywalking.apm.network.logging.v3.LogData
;
import
org.apache.skywalking.apm.network.logging.v3.LogDataBody
;
import
org.apache.skywalking.apm.network.servicemesh.v3.ServiceMeshMetric
;
import
org.apache.skywalking.apm.network.servicemesh.v3.ServiceMeshMetric
;
import
org.apache.skywalking.oap.log.analyzer.module.LogAnalyzerModule
;
import
org.apache.skywalking.oap.log.analyzer.module.LogAnalyzerModule
;
import
org.apache.skywalking.oap.log.analyzer.provider.log.ILogAnalyzerService
;
import
org.apache.skywalking.oap.log.analyzer.provider.log.ILogAnalyzerService
;
...
@@ -35,8 +33,6 @@ import org.apache.skywalking.oap.server.receiver.envoy.als.LogEntry2MetricsAdapt
...
@@ -35,8 +33,6 @@ import org.apache.skywalking.oap.server.receiver.envoy.als.LogEntry2MetricsAdapt
import
org.apache.skywalking.oap.server.receiver.envoy.als.Role
;
import
org.apache.skywalking.oap.server.receiver.envoy.als.Role
;
import
org.apache.skywalking.oap.server.receiver.envoy.als.ServiceMetaInfo
;
import
org.apache.skywalking.oap.server.receiver.envoy.als.ServiceMetaInfo
;
import
static
org
.
apache
.
skywalking
.
oap
.
server
.
library
.
util
.
ProtoBufJsonUtils
.
toJSON
;
/**
/**
* {@code LogsPersistence} analyzes the error logs and persists them to the log system.
* {@code LogsPersistence} analyzes the error logs and persists them to the log system.
*/
*/
...
@@ -69,7 +65,7 @@ public class LogsPersistence implements ALSHTTPAnalysis {
...
@@ -69,7 +65,7 @@ public class LogsPersistence implements ALSHTTPAnalysis {
}
}
final
LogData
logData
=
convertToLogData
(
entry
,
result
);
final
LogData
logData
=
convertToLogData
(
entry
,
result
);
logAnalyzerService
.
doAnalysis
(
logData
);
logAnalyzerService
.
doAnalysis
(
logData
,
entry
);
}
catch
(
final
Exception
e
)
{
}
catch
(
final
Exception
e
)
{
log
.
error
(
"Failed to persist Envoy access log"
,
e
);
log
.
error
(
"Failed to persist Envoy access log"
,
e
);
}
}
...
@@ -81,27 +77,17 @@ public class LogsPersistence implements ALSHTTPAnalysis {
...
@@ -81,27 +77,17 @@ public class LogsPersistence implements ALSHTTPAnalysis {
return
prev
;
return
prev
;
}
}
public
LogData
convertToLogData
(
final
HTTPAccessLogEntry
logEntry
,
public
LogData
convertToLogData
(
final
HTTPAccessLogEntry
logEntry
,
final
Result
result
)
{
final
Result
result
)
throws
Exception
{
final
ServiceMetaInfo
service
=
result
.
getService
();
final
ServiceMetaInfo
service
=
result
.
getService
();
final
ServiceMeshMetric
.
Builder
metrics
=
new
LogEntry2MetricsAdapter
(
logEntry
,
null
,
null
).
adaptCommonPart
();
final
ServiceMeshMetric
.
Builder
metrics
=
new
LogEntry2MetricsAdapter
(
logEntry
,
null
,
null
).
adaptCommonPart
();
return
LogData
return
LogData
.
newBuilder
()
.
newBuilder
()
.
setService
(
service
.
getServiceName
())
.
setService
(
service
.
getServiceName
())
.
setServiceInstance
(
service
.
getServiceInstanceName
())
.
setServiceInstance
(
service
.
getServiceInstanceName
())
.
setTimestamp
(
metrics
.
getEndTime
())
.
setTimestamp
(
metrics
.
getEndTime
())
.
setBody
(
LogDataBody
.
newBuilder
()
.
setJson
(
JSONLog
.
newBuilder
()
.
setJson
(
toJSON
(
logEntry
))
)
)
.
build
();
.
build
();
}
}
}
}
oap-server/server-receiver-plugin/envoy-metrics-receiver-plugin/src/main/java/org/apache/skywalking/oap/server/receiver/envoy/persistence/TCPLogsPersistence.java
浏览文件 @
982001c6
...
@@ -21,9 +21,7 @@ package org.apache.skywalking.oap.server.receiver.envoy.persistence;
...
@@ -21,9 +21,7 @@ package org.apache.skywalking.oap.server.receiver.envoy.persistence;
import
io.envoyproxy.envoy.data.accesslog.v3.TCPAccessLogEntry
;
import
io.envoyproxy.envoy.data.accesslog.v3.TCPAccessLogEntry
;
import
io.envoyproxy.envoy.service.accesslog.v3.StreamAccessLogsMessage
;
import
io.envoyproxy.envoy.service.accesslog.v3.StreamAccessLogsMessage
;
import
lombok.extern.slf4j.Slf4j
;
import
lombok.extern.slf4j.Slf4j
;
import
org.apache.skywalking.apm.network.logging.v3.JSONLog
;
import
org.apache.skywalking.apm.network.logging.v3.LogData
;
import
org.apache.skywalking.apm.network.logging.v3.LogData
;
import
org.apache.skywalking.apm.network.logging.v3.LogDataBody
;
import
org.apache.skywalking.apm.network.servicemesh.v3.ServiceMeshMetric
;
import
org.apache.skywalking.apm.network.servicemesh.v3.ServiceMeshMetric
;
import
org.apache.skywalking.oap.log.analyzer.module.LogAnalyzerModule
;
import
org.apache.skywalking.oap.log.analyzer.module.LogAnalyzerModule
;
import
org.apache.skywalking.oap.log.analyzer.provider.log.ILogAnalyzerService
;
import
org.apache.skywalking.oap.log.analyzer.provider.log.ILogAnalyzerService
;
...
@@ -35,8 +33,6 @@ import org.apache.skywalking.oap.server.receiver.envoy.als.ServiceMetaInfo;
...
@@ -35,8 +33,6 @@ import org.apache.skywalking.oap.server.receiver.envoy.als.ServiceMetaInfo;
import
org.apache.skywalking.oap.server.receiver.envoy.als.tcp.TCPAccessLogAnalyzer
;
import
org.apache.skywalking.oap.server.receiver.envoy.als.tcp.TCPAccessLogAnalyzer
;
import
org.apache.skywalking.oap.server.receiver.envoy.als.tcp.TCPLogEntry2MetricsAdapter
;
import
org.apache.skywalking.oap.server.receiver.envoy.als.tcp.TCPLogEntry2MetricsAdapter
;
import
static
org
.
apache
.
skywalking
.
oap
.
server
.
library
.
util
.
ProtoBufJsonUtils
.
toJSON
;
/**
/**
* {@code LogsPersistence} analyzes the error logs and persists them to the log system.
* {@code LogsPersistence} analyzes the error logs and persists them to the log system.
*/
*/
...
@@ -69,7 +65,7 @@ public class TCPLogsPersistence implements TCPAccessLogAnalyzer {
...
@@ -69,7 +65,7 @@ public class TCPLogsPersistence implements TCPAccessLogAnalyzer {
}
}
final
LogData
logData
=
convertToLogData
(
entry
,
result
);
final
LogData
logData
=
convertToLogData
(
entry
,
result
);
logAnalyzerService
.
doAnalysis
(
logData
);
logAnalyzerService
.
doAnalysis
(
logData
,
entry
);
}
catch
(
final
Exception
e
)
{
}
catch
(
final
Exception
e
)
{
log
.
error
(
"Failed to persist Envoy access log"
,
e
);
log
.
error
(
"Failed to persist Envoy access log"
,
e
);
}
}
...
@@ -81,27 +77,17 @@ public class TCPLogsPersistence implements TCPAccessLogAnalyzer {
...
@@ -81,27 +77,17 @@ public class TCPLogsPersistence implements TCPAccessLogAnalyzer {
return
prev
;
return
prev
;
}
}
public
LogData
convertToLogData
(
final
TCPAccessLogEntry
logEntry
,
public
LogData
convertToLogData
(
final
TCPAccessLogEntry
logEntry
,
final
Result
result
)
{
final
Result
result
)
throws
Exception
{
final
ServiceMetaInfo
service
=
result
.
getService
();
final
ServiceMetaInfo
service
=
result
.
getService
();
final
ServiceMeshMetric
.
Builder
metrics
=
new
TCPLogEntry2MetricsAdapter
(
logEntry
,
null
,
null
).
adaptCommonPart
();
final
ServiceMeshMetric
.
Builder
metrics
=
new
TCPLogEntry2MetricsAdapter
(
logEntry
,
null
,
null
).
adaptCommonPart
();
return
LogData
return
LogData
.
newBuilder
()
.
newBuilder
()
.
setService
(
service
.
getServiceName
())
.
setService
(
service
.
getServiceName
())
.
setServiceInstance
(
service
.
getServiceInstanceName
())
.
setServiceInstance
(
service
.
getServiceInstanceName
())
.
setTimestamp
(
metrics
.
getEndTime
())
.
setTimestamp
(
metrics
.
getEndTime
())
.
setBody
(
LogDataBody
.
newBuilder
()
.
setJson
(
JSONLog
.
newBuilder
()
.
setJson
(
toJSON
(
logEntry
))
)
)
.
build
();
.
build
();
}
}
}
}
oap-server/server-receiver-plugin/skywalking-log-recevier-plugin/src/main/java/org/apache/skywalking/oap/server/recevier/log/provider/handler/grpc/LogReportServiceHandler.java
浏览文件 @
982001c6
...
@@ -89,7 +89,7 @@ public class LogReportServiceHandler extends LogReportServiceGrpc.LogReportServi
...
@@ -89,7 +89,7 @@ public class LogReportServiceHandler extends LogReportServiceGrpc.LogReportServi
try
{
try
{
LogData
.
Builder
builder
=
logData
.
toBuilder
();
LogData
.
Builder
builder
=
logData
.
toBuilder
();
setServiceName
(
builder
);
setServiceName
(
builder
);
logAnalyzerService
.
doAnalysis
(
builder
);
logAnalyzerService
.
doAnalysis
(
builder
,
null
);
}
catch
(
Exception
e
)
{
}
catch
(
Exception
e
)
{
errorCounter
.
inc
();
errorCounter
.
inc
();
log
.
error
(
e
.
getMessage
(),
e
);
log
.
error
(
e
.
getMessage
(),
e
);
...
...
oap-server/server-receiver-plugin/skywalking-log-recevier-plugin/src/main/java/org/apache/skywalking/oap/server/recevier/log/provider/handler/rest/LogReportServiceRestHandler.java
浏览文件 @
982001c6
...
@@ -81,7 +81,7 @@ public class LogReportServiceRestHandler extends JettyHandler {
...
@@ -81,7 +81,7 @@ public class LogReportServiceRestHandler extends JettyHandler {
ProtoBufJsonUtils
.
fromJSON
(
it
.
toString
(),
builder
);
ProtoBufJsonUtils
.
fromJSON
(
it
.
toString
(),
builder
);
logs
.
add
(
builder
);
logs
.
add
(
builder
);
}
}
logs
.
forEach
(
logAnalyzerService:
:
doAnalysis
);
logs
.
forEach
(
it
->
logAnalyzerService
.
doAnalysis
(
it
,
null
)
);
}
catch
(
final
Exception
e
)
{
}
catch
(
final
Exception
e
)
{
log
.
error
(
e
.
getMessage
(),
e
);
log
.
error
(
e
.
getMessage
(),
e
);
errorCounter
.
inc
();
errorCounter
.
inc
();
...
...
编辑
预览
Markdown
is supported
0%
请重试
或
添加新附件
.
添加附件
取消
You are about to add
0
people
to the discussion. Proceed with caution.
先完成此消息的编辑!
取消
想要评论请
注册
或
登录