diff --git a/docs/README.md b/docs/README.md index f7c6bed79524c6daa3ab59c1500b51297230aaf7..3bdfa68267a551bd772b79a4c57a6748ba2c297b 100644 --- a/docs/README.md +++ b/docs/README.md @@ -21,6 +21,7 @@ * [Plugin development guide](en/Plugin-Development-Guide.md) * Protocol * [Cross Process Propagation Headers Protocol, v1.0](en/Skywalking-Cross-Process-Propagation-Headers-Protocol-v1.md) + * [SkyWalking Trace Data Protocol](en/Trace-Data-Protocol.md) * [Roadmap](ROADMAP.md) * Resources provided by community * [Public speakings](https://github.com/OpenSkywalking/Community#public-speakings) diff --git a/docs/cn/Trace-Data-Protocol-CN.md b/docs/cn/Trace-Data-Protocol-CN.md index 32b649afbc4a9170df7b2a47296529098aa0cff5..d9b8ab46d3e65e23c1dc5b75479e18872e628f5c 100644 --- a/docs/cn/Trace-Data-Protocol-CN.md +++ b/docs/cn/Trace-Data-Protocol-CN.md @@ -40,11 +40,8 @@ JSON数组，数组的每个元素，为一个有效的gRPC服务地址。 ### 协议内容 [gRPC service define](../..apm-protocol/apm-network/src/main/proto/ApplicationRegisterService.proto) -- 首次调用时，applicationCode为客户端设置的应用名（显示在拓扑图和应用列表上的名字）。之后随着追踪过程，会上报此应用相关的周边服务的`ip:port`地址列表 -- KeyWithIntegerValue 返回，key为上报的applicationCode或ip:port地址，value为对应的id。applicationCode对应的返回id,在后续协议中，被称为applicationId。 -- 此服务按需调用，本地无法找到ip:port对应的id时，可异步发起调用。 -- 获取applicationId的操作是必选。后续追踪数据依赖此id -- 获取ip:port对应的id是可选，使用id，会有效提高collector处理效率，降低网络消耗。 +- applicationCode为客户端设置的应用名. +- applicationCode对应的返回id,在后续协议中，被称为applicationId。 ## 应用实例发现服务 ### 简介 diff --git a/docs/en/Trace-Data-Protocol.md b/docs/en/Trace-Data-Protocol.md new file mode 100644 index 0000000000000000000000000000000000000000..ee796cb4803e0fab6ece03f1dbe2d172b64b34ea --- /dev/null +++ b/docs/en/Trace-Data-Protocol.md @@ -0,0 +1,231 @@ +# Trace Data Protocol +Trace Data Protocol describes the data format between SkyWalking agent/sniffer and backend. + +## Abstract +This protocol includes the downstream and upstream data format. Other languages agents/SDKs can use this protocol to +uplink data to the SkyWalking backend. + +- Discovery service provided by HTTP only +- Other services, includes Register, Trace, etc., provided by HTTP/JSON and gRPC both. + +### Version +v1.1 + +#### gRPC proto files +[gRPC proto files](../../apm-protocol/apm-network/src/main/proto) + +## Collector discovery service +### Abstract +**Collector discovery service should be the first service after agent started** +Through this service, get the gRPC service list. The agent can choose any one of them for uplink data. Recommend to +acquire the list periodically. + +### HTTP GET +- Input +GET：http://collectorIp:port/agent/grpc + +- Output +JSON Array. Each element in the array ia a valid gRPC service address. +```json +["ip address1:port1","ip address2:port2","ip address3:port3"] +``` + +## Application Register Service +### Abstract +Register Application Code to the backend, and receive an integer represents the application. + +[gRPC service define](../..apm-protocol/apm-network/src/main/proto/ApplicationRegisterService.proto) + +- applicationCode is the config in your `agent.config`. +- The return id is **ApplicationId** as the value in `KeyWithIntegerValue`, which will be used in further data uplink. + +## Discovery Service + +### Register Instance Service +[gRPC service define](../../apm-protocol/apm-network/src/main/proto/DiscoveryService.proto#L11-L12) + +- agentUUID generated by agent, should be unique. Stay same before reboot, at least. +- **ApplicationInstanceId** will be used in further data uplink. + +### Heart beat service +[gRPC service define](../../apm-protocol/apm-network/src/main/proto/DiscoveryService.proto#L14-L15) + +- Recommend to report heart beat every 20-60 seconds. +- Java agent don't use this, because JVM metrics upstream replace the capabilities of this. + +## Service Name Discovery Service +### Abstract +Replace the literal String service(operation) name by an id(integer) + +[gRPC service define](../../apm-protocol/apm-network/src/main/proto/DiscoveryService.proto#L53-L74) + +- Optional service, reduce the network cost but use more memory as a buffer mapping. + +## Network Address Register Service +### Abstract +Network Address includes all remove service address, includes ip, port, hostname, etc., which used in RPC framework, MQ, DB, etc. + +[gRPC service define](../../apm-protocol/apm-network/src/main/proto/NetworkAddressRegisterService.proto) + +- Optional service, reduce the network cost but use more memory as a buffer mapping. + +## JVM Metrics Service +### Abstract +Uplink the JVM metrics, including PermSize, HeapSize, CPU, Memory, etc., every second. + +[gRPC service define](../../apm-protocol/apm-network/src/main/proto/JVMMetricsService.proto) + +## Trace Segment Service +[gRPC service define](../../apm-protocol/apm-network/src/main/proto/TraceSegmentService.proto) + +- UniqueId represents segmentId and globalTraceId. It have 3 parts(Longs), 1) applicationInstanceId, 2) ThreadId, 3) Timestamp + 10000 + seq(seq is in [0, 100000) ) +- Span data please refs to [Plugin Development Guide](Plugin-Development-Guide.md) +- Id and name both exist, please use id if possible. + - operationNameId/operationName + - networkAddress/networkAddressId + - entryServiceName/entryServiceId + - parentServiceName/parentServiceId + - peerId/peer +- componentIds are defined in backend, [here](../../apm-protocol/apm-network/src/main/java/org/apache/skywalking/apm/network/trace/component/ComponentsDefine.java) + +# HTTP JSON Services +All HTTP Services match the gRPC services, just adjust use short keys. + +## Instance Register Service +- http://ip:port/instance/register(default: localhost:12800) + +Input: +``` +{ + ai: x, #applicationId + au: "", #agentUUID + rt: x, #registerTime + oi: "", #osinfo +} +``` + +Output: +``` +{ + ai: x, #applicationId + ii: x, #applicationInstanceId +} +``` + +## Heart beat service +- http://ip:port/instance/heartbeat(default: localhost:12800) + +Input: +``` +{ + "ii": x, #applicationInstanceId + "ht": x #heartbeatTime, java timestamp format +} +``` + +Output: empty + +## Service Name Discovery +- http://ip:port/servicename/discovery(default: localhost:12800) + +Input： +``` +{ + ai: x, #applicationId + sn: "", #serviceName + st: x, #srcSpanType +} +``` + +Output： +``` +{ + si: x, #osinfo + el: { #element + ai: x, #applicationId + sn: "", #serviceName + st: x, #srcSpanType + } +} +``` + +## Trace Segment Service +Input： +``` +[ + { + "gt": [[230150, 185809, 24040000]], + "sg": { //TraceSegmentObject + "ts": [137150, 185809, 48780000], + "ai": 2, //applicationId + "ii": 3, //applicationInstanceId + "ss": [ //SpanObject + { + "si": 0, //spanId + "tv": 0, //SpanType + "lv": 2, //SpanLayer + "ps": -1, //parentSpanId + "st": 1501858094726, //startTime + "et": 1501858096804, //endTime + "ci": 3, //componentId + "cn": "", //component + "oi": 0, //operationNameId + "on": "org.skywaking.apm.testcase.dubbo.services.GreetService.doBusiness()", //operationName + "pi": 0, //peerId + "pn": "", //peer + "ie": false, //isError + "rs": [ //TraceSegmentReference + { + "pts": [230150, 185809, 24040000], //parentTraceSegmentId + "pii": 2, //parentApplicationInstanceId + "psp": 1, //parentSpanId + "psi": 0, //parentServiceId + "psn": "/dubbox-case/case/dubbox-rest", //parentServiceName + "ni": 0, //networkAddressId + "nn": "172.25.0.4:20880", //networkAddress + "eii": 2, //entryApplicationInstanceId + "esi": 0, //entryServiceId + "esn": "/dubbox-case/case/dubbox-rest", //entryServiceName + "rn": 0 //RefType + } + ], + "to": [ //KeyWithStringValue + { + "k": "url", //key + "v": "rest://172.25.0.4:20880/org.skywaking.apm.testcase.dubbo.services.GreetService.doBusiness()" //value + }, + { + "k": "http.method", + "v": "GET" + } + ], + "lo": { //LogMessage + "t": 1501858094726, + "d": [ + "k": "NullPointException", + "v": "Error Stack" + } + } + }, + { + "si": 1, + "tv": 1, + "lv": 1, + "ps": 0, + "st": 1501858094726, + "et": 1501858095804, + "ci": 9, + "cn": "", + "oi": 0, + "on": "mongodb://[username:password@]host1[:port1][,host2[:port2],...[,hostN[:portN]]][/[database][?options]]", + "pi": 0, + "pn": "localhost:27017", + "ie": false, + "to": [], + "lo": [] + } + ] + } + } +] +```