Trace-Data-Protocol.md

# Trace Data Protocol 
Trace Data Protocol describes the data format between SkyWalking agent/sniffer and backend. 

## Abstract
This protocol includes the downstream and upstream data format. Other languages agents/SDKs can use this protocol to 
uplink data to the SkyWalking backend.

- Discovery service provided by HTTP only
- Other services, includes Register, Trace, etc., provided by HTTP/JSON and gRPC both. 

### Version
v1.1

#### gRPC proto files
[gRPC proto files](https://github.com/apache/incubator-skywalking-data-collect-protocol/tree/v1.1.1)

## Collector discovery service
### Abstract
**Collector discovery service should be the first service after agent started** 
Through this service, get the gRPC service list. The agent can choose any one of them for uplink data. Recommend to 
acquire the list periodically.

### HTTP GET
- Input
GET：http://collectorIp:port/agent/gRPC 

- Output
JSON Array. Each element in the array ia a valid gRPC service address.
```json
["ip address1:port1","ip address2:port2","ip address3:port3"]
```

## Application Register Service
### Abstract
Register Application Code to the backend, and receive an integer represents the application.

[gRPC service define](https://github.com/apache/incubator-skywalking-data-collect-protocol/blob/v1.1.1/ApplicationRegisterService.proto)

- applicationCode is the config in your `agent.config`.
- The return id is **ApplicationId** as the value in `KeyWithIntegerValue`, which will be used in further data uplink.

## Discovery Service

### Register Instance Service
[gRPC service define](https://github.com/apache/incubator-skywalking-data-collect-protocol/blob/v1.1.1/DiscoveryService.proto#L29)

- agentUUID generated by agent, should be unique. Stay same before reboot, at least.
- **ApplicationInstanceId** will be used in further data uplink.

### Heart beat service
[gRPC service define](https://github.com/apache/incubator-skywalking-data-collect-protocol/blob/v1.1.1/DiscoveryService.proto#L32)

- Recommend to report heart beat every 20-60 seconds.
- Java agent don't use this, because JVM metrics upstream replace the capabilities of this. 

## Service Name Discovery Service
### Abstract
Replace the literal String service(operation) name by an id(integer)

[gRPC service define](.https://github.com/apache/incubator-skywalking-data-collect-protocol/blob/v1.1.1/DiscoveryService.proto#L70)

- Optional service, reduce the network cost but use more memory as a buffer mapping.

## Network Address Register Service
### Abstract
Network Address includes all remove service address, includes ip, port, hostname, etc., which used in RPC framework, MQ, DB, etc.

[gRPC service define](https://github.com/apache/incubator-skywalking-data-collect-protocol/blob/v1.1.1/NetworkAddressRegisterService.proto)

- Optional service, reduce the network cost but use more memory as a buffer mapping.

## JVM Metrics Service
### Abstract
Uplink the JVM metrics, including PermSize, HeapSize, CPU, Memory, etc., every second.

[gRPC service define](https://github.com/apache/incubator-skywalking-data-collect-protocol/blob/v1.1.1/JVMMetricsService.proto)

## Trace Segment Service
[gRPC service define](https://github.com/apache/incubator-skywalking-data-collect-protocol/blob/v1.1.1/TraceSegmentService.proto)

- UniqueId represents segmentId and globalTraceId. It have 3 parts(Longs), 1) applicationInstanceId, 2) ThreadId, 3) Timestamp + 10000 + seq(seq is in [0, 100000) )
- Span data please refs to [Plugin Development Guide](../setup/service-agent/java-agent/Plugin-Development-Guide.md)
- Id and name both exist, please use id if possible.
  - operationNameId/operationName 
  - networkAddress/networkAddressId
  - entryServiceName/entryServiceId
  - parentServiceName/parentServiceId
  - peerId/peer
- componentIds are defined in backend, [here](../../../apm-protocol/apm-network/src/main/java/org/apache/skywalking/apm/network/trace/component/ComponentsDefine.java)

# HTTP JSON Services
All HTTP Services match the gRPC services, just adjust use short keys.

## Instance Register Service
- http://ip:port/instance/register(default: localhost:12800) 

Input:
```
{
    ai: x, #applicationId
    au: "", #agentUUID
    rt: x, #registerTime
    oi: "", #osinfo
}
```

Output:
```
{
    ai: x, #applicationId
    ii: x, #applicationInstanceId
}
```

## Heart beat service
- http://ip:port/instance/heartbeat(default: localhost:12800) 

Input:
```
{
    "ii": x, #applicationInstanceId
    "ht": x #heartbeatTime, java timestamp format
}
```

Output: empty

## Service Name Discovery 
- http://ip:port/servicename/discovery(default: localhost:12800) 

Input：
```
{
    ai: x, #applicationId
    sn: "", #serviceName
    st: x, #srcSpanType
}
```

Output：
```
{
    si: x, #osinfo
    el: { #element
        ai: x, #applicationId
        sn: "", #serviceName
        st: x, #srcSpanType
    }
}
```

## Trace Segment Service
Input：
```
[
  {
    "gt": [[230150, 185809, 24040000]], 
    "sg": { //TraceSegmentObject 
      "ts": [137150, 185809, 48780000], 
      "ai": 2, //applicationId
      "ii": 3, //applicationInstanceId
      "ss": [ //SpanObject
        {
          "si": 0, //spanId
          "tv": 0, //SpanType
          "lv": 2, //SpanLayer
          "ps": -1, //parentSpanId
          "st": 1501858094726, //startTime
          "et": 1501858096804, //endTime
          "ci": 3, //componentId
          "cn": "", //component
          "oi": 0, //operationNameId
          "on": "org.skywaking.apm.testcase.dubbo.services.GreetService.doBusiness()", //operationName
          "pi": 0, //peerId
          "pn": "", //peer
          "ie": false, //isError
          "rs": [ //TraceSegmentReference
            {
              "pts": [230150, 185809, 24040000], //parentTraceSegmentId
              "pii": 2, //parentApplicationInstanceId
              "psp": 1, //parentSpanId
              "psi": 0, //parentServiceId
              "psn": "/dubbox-case/case/dubbox-rest", //parentServiceName
              "ni": 0,  //networkAddressId
              "nn": "172.25.0.4:20880", //networkAddress
              "eii": 2, //entryApplicationInstanceId
              "esi": 0, //entryServiceId
              "esn": "/dubbox-case/case/dubbox-rest", //entryServiceName
              "rv": 0 //RefTypeValue
            }
          ],
          "to": [ //KeyWithStringValue
            {
              "k": "url", //key
              "v": "rest://172.25.0.4:20880/org.skywaking.apm.testcase.dubbo.services.GreetService.doBusiness()" //value
            },
            {
              "k": "http.method",
              "v": "GET"
            }
          ],
          "lo": [{
                "ti": 1501858094726,
                "ld": [{ 
                        "k": "NullPointException",
                        "v": "Error Stack"
                    }]
           }]
        },
        {
          "si": 1,
          "tv": 1,
          "lv": 1,
          "ps": 0,
          "st": 1501858094726,
          "et": 1501858095804,
          "ci": 9,
          "cn": "",
          "oi": 0,
          "on": "mongodb://[username:password@]host1[:port1][,host2[:port2],...[,hostN[:portN]]][/[database][?options]]",
          "pi": 0,
          "pn": "localhost:27017",
          "ie": false,
          "to": [],
          "lo": []
        }
      ]
    }
  }
]
```