Merge pull request #713 from apache/more-en-documents

Provide plugin development guide EN version.

docs/README.md

@@ -19,6 +19,7 @@
     * [Java Agent Performance Test](https://skywalkingtest.github.io/Agent-Benchmarks/)
   * Development Guides
     * [How to build project](en/How-to-build.md)
+    * [Plugin development guide](en/Plugin-Development-Guide.md)
     * Protocol
       * [Cross Process Propagation Headers Protocol, v1.0](en/Skywalking-Cross-Process-Propagation-Headers-Protocol-v1.md)
   * FAQ

docs/README_ZH.md

@@ -23,7 +23,7 @@
     * [工程编译指南](cn/How-to-build-CN.md)
     * [插件开发指南](cn/Plugin-Development-Guide-CN.md)
     * 交互协议
-        * [Cross Process Propagation Headers Protocol, v1.0 | 跨进程追踪上下文传递协议](cn/Skywalking-Cross-Process-Propagation-Headers-Protocol-CN-v1.md)
+        * [Cross Process Propagation Headers Protocol, v1.0  跨进程追踪上下文传递协议](cn/Skywalking-Cross-Process-Propagation-Headers-Protocol-CN-v1.md)
         * [SkyWalking Trace Data Protocol 探针与Collector间网络协议](cn/Trace-Data-Protocol-CN.md)
   * FAQ
     * [Trace查询有数据，但是没有拓扑图和JVM数据?](cn/FAQ/Why-have-traces-no-others-CN.md)

docs/cn/How-to-build-CN.md

@@ -2,16 +2,17 @@
 本文档用于指导开发者，在本地开发环境中编译工程。
 
 ### 前言
-因为工程结构和代码依赖会随版本变化，如果读者熟悉travis-ci,则可直接参考[.travis.yml](https://github.com/wu-sheng/sky-walking/blob/master/.travis.yml)
+因为工程结构和代码依赖会随版本变化，如果读者熟悉travis-ci,则可直接参考[.travis.yml](../../.travis.yml)
 
 ### 编译步骤
 1. 准备环境，jdk8，Maven
 1. 执行`mvn clean package`
+1. 生成包在`/packages`目录下，包括一个`skywalking-agent`的探针目录，以及两个collector包（.tar.gz是linux环境，.zip是windows环境）
 
 ### 在IntelliJ IDEA中编译工程
 上述步骤在命令行中，能够很好的编译工程，但导入到编译器中的工程依然会有一些报错，我们需要进行几步简单的操作。
 1. 在IntelliJ Terminal中，执行`mvn compile -Dmaven.test.skip=true`进行编译
 1. 设置gRPC的自动生成代码目录，为源码目录
-  - **apm-network/target/generated--sources/protobuf**目录下的`grpc-java`和`java`目录
+  - **apm-protocol/apm-network/target/generated-sources/protobuf**目录下的`grpc-java`和`java`目录
   - **apm-collector/apm-collector-remote/apm-remote-grpc-provider/target/protobuf**目录下的`grpc-java`和`java`目录
 

docs/cn/Plugin-Development-Guide-CN.md

 ## 插件开发指南
-本文档描述 [v3.2+](https://github.com/OpenTracing/skywalking/releases) 插件开发方法、使用的API，以及注意事项。
+这边文档描述插件的开发和贡献方法
 
-### 核心概念
-#### 一. Span
+## 核心概念
+### 一. Span
 Span是追踪系统中的通用概念（有时候被翻译成埋点），关于Span的定义，请参考[OpenTracing 中文版](https://github.com/opentracing-contrib/opentracing-specification-zh/blob/master/specification.md#opentracing数据模型)。
 
 SkyWalking作为OpenTracing的支持者，在核心实现中，与标准有较高的相似度。当然，作为实际产品的需要，我们一会扩展相关概念。
@@ -18,14 +18,17 @@ LocalSpan代表一个普通的Span,代表任意一个本地逻辑块（或方法
 1.3 ExitSpan
 ExitSpan也可以称为LeafSpan(SkyWalking的早期版本中的称呼)，代表了一个远程服务的客户端调用。如：一次JDBC调用。
 
-#### 二. ContextCarrier
+### 二. ContextCarrier
 分布式追踪要解决的一个重要问题，就是跨进程调用链连接的问题，ContextCarrier的概念就是为了解决这种场景。
 
 当发生一次**A->B**的网络调用时：
-1. 需要在客户端生成(inject操作)ContextCarrier，并序列化成String
-1. 将这个String加入RPC调用的正文（或HEAD）中，传递到服务端
+1. 创建一个空的ContextCarrier
+1. 通过`ContextManager#createExitSpan`方法创建一个ExitSpan，或者使用`ContextManager#inject`，在过程中传入并初始化`ContextCarrier`
+1. 将`ContextCarrier`中所有元素放入请求头（如：HTTP头）或消息正文（如 Kafka）
+1. `ContextCarrier`随请求传输到服务端
 1. 服务端收到后，转换为新的ContextCarrier
-1. 通过提取操作（extract操作）建立关联
+1. 通过`ContestManager#createEntrySpan`方法创建EntrySpan，或者使用`ContextManager#extract`，建立分布式调用关联
+
 
 以HTTPComponent调用Tomcat为例：
 1. 客户端（HTTPComponent端）
@@ -34,8 +37,7 @@ ExitSpan也可以称为LeafSpan(SkyWalking的早期版本中的称呼)，代表
             CarrierItem next = contextCarrier.items();
             while (next.hasNext()) {
                 next = next.next();
-                //向HTTP或者其他RPC HEAD中设置上下文
-                heads.put(next.getHeadKey(), next.getHeadValue());
+                httpRequest.setHeader(next.getHeadKey(), next.getHeadValue());
             }
 ```
 
@@ -45,14 +47,13 @@ ExitSpan也可以称为LeafSpan(SkyWalking的早期版本中的称呼)，代表
             CarrierItem next = contextCarrier.items();
             while (next.hasNext()) {
                 next = next.next();
-                //从HTTP或者其他RPC HEAD中，根据指定的KEY，提取上下文
-                next.setHeadValue(heads.get(next.getHeadKey()));
+                next.setHeadValue(request.getHeader(next.getHeadKey()));
             }
 
             span = ContextManager.createEntrySpan(“/span/operation/name”, contextCarrier);
 ```
 
-#### 三. ContextSnapshot
+### 三. ContextSnapshot
 除了跨进程的RPC调用，另外一种追踪的常见场景是跨线程保持链路连接。跨线程和跨进程有很高的相似度，都是需要完成上下文的传递工作。
 所以ContextSnapshot具有和ContextCarrier十分类似的API风格。
 
@@ -61,8 +62,8 @@ ExitSpan也可以称为LeafSpan(SkyWalking的早期版本中的称呼)，代表
 1. 将这个ContextSnapshot对象传递到B线程中
 1. B线程通过ContextManager#continued操作完成上下文传递
 
-### 核心API
-#### 一. ContextManager
+## 核心API
+### 一. ContextManager
 ContextManager提供了追踪相关操作的主入口
 
 1. 创建EntrySpan
@@ -83,7 +84,7 @@ public static AbstractSpan createExitSpan(String operationName, ContextCarrier c
 ```
 根据服务名，跨进程传递的ContextCarrier（空容器）和远端服务地址（IP、主机名、域名 + 端口），创建ExitSpan
 
-#### 二. AbstractSpan
+### 二. AbstractSpan
 AbstractSpan提供了Span内部，进行操作的各项API
 
 ```java
@@ -141,17 +142,21 @@ AbstractSpan提供了Span内部，进行操作的各项API
 ```
 Span的操作语义和OpenTracing类似。
 
-SpanLayer为我们的特有概念，如果是远程调用类的服务，请设置此属性，包括4个属性值
+SpanLayer为我们的特有概念，如果是远程调用类的服务，请设置此属性，包括5个属性值
+1. UNKNOWN, 默认
 1. DB
 1. RPC_FRAMEWORK，非HTTP类型的RPC框架，如：原生的DUBBO，MOTAN
 1. HTTP
 1. MQ
 
-### 开发插件
-#### 一. 简介
+Component ID被SkyWalking项目组定义和保护。0到10000为保留值，如果你希望贡献新插件，可以在插件pull request通过，并提交的自动化
+测试用户被接收后，申请自己的组件ID。私有插件，请使用10000以上的ID，避免重复。
+
+## 开发插件
+### 一. 简介
 因为所有的程序调用都是基于方法的，所以插件实际上就是基于方法的拦截，类似面向切面编程的AOP技术。SkyWalking底层已经完成相关的技术封装，所以插件开发者只需要定位需要拦截的类、方法，然后结合上文中的追踪API，即可完成插件的开发。
 
-#### 二. 拦截类型
+### 二. 拦截类型
 根据Java方法，共有三种拦截类型
 1. 拦截构造函数
 1. 拦截实例方法
@@ -163,7 +168,7 @@ SpanLayer为我们的特有概念，如果是远程调用类的服务，请设
 
 当然，也可以同时支持实例和静态方法，直接继承ClassEnhancePluginDefine。但是，这种情况很少。
 
-#### 三. 实现自己的插件定义
+### 三. 实现自己的插件定义
 我们以继承ClassInstanceMethodsEnhancePluginDefine为例（ClassStaticMethodsEnhancePluginDefine十分类似，不再重复描述），描述定义插件的全过程
 
 1. 定义目标类名称
@@ -171,7 +176,7 @@ SpanLayer为我们的特有概念，如果是远程调用类的服务，请设
 protected abstract ClassMatch enhanceClass();
 ```
 
-ClassMatch反应类的匹配方式，目前提供三种：
+ClassMatch反应类的匹配方式，目前提供四种：
 
 * byName, 通过类名完整匹配
 * byClassAnnotationMatch, 通过类标注进行匹配
@@ -192,7 +197,7 @@ protected ClassMatch enhanceClassName() {
 
 ```
 
-2. 定义构造函数拦截点
+2. 定义方法拦截点
 ```java
 protected InstanceMethodsInterceptPoint[] getInstanceMethodsInterceptPoints();
 
@@ -223,7 +228,7 @@ tomcat-7.x/8.x=TomcatInstrumentation
 * 插件名称，要求全局唯一，命名规范：目标组件+版本号
 * 插件定义类全名
 
-#### 四. 实现拦截器逻辑
+### 四. 实现拦截器逻辑
 我们继续以实现实例方法拦截为例，拦截器需要实现org.apache.skywalking.apm.agent.core.plugin.interceptor.enhance.InstanceMethodsAroundInterceptor。
 ```java
 /**
@@ -264,7 +269,7 @@ public interface InstanceMethodsAroundInterceptor {
 
 可以在方法执行前、执行后、执行异常三个点，进行拦截，设置修改方法参数（执行前），并调用核心API，设置追踪逻辑。
 
-### 贡献插件到主仓库
+## 贡献插件到主仓库
 我们鼓励大家共同贡献支持各个类库的插件。
 
 大家需支持以下步骤执行：

docs/en/How-to-build.md

@@ -10,5 +10,5 @@ This document helps people to compile and build the project in your maven and ID
 1. Import the project as a maven project
 1. Run `mvn compile -Dmaven.test.skip=true` to compile project and generate source codes. Because we use gRPC and protobuf.
 1. Set **Generated Source Codes** folders.
-    * `grpc-java` and `java` folders in **apm-network/target/generated--sources/protobuf**
+    * `grpc-java` and `java` folders in **apm-protocol/apm-network/target/generated-sources/protobuf**
     * `grpc-java` and `java` folders in **apm-collector/apm-collector-remote/apm-remote-grpc-provider/target/protobuf**
\ No newline at end of file

docs/en/Plugin-Development-Guide.md

+# Plugin Development Guide
+This document describe how to understand, develop and contribute plugin.
+
+## Concepts
+### Span
+Span is an important and common concept in distributed tracing system. Learn **Span** from 
+[Google Dapper Paper](https://research.google.com/pubs/pub36356.html)  and
+[OpenTracing](http://opentracing.io)
+
+SkyWalking supports OpenTracing and OpenTracing-Java API from 2017. Our Span concepts are similar with the paper and OpenTracing.
+Also we extend the Span.
+
+There are three types of Span
+
+1.1 EntrySpan
+EntrySpan represents a service provider, also the endpoint of server side. As an APM system, we are targeting the 
+application servers. So almost all the services and MQ-comsumer are EntrySpan(s).
+
+1.2 LocalSpan
+LocalSpan represents a normal Java method, which don't relate with remote service, neither a MQ producer/comsumer
+nor a service(e.g. HTTP service) provider/consumer.
+
+1.3 ExitSpan
+ExitSpan represents a client of service or MQ-producer, as named as `LeafSpan` at early age of SkyWalking.
+e.g. accessing DB by JDBC, reading Redis/Memcached are cataloged an ExitSpan. 
+
+### ContextCarrier
+In order to implement distributed tracing, the trace across process need to be bind, and the context should propagate 
+across the process. That is ContextCarrier's duty.
+
+Here are the steps about how to use **ContextCarrier** in a `A->B` distributed call.
+1. Create a new and empty `ContextCarrier` at client side.
+1. Create an ExitSpan by `ContextManager#createExitSpan` or use `ContextManager#inject` to init the `ContextCarrier`.
+1. Put all items of `ContextCarrier` into heads(e.g. HTTP HEAD), attachments(e.g. Dubbo RPC framework) or messages(e.g. Kafka)
+1. The `ContextCarrier` propagates to server side by the service call.
+1. At server side, get all items from heads, attachments or messages.
+1. Create an EntrySpan by `ContestManager#createEntrySpan` or use `ContextManager#extract` to bind the client and server.
+
+
+Let's demonstrate the steps by Apache HTTPComponent client plugin and Tomcat 7 server plugin
+1. Client side steps by Apache HTTPComponent client plugin
+```java
+            span = ContextManager.createExitSpan("/span/operation/name", contextCarrier, "ip:port");
+            CarrierItem next = contextCarrier.items();
+            while (next.hasNext()) {
+                next = next.next();
+                httpRequest.setHeader(next.getHeadKey(), next.getHeadValue());
+            }
+```
+
+2. Server side steps by Tomcat 7 server plugin
+```java
+            ContextCarrier contextCarrier = new ContextCarrier();
+            CarrierItem next = contextCarrier.items();
+            while (next.hasNext()) {
+                next = next.next();
+                next.setHeadValue(request.getHeader(next.getHeadKey()));
+            }
+
+            span = ContextManager.createEntrySpan(“/span/operation/name”, contextCarrier);
+```
+
+### ContextSnapshot
+Besides across process, across thread but in a process need to be supported, because async process(In-memory MQ) 
+and batch process are common in Java. Across process and across thread are similar, because they are both about propagating
+context. The only difference is that, don't need to serialize for across thread.
+
+Here are the three steps about across thread propagation:
+1. Use `ContextManager#capture` to get the ContextSnapshot object.
+1. Let the sub-thread access the ContextSnapshot by any way, through method arguments or carried by an existed arguments
+1. Use `ContextManager#continued` in sub-thread.
+
+## Core APIs
+### ContextManager
+ContextManager provides all major and primary APIs.
+
+1. Create EntrySpan
+```java
+public static AbstractSpan createEntrySpan(String operationName, ContextCarrier carrier)
+```
+Create EntrySpan by operation name(e.g. service name, uri) and **ContextCarrier**.
+
+2. Create LocalSpan
+```java
+public static AbstractSpan createLocalSpan(String operationName)
+```
+Create LocalSpan by operation name(e.g. full method signature)
+
+3. Create ExitSpan
+```java
+public static AbstractSpan createExitSpan(String operationName, ContextCarrier carrier, String remotePeer)
+```
+Create ExitSpan by operation name(e.g. service name, uri) and new **ContextCarrier** and peer address
+(e.g. ip+port, hostname+port)
+
+### AbstractSpan
+```java
+    /**
+     * Set the component id, which defines in {@link ComponentsDefine}
+     *
+     * @param component
+     * @return the span for chaining.
+     */
+    AbstractSpan setComponent(Component component);
+
+    /**
+     * Only use this method in explicit instrumentation, like opentracing-skywalking-bridge.
+     * It it higher recommend don't use this for performance consideration.
+     *
+     * @param componentName
+     * @return the span for chaining.
+     */
+    AbstractSpan setComponent(String componentName);
+
+    AbstractSpan setLayer(SpanLayer layer);
+
+    /**
+     * Set a key:value tag on the Span.
+     *
+     * @return this Span instance, for chaining
+     */
+    AbstractSpan tag(String key, String value);
+
+    /**
+     * Record an exception event of the current walltime timestamp.
+     *
+     * @param t any subclass of {@link Throwable}, which occurs in this span.
+     * @return the Span, for chaining
+     */
+    AbstractSpan log(Throwable t);
+
+    AbstractSpan errorOccurred();
+
+    /**
+     * Record an event at a specific timestamp.
+     *
+     * @param timestamp The explicit timestamp for the log record.
+     * @param event the events
+     * @return the Span, for chaining
+     */
+    AbstractSpan log(long timestamp, Map<String, ?> event);
+
+    /**
+     * Sets the string name for the logical operation this span represents.
+     *
+     * @return this Span instance, for chaining
+     */
+    AbstractSpan setOperationName(String operationName);
+```
+Besides set operation name, tags and logs, two attributes shoule be set, which are component and layer, 
+especially for EntrySpan and ExitSpan
+
+SpanLayer is the catalog of span. Here are 5 values:
+1. UNKNOWN (default)
+1. DB
+1. RPC_FRAMEWORK, for a RPC framework, not an ordinary HTTP
+1. HTTP
+1. MQ
+
+Component IDs are defined and protected by SkyWalking project, 0 -> 10000 IDs are reserved. If you want to contribute
+a new plugin, you can ask for an official ID, after your pull request approved and automatic test cases accepted by PMC.
+Please use > 10000 ID, if you are going to develop a private plugin or don't intend to contribute the plugin to community,
+to avoid the ID conflict.
+
+## Develop a plugin
+### Abstract
+The basic method to trace is intercepting a Java method, by using byte code manipulation tech and AOP concept.
+SkyWalking boxed the byte code manipulation tech and tracing context propagation,
+so you just need to define the intercept point(a.k.a. aspect pointcut in Spring)
+
+### Intercept
+SkyWalking provide two common defines to intercept Contructor, instance method and class method.
+* Extend `ClassInstanceMethodsEnhancePluginDefine` defines `Contructor` intercept points and `instance method` intercept points.
+* Extend `ClassStaticMethodsEnhancePluginDefine` definec `class method` intercept points.
+
+Of course, you can extend `ClassEnhancePluginDefine` to set all intercept points. But it is unusual. 
+
+### Implement plugin
+I will demonstrate about how to implement a plugin by extending `ClassInstanceMethodsEnhancePluginDefine`
+
+1. Define the target class name
+```java
+protected abstract ClassMatch enhanceClass();
+```
+
+ClassMatch represents how to match the target classes, there are 4 ways:
+* byName, through the full class name(package name + `.` + class name)
+* byClassAnnotationMatch, through the class existed certain annotations.
+* byMethodAnnotationMatch, through the class's method existed certain annotations.
+* byHierarchyMatch, throught the class's parent classes or interfaces
+
+**Attentions**:
+* Forbid to use `*.class.getName()` to get the class String name. Recommend you to use literal String. This is for 
+avoiding ClassLoader issues.
+* `by*AnnotationMatch` doesn't support the inherited annotations.
+* Don't recommend use `byHierarchyMatch`, unless it is really necessary. Because using it may trigger intercepting 
+many unexcepted methods, which causes performance issues and concerns.
+
+Example：
+```java
+@Override
+protected ClassMatch enhanceClassName() {
+    return byName("org.apache.catalina.core.StandardEngineValve");		
+}		      
+
+```
+
+2. Define an instance method intercept point
+```java
+protected InstanceMethodsInterceptPoint[] getInstanceMethodsInterceptPoints();
+
+public interface InstanceMethodsInterceptPoint {
+    /**
+     * class instance methods matcher.
+     *
+     * @return methods matcher
+     */
+    ElementMatcher<MethodDescription> getMethodsMatcher();
+
+    /**
+     * @return represents a class name, the class instance must instanceof InstanceMethodsAroundInterceptor.
+     */
+    String getMethodsInterceptor();
+
+    boolean isOverrideArgs();
+}
+```
+Also use `Matcher` to set the target methods. Return **true** in `isOverrideArgs`, if you want to change the argument
+ref in interceptor.
+
+The following sections will tell you how to implement the interceptor.
+
+3. Add plugin define into skywalking-plugin.def file
+```properties
+tomcat-7.x/8.x=TomcatInstrumentation
+```
+
+
+### Implement an interceptor
+As an interceptor for an instance method, the interceptor implements 
+`org.apache.skywalking.apm.agent.core.plugin.interceptor.enhance.InstanceMethodsAroundInterceptor`
+```java
+/**
+ * A interceptor, which intercept method's invocation. The target methods will be defined in {@link
+ * ClassEnhancePluginDefine}'s subclass, most likely in {@link ClassInstanceMethodsEnhancePluginDefine}
+ *
+ * @author wusheng
+ */
+public interface InstanceMethodsAroundInterceptor {
+    /**
+     * called before target method invocation.
+     *
+     * @param result change this result, if you want to truncate the method.
+     * @throws Throwable
+     */
+    void beforeMethod(EnhancedInstance objInst, Method method, Object[] allArguments, Class<?>[] argumentsTypes,
+        MethodInterceptResult result) throws Throwable;
+
+    /**
+     * called after target method invocation. Even method's invocation triggers an exception.
+     *
+     * @param ret the method's original return value.
+     * @return the method's actual return value.
+     * @throws Throwable
+     */
+    Object afterMethod(EnhancedInstance objInst, Method method, Object[] allArguments, Class<?>[] argumentsTypes,
+        Object ret) throws Throwable;
+
+    /**
+     * called when occur exception.
+     *
+     * @param t the exception occur.
+     */
+    void handleMethodException(EnhancedInstance objInst, Method method, Object[] allArguments, Class<?>[] argumentsTypes,
+        Throwable t);
+}
+```
+Use the core APIs in before, after and exception handle stages.
+
+### Contribute plugins into Apache SkyWalking repository
+We are welcome everyone to contribute plugins.
+
+Please follow there steps:
+1. Submit an issue about which plugins are you going to contribute, including supported version.
+1. Create sub modules under `apm-sniffer/apm-sdk-plugin`, and the name should include supported library name and versions
+1. Follow this guide to develop. Make sure comments and test cases are provided.
+1. Develop and test.
+1. Send the pull request and ask for review, and provide the automatic test cases by following PMC members guides.
+1. The plugin committers approves your plugins after automatic test cases provided and the tests passed in our CI.
+1. The plugin accepted by SkyWalking.