未验证 提交 1b8dbd3f 编写于 作者: wu-sheng's avatar wu-sheng 提交者: GitHub

Polish doc, restructure documents about receivers and fetchers (#8043)

1. Remove general receiver and fetcher docs
1. Add more specific menu with docs to help users to find documents easier.
上级 54b72c88
......@@ -62,6 +62,9 @@ Release Notes.
#### Documentation
* Enhance documents about the data report and query protocols.
* Restructure documents about receivers and fetchers.
1. Remove general receiver and fetcher docs
2. Add more specific menu with docs to help users to find documents easier.
All issues and pull requests are [here](https://github.com/apache/skywalking/milestone/101?closed=1)
......
......@@ -12,7 +12,7 @@ is only used as an ingress of the whole cluster, but with the Service Mesh and s
- **3rd-party instrument library**. SkyWalking accepts many widely used instrument libraries data formats. It analyzes the
data, transfers it to SkyWalking's formats of trace, metrics or both. This feature starts with accepting Zipkin span data. See
[Receiver for other tracers](../setup/backend/backend-receivers.md) for more information.
[Receiver for Zipkin traces](../setup/backend/zipkin-trace.md) for more information.
You don't need to use **Language based native agent** and **Service Mesh probe** at the same time, since they both serve to collect
metrics data. Otherwise, your system will suffer twice the payload, and the analytic numbers will be doubled.
......@@ -31,6 +31,6 @@ like logs. In other words, the backend saves them, and builds the links between
## What is next?
- Learn more about the probes supported by SkyWalking in [Service auto instrument agent](service-agent.md), [Manual instrument SDK](manual-sdk.md),
[Service Mesh probe](service-mesh-probe.md) and [Zipkin receiver](../setup/backend/backend-receivers.md#zipkin-receiver).
[Service Mesh probe](service-mesh-probe.md) and [Zipkin receiver](../setup/backend/zipkin-trace.md).
- After understanding how the probe works, see the [backend overview](backend-overview.md) for more on analysis and persistence.
......@@ -98,7 +98,7 @@ storage implementor in addition to the official one.
- Customize analysis using OAL scripts. OAL scripts are located in `config/oal/*.oal`. You could modify them and reboot the OAP server. Read
[Observability Analysis Language Introduction](../concepts-and-designs/oal.md) to learn more about OAL scripts.
- [Source and scope extension for new metrics](source-extension.md). For analysis of a new metric which SkyWalking
hasn't yet provided. Add a new receiver, rather than choosing an [existing receiver](../setup/backend/backend-receivers.md).
hasn't yet provided, add a new receiver.
You would most likely have to add a new source and scope. To learn how to do this, read the document.
### UI developer
......
......@@ -33,7 +33,7 @@ Typically, agent meter plugin(e.g. [Java Meter Plugin](https://skywalking.apache
Satellite [Prometheus fetcher](https://skywalking.apache.org/docs/skywalking-satellite/latest/en/setup/plugins/fetcher_prometheus-metrics-fetcher/)
would transfer metrics into native format and forward to SkyWalking OAP server.
About receiving 3rd party formats metrics, read [Meter receiver](../setup/backend/backend-meter.md) and [OpenTelemetry receiver](../setup/backend/backend-receivers.md#opentelemetry-receiver) docs for more details.
About receiving 3rd party formats metrics, read [Meter receiver](../setup/backend/backend-meter.md) and [OpenTelemetry receiver](../setup/backend/opentelemetry-receiver.md) docs for more details.
### Browser probe protocol
......
# Setup External Communication Channels
SkyWalking has default activated gRPC/HTTP servers in the core module, which serve for both internal communication
and external data report or query.
In some advanced scenarios, such as security requirements, specific gRPC/HTTP servers should be exposed for external
requests.
```yaml
receiver-sharing-server:
selector: ${SW_RECEIVER_SHARING_SERVER:default}
default:
# For Jetty server
restHost: ${SW_RECEIVER_SHARING_REST_HOST:0.0.0.0}
restPort: ${SW_RECEIVER_SHARING_REST_PORT:0}
restContextPath: ${SW_RECEIVER_SHARING_REST_CONTEXT_PATH:/}
restMinThreads: ${SW_RECEIVER_SHARING_JETTY_MIN_THREADS:1}
restMaxThreads: ${SW_RECEIVER_SHARING_JETTY_MAX_THREADS:200}
restIdleTimeOut: ${SW_RECEIVER_SHARING_JETTY_IDLE_TIMEOUT:30000}
restAcceptorPriorityDelta: ${SW_RECEIVER_SHARING_JETTY_DELTA:0}
restAcceptQueueSize: ${SW_RECEIVER_SHARING_JETTY_QUEUE_SIZE:0}
httpMaxRequestHeaderSize: ${SW_RECEIVER_SHARING_HTTP_MAX_REQUEST_HEADER_SIZE:8192}
# For gRPC server
gRPCHost: ${SW_RECEIVER_GRPC_HOST:0.0.0.0}
gRPCPort: ${SW_RECEIVER_GRPC_PORT:0}
maxConcurrentCallsPerConnection: ${SW_RECEIVER_GRPC_MAX_CONCURRENT_CALL:0}
maxMessageSize: ${SW_RECEIVER_GRPC_MAX_MESSAGE_SIZE:0}
gRPCThreadPoolQueueSize: ${SW_RECEIVER_GRPC_POOL_QUEUE_SIZE:0}
gRPCThreadPoolSize: ${SW_RECEIVER_GRPC_THREAD_POOL_SIZE:0}
gRPCSslEnabled: ${SW_RECEIVER_GRPC_SSL_ENABLED:false}
gRPCSslKeyPath: ${SW_RECEIVER_GRPC_SSL_KEY_PATH:""}
gRPCSslCertChainPath: ${SW_RECEIVER_GRPC_SSL_CERT_CHAIN_PATH:""}
gRPCSslTrustedCAsPath: ${SW_RECEIVER_GRPC_SSL_TRUSTED_CAS_PATH:""}
authentication: ${SW_AUTHENTICATION:""}
```
Set `restPort`(HTTP) and `gRPCPort`(gRPC) to a legal port(greater than 0), would initialize new gRPC/HTTP servers for
external requests with other relative settings. In this case, `core/gRPC` and `core/rest` could be served for cluster
internal communication only.
# K8s monitoring
SkyWalking leverages K8s kube-state-metrics and cAdvisor for collecting metrics data from K8s, and leverages OpenTelemetry Collector to transfer the metrics to
[OpenTelemetry receiver](backend-receivers.md#opentelemetry-receiver) and into the [Meter System](./../../concepts-and-designs/meter.md). This feature requires authorizing the OAP Server to access K8s's `API Server`.
[OpenTelemetry receiver](opentelemetry-receiver.md) and into the [Meter System](./../../concepts-and-designs/meter.md). This feature requires authorizing the OAP Server to access K8s's `API Server`.
We define the k8s-cluster as a `Service` in the OAP, and use `k8s-cluster::` as a prefix to identify it.
We define the k8s-node as an `Instance` in the OAP, and set its name as the K8s `node name`.
We define the k8s-service as an `Endpoint` in the OAP, and set its name as `$serviceName.$namespace`.
......@@ -14,7 +14,7 @@ We define the k8s-service as an `Endpoint` in the OAP, and set its name as `$ser
1. Setup [kube-state-metric](https://github.com/kubernetes/kube-state-metrics#kubernetes-deployment).
2. cAdvisor is integrated into `kubelet` by default.
3. Set up [OpenTelemetry Collector ](https://opentelemetry.io/docs/collector/getting-started/#kubernetes). For details on Prometheus Receiver in OpenTelemetry Collector for K8s, refer to [here](https://github.com/prometheus/prometheus/blob/main/documentation/examples/prometheus-kubernetes.yml). For a quick start, we have provided a full example for OpenTelemetry Collector configuration [otel-collector-config.yaml](otel-collector-config.yaml).
4. Config SkyWalking [OpenTelemetry receiver](backend-receivers.md#opentelemetry-receiver).
4. Config SkyWalking [OpenTelemetry receiver](opentelemetry-receiver.md).
## Supported Metrics
From the different points of view to monitor K8s, there are 3 kinds of metrics: [Cluster](#cluster) / [Node](#node) / [Service](#service)
......
......@@ -20,11 +20,15 @@ kafka-fetcher:
## Meter collection
Metrics named in OAL script could be used in MAL, as described in [Official OAL script](../../guides/backend-oal-scripts.md)
Custom metrics may be collected by Manual Meter API. You may find correct APIs diving into [Server Agents](../service-agent/server-agents.md).
Custom metrics may be collected by Manual Meter API.
Custom metrics collected cannot be used directly, they should be configured in `meter-analyzer-config` configuration files, which is described in next part.
The receiver adds labels with `key = service` and `key = instance` to the collected data samples,
and values from service and service instance name defined in SkyWalking Agent,
for identification of the metric data.
A typical manual meter API set is [Spring Sleuth APIs](spring-sleuth-setup.md)
## Configuration file
The meter receiver is configured via a configuration file. The configuration file defines everything related to receiving
from agents, as well as which rule files to load.
......
# Choosing a receiver
Receiver is a defined concept in SkyWalking's backend. All modules which are responsible for receiving telemetry
or tracing data from other systems being monitored are all called **receivers**. If you are looking for the pull mode,
take a look at the [fetcher document](backend-fetcher.md).
We have the following receivers, and `default` implementors are provided in our Apache distribution.
1. **receiver-trace**. gRPC and HTTPRestful services that accept SkyWalking format traces.
1. **receiver-register**. gRPC and HTTPRestful services that provide service, service instance and endpoint register.
1. **service-mesh**. gRPC services that accept data from inbound mesh probes.
1. **receiver-jvm**. gRPC services that accept JVM metrics data.
1. **envoy-metric**. Envoy `metrics_service` and `ALS(access log service)` are supported by this receiver. The OAL script supports all GAUGE type metrics.
1. **receiver-profile**. gRPC services that accept profile task status and snapshot reporter.
1. **receiver-otel**. See [details](#opentelemetry-receiver). A receiver for analyzing metrics data from OpenTelemetry.
1. **receiver-meter**. See [details](backend-meter.md). A receiver for analyzing metrics in SkyWalking native meter format.
1. **receiver-browser**. gRPC services that accept browser performance data and error log.
1. **receiver-log**. A receiver for native log format. See [Log Analyzer](log-analyzer.md) for advanced features.
1. **configuration-discovery**. gRPC services that handle configurationDiscovery.
1. **receiver-event**. gRPC services that handle events data.
1. **receiver-zabbix**. See [details](backend-zabbix.md).
1. Experimental receivers.
1. **receiver_zipkin**. See [details](#zipkin-receiver).
The sample settings of these receivers are by default included in `application.yml`, and also listed here:
```yaml
receiver-register:
selector: ${SW_RECEIVER_REGISTER:default}
default:
receiver-trace:
selector: ${SW_RECEIVER_TRACE:default}
default:
receiver-jvm:
selector: ${SW_RECEIVER_JVM:default}
default:
service-mesh:
selector: ${SW_SERVICE_MESH:default}
default:
envoy-metric:
selector: ${SW_ENVOY_METRIC:default}
default:
acceptMetricsService: ${SW_ENVOY_METRIC_SERVICE:true}
alsHTTPAnalysis: ${SW_ENVOY_METRIC_ALS_HTTP_ANALYSIS:""}
receiver_zipkin:
selector: ${SW_RECEIVER_ZIPKIN:-}
default:
host: ${SW_RECEIVER_ZIPKIN_HOST:0.0.0.0}
port: ${SW_RECEIVER_ZIPKIN_PORT:9411}
contextPath: ${SW_RECEIVER_ZIPKIN_CONTEXT_PATH:/}
jettyMinThreads: ${SW_RECEIVER_ZIPKIN_JETTY_MIN_THREADS:1}
jettyMaxThreads: ${SW_RECEIVER_ZIPKIN_JETTY_MAX_THREADS:200}
jettyIdleTimeOut: ${SW_RECEIVER_ZIPKIN_JETTY_IDLE_TIMEOUT:30000}
jettyAcceptorPriorityDelta: ${SW_RECEIVER_ZIPKIN_JETTY_DELTA:0}
jettyAcceptQueueSize: ${SW_RECEIVER_ZIPKIN_QUEUE_SIZE:0}
receiver-profile:
selector: ${SW_RECEIVER_PROFILE:default}
default:
receiver-browser:
selector: ${SW_RECEIVER_BROWSER:default}
default:
sampleRate: ${SW_RECEIVER_BROWSER_SAMPLE_RATE:10000}
log-analyzer:
selector: ${SW_LOG_ANALYZER:default}
default:
lalFiles: ${SW_LOG_LAL_FILES:default}
malFiles: ${SW_LOG_MAL_FILES:""}
configuration-discovery:
selector: ${SW_CONFIGURATION_DISCOVERY:default}
default:
receiver-event:
selector: ${SW_RECEIVER_EVENT:default}
default:
```
## gRPC/HTTP server for receiver
By default, all gRPC/HTTP services should be served at `core/gRPC` and `core/rest`.
But the `receiver-sharing-server` module allows all receivers to be served at
different ip:port, if you set them explicitly.
```yaml
receiver-sharing-server:
selector: ${SW_RECEIVER_SHARING_SERVER:default}
default:
host: ${SW_RECEIVER_JETTY_HOST:0.0.0.0}
contextPath: ${SW_RECEIVER_JETTY_CONTEXT_PATH:/}
authentication: ${SW_AUTHENTICATION:""}
jettyMinThreads: ${SW_RECEIVER_SHARING_JETTY_MIN_THREADS:1}
jettyMaxThreads: ${SW_RECEIVER_SHARING_JETTY_MAX_THREADS:200}
jettyIdleTimeOut: ${SW_RECEIVER_SHARING_JETTY_IDLE_TIMEOUT:30000}
jettyAcceptorPriorityDelta: ${SW_RECEIVER_SHARING_JETTY_DELTA:0}
jettyAcceptQueueSize: ${SW_RECEIVER_SHARING_JETTY_QUEUE_SIZE:0}
```
Note: If you add these settings, make sure that they are not the same as the core module. This is because gRPC/HTTP servers of the core are still used for UI and OAP internal communications.
## OpenTelemetry receiver
The OpenTelemetry receiver supports ingesting agent metrics by meter-system. The OAP can load the configuration at bootstrap.
If the new configuration is not well-formed, the OAP may fail to start up. The files are located at `$CLASSPATH/otel-<handler>-rules`.
E.g. The `oc` handler loads rules from `$CLASSPATH/otel-oc-rules`.
Supported handlers:
* `oc`: [OpenCensus](https://github.com/open-telemetry/opentelemetry-collector-contrib/blob/a08903f05d3a544f548535c222b1c205b9f5a154/exporter/opencensusexporter/README.md) gRPC service handler.
**Notice:** Set `SW_OTEL_RECEIVER=default` through system environment or change `receiver-otel/selector=${SW_OTEL_RECEIVER:default}` to activate the OpenTelemetry receiver.
The rule file should be in YAML format, defined by the scheme described in [prometheus-fetcher](./backend-fetcher.md).
Note: `receiver-otel` only supports the `group`, `defaultMetricLevel`, and `metricsRules` nodes of the scheme due to its push mode.
To activate the `oc` handler and relevant rules of `istio`:
```yaml
receiver-otel:
// Change selector value to default, for activating the otel receiver.
selector: ${SW_OTEL_RECEIVER:default}
default:
enabledHandlers: ${SW_OTEL_RECEIVER_ENABLED_HANDLERS:"oc"}
enabledOcRules: ${SW_OTEL_RECEIVER_ENABLED_OC_RULES:"istio-controlplane"}
```
The receiver adds labels with `key = node_identifier_host_name` and `key = node_identifier_pid` to the collected data samples,
and values from `Node.identifier.host_name` and `Node.identifier.pid` defined in OpenCensus Agent Proto,
for identification of the metric data.
| Rule Name | Description | Configuration File | Data Source |
|----|----|-----|----|
|istio-controlplane| Metrics of Istio control panel | otel-oc-rules/istio-controlplane.yaml | Istio Control Panel -> OpenTelemetry Collector --OC format--> SkyWalking OAP Server |
|oap| Metrics of SkyWalking OAP server itself | otel-oc-rules/oap.yaml | SkyWalking OAP Server(SelfObservability) -> OpenTelemetry Collector --OC format--> SkyWalking OAP Server |
|vm| Metrics of VMs | otel-oc-rules/vm.yaml | Prometheus node-exporter(VMs) -> OpenTelemetry Collector --OC format--> SkyWalking OAP Server |
|k8s-cluster| Metrics of K8s cluster | otel-oc-rules/k8s-cluster.yaml | K8s kube-state-metrics -> OpenTelemetry Collector --OC format--> SkyWalking OAP Server |
|k8s-node| Metrics of K8s cluster | otel-oc-rules/k8s-node.yaml | cAdvisor & K8s kube-state-metrics -> OpenTelemetry Collector --OC format--> SkyWalking OAP Server |
|k8s-service| Metrics of K8s cluster | otel-oc-rules/k8s-service.yaml | cAdvisor & K8s kube-state-metrics -> OpenTelemetry Collector --OC format--> SkyWalking OAP Server |
## Meter receiver
The meter receiver supports accepting the metrics into the meter-system. The OAP can load the configuration at bootstrap.
The file is written in YAML format, defined by the scheme described in [backend-meter](./backend-meter.md).
To activate the `default` implementation:
```yaml
receiver-meter:
selector: ${SW_RECEIVER_METER:default}
default:
```
To activate the meter rule files:
Put your customized meter file xxx.yaml ( [mal](../../concepts-and-designs/mal.md) format) in the `config/meter-analyzer-config` directory and configure meteranalyzer `activefiles=${SW_ METER_ ANALYZER_ ACTIVE_ FILES:xxx}`
```yaml
agent-analyzer:
selector: ${SW_AGENT_ANALYZER:default}
default:
meterAnalyzerActiveFiles: ${SW_METER_ANALYZER_ACTIVE_FILES:} # Which files could be meter analyzed, files split by ","
```
The receiver adds labels with `key = service` and `key = instance` to the collected data samples,
and values from service and service instance name defined in SkyWalking Agent,
for identification of the metric data.
## Zipkin receiver
The Zipkin receiver makes the OAP server work as an alternative Zipkin server implementation. It supports Zipkin v1/v2 formats through HTTP service.
Make sure you use this with `SW_STORAGE=zipkin-elasticsearch` option to activate Zipkin storage implementation.
Once this receiver and storage are activated, SkyWalking's native traces would be ignored, and SkyWalking wouldn't analyze topology, metrics, and endpoint
dependency from Zipkin's trace.
Use the following config to activate it.
```yaml
receiver_zipkin:
selector: ${SW_RECEIVER_ZIPKIN:-}
default:
host: ${SW_RECEIVER_ZIPKIN_HOST:0.0.0.0}
port: ${SW_RECEIVER_ZIPKIN_PORT:9411}
contextPath: ${SW_RECEIVER_ZIPKIN_CONTEXT_PATH:/}
jettyMinThreads: ${SW_RECEIVER_ZIPKIN_JETTY_MIN_THREADS:1}
jettyMaxThreads: ${SW_RECEIVER_ZIPKIN_JETTY_MAX_THREADS:200}
jettyIdleTimeOut: ${SW_RECEIVER_ZIPKIN_JETTY_IDLE_TIMEOUT:30000}
jettyAcceptorPriorityDelta: ${SW_RECEIVER_ZIPKIN_JETTY_DELTA:0}
jettyAcceptQueueSize: ${SW_RECEIVER_ZIPKIN_QUEUE_SIZE:0}
```
NOTE: Zipkin receiver requires `zipkin-elasticsearch` storage implementation to be activated.
Read [this](backend-storage.md#elasticsearch-7-with-zipkin-trace-extension) doc to learn about Zipkin as a storage option.
......@@ -95,8 +95,7 @@ The required modules are listed here:
capabilities. See [**Cluster Management**](backend-cluster.md) for more details.
1. **Storage**. Makes the analysis result persistent. See [**Choose storage**](backend-storage.md) for more details
1. **Query**. Provides query interfaces to UI.
1. **Receiver** and **Fetcher**. Expose the service to the agents and probes, or read telemetry data from a channel.
See [Receiver](backend-receivers.md) and [Fetcher](backend-fetcher.md) documents for more details.
1. **Receiver** and **Fetcher**. Expose the service to the agents and probes, or read telemetry data from a channel.
## FAQs
#### Why do we need to set the timezone? And when do we do it?
......
......@@ -91,7 +91,7 @@ staticConfig:
...
```
### Service discovery (k8s)
If you deploy an oap-server cluster on k8s, the oap-server instance (pod) would not have a static IP or hostname. We can leverage [OpenTelemetry Collector](https://opentelemetry.io/docs/collector/getting-started/#kubernetes) to discover the oap-server instance, and scrape & transfer the metrics to OAP [OpenTelemetry receiver](backend-receivers.md#opentelemetry-receiver).
If you deploy an oap-server cluster on k8s, the oap-server instance (pod) would not have a static IP or hostname. We can leverage [OpenTelemetry Collector](https://opentelemetry.io/docs/collector/getting-started/#kubernetes) to discover the oap-server instance, and scrape & transfer the metrics to OAP [OpenTelemetry receiver](opentelemetry-receiver.md).
On how to install SkyWalking on k8s, you can refer to [Apache SkyWalking Kubernetes](https://github.com/apache/skywalking-kubernetes).
......
# VMs monitoring
SkyWalking leverages Prometheus node-exporter to collect metrics data from the VMs, and leverages OpenTelemetry Collector to transfer the metrics to
[OpenTelemetry receiver](backend-receivers.md#opentelemetry-receiver) and into the [Meter System](./../../concepts-and-designs/meter.md).
[OpenTelemetry receiver](opentelemetry-receiver.md) and into the [Meter System](./../../concepts-and-designs/meter.md).
We define the VM entity as a `Service` in OAP, and use `vm::` as a prefix to identify it.
## Data flow
......@@ -13,7 +13,7 @@ We define the VM entity as a `Service` in OAP, and use `vm::` as a prefix to ide
1. Setup [Prometheus node-exporter](https://prometheus.io/docs/guides/node-exporter/).
2. Setup [OpenTelemetry Collector ](https://opentelemetry.io/docs/collector/). This is an example for OpenTelemetry Collector configuration [otel-collector-config.yaml](../../../../test/e2e-v2/cases/vm/prometheus-node-exporter/otel-collector-config.yaml).
3. Config SkyWalking [OpenTelemetry receiver](backend-receivers.md#opentelemetry-receiver).
3. Config SkyWalking [OpenTelemetry receiver](opentelemetry-receiver.md).
## Supported Metrics
......
# Zabbix Receiver
The Zabbix receiver acceps metrics of [Zabbix Agent Active Checks protocol](https://www.zabbix.com/documentation/current/manual/appendix/items/activepassive#active_checks) format into the [Meter System](./../../concepts-and-designs/meter.md).
The Zabbix receiver accepts metrics of [Zabbix Agent Active Checks protocol](https://www.zabbix.com/documentation/current/manual/appendix/items/activepassive#active_checks) format into the [Meter System](./../../concepts-and-designs/meter.md).
Zabbix Agent is based on GPL-2.0 License.
## Module definition
......
......@@ -170,39 +170,32 @@ core|default|role|Option values: `Mixed/Receiver/Aggregator`. **Receiver** mode
| - | - | lalFiles | The LAL configuration file names (without file extension) to be activated. Read [LAL](../../concepts-and-designs/lal.md) for more details. | SW_LOG_LAL_FILES | default |
| - | - | malFiles | The MAL configuration file names (without file extension) to be activated. Read [LAL](../../concepts-and-designs/lal.md) for more details. | SW_LOG_MAL_FILES | "" |
| event-analyzer | default | Event Analyzer. | SW_EVENT_ANALYZER | default |
| receiver-register|default| Read [receiver doc](backend-receivers.md) for more details. | - | - |
| receiver-trace|default| Read [receiver doc](backend-receivers.md) for more details. | - | - |
| receiver-jvm| default| Read [receiver doc](backend-receivers.md) for more details. | - | - |
| receiver-clr| default| Read [receiver doc](backend-receivers.md) for more details. | - | - |
| receiver-profile| default| Read [receiver doc](backend-receivers.md) for more details. | - | - |
| receiver-zabbix| default| Read [receiver doc](backend-zabbix.md) for more details. | - | - |
| receiver-register|default| gRPC and HTTPRestful services that provide service, service instance and endpoint register. | - | - |
| receiver-trace|default| gRPC and HTTPRestful services that accept SkyWalking format traces. | - | - |
| receiver-jvm| default| gRPC services that accept JVM metrics data. | - | - |
| receiver-clr| default|gRPC services that accept .Net CLR metrics data. | - | - |
| receiver-profile| default| gRPC services that accept profile task status and snapshot reporter. | - | - |
| receiver-zabbix| default| TCP receiver accepts Zabbix format metrics. | - | - |
| - | - | port| Exported TCP port. Zabbix agent could connect and transport data. | SW_RECEIVER_ZABBIX_PORT | 10051 |
| - | - | host| Binds to host. | SW_RECEIVER_ZABBIX_HOST | 0.0.0.0 |
| - | - | activeFiles| Enables config when agent request is received. | SW_RECEIVER_ZABBIX_ACTIVE_FILES | agent |
| service-mesh| default| Read [receiver doc](backend-receivers.md) for more details. | - | - |
| envoy-metric| default| Read [receiver doc](backend-receivers.md) for more details. | - | - |
| service-mesh| default| gRPC services that accept data from inbound mesh probes. | - | - |
| envoy-metric| default| Envoy `metrics_service` and `ALS(access log service)` are supported by this receiver. The OAL script supports all GAUGE type metrics. | - | - |
| - | - | acceptMetricsService | Starts Envoy Metrics Service analysis. | SW_ENVOY_METRIC_SERVICE | true|
| - | - | alsHTTPAnalysis | Starts Envoy HTTP Access Log Service analysis. Value = `k8s-mesh` means starting the analysis. | SW_ENVOY_METRIC_ALS_HTTP_ANALYSIS | - |
| - | - | alsTCPAnalysis | Starts Envoy TCP Access Log Service analysis. Value = `k8s-mesh` means starting the analysis. | SW_ENVOY_METRIC_ALS_TCP_ANALYSIS | - |
| - | - | k8sServiceNameRule | `k8sServiceNameRule` allows you to customize the service name in ALS via Kubernetes metadata. The available variables are `pod` and `service`. E.g. you can use `${service.metadata.name}-${pod.metadata.labels.version}` to append the version number to the service name. Note that when using environment variables to pass this configuration, use single quotes(`''`) to avoid being evaluated by the shell. | - |
| receiver-otel | default | Read [receiver doc](backend-receivers.md) for more details. | - | - |
| receiver-otel | default | A receiver for analyzing metrics data from OpenTelemetry. | - | - |
| - | - | enabledHandlers| Enabled handlers for otel. | SW_OTEL_RECEIVER_ENABLED_HANDLERS | - |
| - | - | enabledOcRules| Enabled metric rules for OC handler. | SW_OTEL_RECEIVER_ENABLED_OC_RULES | - |
| receiver_zipkin |default| Read [receiver doc](backend-receivers.md). | - | - |
| receiver_zipkin |default| A receiver for Zipkin traces. | - | - |
| - | - | restHost| Binding IP of RESTful services. |SW_RECEIVER_ZIPKIN_HOST|0.0.0.0|
| - | - | restPort | Binding port of RESTful services. | SW_RECEIVER_ZIPKIN_PORT|9411|
| - | - | restContextPath| Web context path of RESTful services. | SW_RECEIVER_ZIPKIN_CONTEXT_PATH|/|
| receiver_jaeger | default| Read [receiver doc](backend-receivers.md). | - | - |
| - | - | gRPCHost|Binding IP of gRPC services. Services include gRPC data report and internal communication among OAP nodes. | SW_RECEIVER_JAEGER_HOST | - |
| - | - | gRPCPort| Binding port of gRPC services. | SW_RECEIVER_JAEGER_PORT | - |
| - | - | gRPCThreadPoolSize| Pool size of gRPC server. | - | CPU core * 4|
| - | - | gRPCThreadPoolQueueSize| Queue size of gRPC server. | - | 10000|
| - | - | maxConcurrentCallsPerConnection | The maximum number of concurrent calls permitted for each incoming connection. Defaults to no limit. | - | - |
| - | - | maxMessageSize | Sets the maximum message size allowed to be received on the server. Empty means 4 MiB. | - | 4M(based on Netty) |
| prometheus-fetcher | default | Read [fetcher doc](backend-fetcher.md) for more details. | - | - |
| prometheus-fetcher | default | Prometheus fetcher reads metrics from Prometheus endpoint, and transfer the metrics into SkyWalking native format for the MAL engine. | - | - |
| - | - | enabledRules | Enabled rules. | SW_PROMETHEUS_FETCHER_ENABLED_RULES | self |
| - | - | maxConvertWorker | The maximize meter convert worker. | SW_PROMETHEUS_FETCHER_NUM_CONVERT_WORKER | -1(by default, half the number of CPU core(s)) |
| kafka-fetcher | default | Read [fetcher doc](backend-fetcher.md) for more details. | - | - |
| kafka-fetcher | default | Read SkyWalking's native metrics/logs/traces through Kafka server. | - | - |
| - | - | bootstrapServers | A list of host/port pairs to use for establishing the initial connection to the Kafka cluster. | SW_KAFKA_FETCHER_SERVERS | localhost:9092 |
| - | - | namespace | Namespace aims to isolate multi OAP cluster when using the same Kafka cluster. If you set a namespace for Kafka fetcher, OAP will add a prefix to topic name. You should also set namespace in `agent.config`. The property is named `plugin.kafka.namespace`. | SW_NAMESPACE | - |
| - | - | groupId | A unique string that identifies the consumer group to which this consumer belongs.| - | skywalking-consumer |
......@@ -222,7 +215,7 @@ core|default|role|Option values: `Mixed/Receiver/Aggregator`. **Receiver** mode
| - | - | topicNameOfManagements | Kafka topic name for service instance reporting and registration. | - | skywalking-managements |
| - | - | topicNameOfLogs | Kafka topic name for native proto log data. | - | skywalking-logs |
| - | - | topicNameOfJsonLogs | Kafka topic name for native json log data. | - | skywalking-logs-json |
| receiver-browser | default | Read [receiver doc](backend-receivers.md) for more details. | - | - | - |
| receiver-browser | default | gRPC services that accept browser performance data and error log. | - | - | - |
| - | - | sampleRate | Sampling rate for receiving trace. Precise to 1/10000. 10000 means sampling rate of 100% by default. | SW_RECEIVER_BROWSER_SAMPLE_RATE | 10000 |
| query | graphql | - | GraphQL query implementation. | - |
| - | - | path | Root path of GraphQL query and mutation. | SW_QUERY_GRAPHQL_PATH | /graphql|
......@@ -271,7 +264,7 @@ core|default|role|Option values: `Mixed/Receiver/Aggregator`. **Receiver** mode
| - | - | targetPort | The port of target gRPC server for receiving export data. | SW_EXPORTER_GRPC_PORT | 9870 |
| health-checker | default | checkIntervalSeconds | The period of checking OAP internal health status (in seconds). | SW_HEALTH_CHECKER_INTERVAL_SECONDS | 5 |
| configuration-discovery | default | disableMessageDigest | If true, agent receives the latest configuration every time, even without making any changes. By default, OAP uses the SHA512 message digest mechanism to detect changes in configuration. | SW_DISABLE_MESSAGE_DIGEST | false
| receiver-event|default| Read [receiver doc](backend-receivers.md) for more details. | - | - |
| receiver-event|default| gRPC services that handle events data. | - | - |
## Note
¹ System Environment Variable name could be declared and changed in `application.yml`. The names listed here are simply provided in the default `application.yml` file.
......@@ -68,7 +68,7 @@ Since `recevier-sharing-server` only receives data from an external source, it d
### Enable mTLS mode on gRPC receiver
Since 8.8.0, SkyWalking supports enable mutual TLS authentication for transporting between clients and OAP servers. To enable `mTLS` mode for gRPC channel requires [Sharing gRPC Server](./backend-receivers.md/#grpchttp-server-for-receiver) enabled, as the following configuration.
Since 8.8.0, SkyWalking supports enable mutual TLS authentication for transporting between clients and OAP servers. To enable `mTLS` mode for gRPC channel requires [Sharing gRPC Server](backend-expose.md) enabled, as the following configuration.
```yaml
receiver-sharing-server:
......
# Open Fetcher
Fetcher is a concept in SkyWalking backend. When reading data from target systems, the pull mode is more suitable than the [receiver](backend-receivers.md). This mode is typically found in metrics SDKs, such as Prometheus.
## Prometheus Fetcher
Suppose you want to enable some `metric-custom.yaml` files stored at `fetcher-prom-rules`, append its name to `enabledRules` of
`prometheus-fetcher` as follows:
```yaml
prometheus-fetcher:
selector: ${SW_PROMETHEUS_FETCHER:default}
default:
enabledRules: ${SW_PROMETHEUS_FETCHER_ENABLED_RULES:"self,metric-custom"}
```
### Configuration file
Prometheus fetcher is configured via a configuration file. The configuration file defines everything related to fetching
services and their instances, as well as which rule files to load.
The OAP can load the configuration at bootstrap. If the new configuration is not well-formed, the OAP fails to start up. The files
are located at `$CLASSPATH/fetcher-prom-rules`.
The file is written in YAML format, defined by the scheme described below. Brackets indicate that a parameter is optional.
A full example can be found [here](../../../../oap-server/server-starter/src/main/resources/fetcher-prom-rules/self.yaml)
Generic placeholders are defined as follows:
* `<duration>`: This is parsed into a textual representation of a duration. The formats accepted are based on
the ISO-8601 duration format `PnDTnHnMn.nS` with days considered to be exactly 24 hours.
* `<labelname>`: A string matching the regular expression \[a-zA-Z_\]\[a-zA-Z0-9_\]*.
* `<labelvalue>`: A string of unicode characters.
* `<host>`: A valid string consisting of a hostname or IP followed by an optional port number.
* `<path>`: A valid URL path.
* `<string>`: A regular string.
```yaml
# How frequently to fetch targets.
fetcherInterval: <duration>
# Per-fetch timeout when fetching this target.
fetcherTimeout: <duration>
# The HTTP resource path on which to fetch metrics from targets.
metricsPath: <path>
#Statically configured targets.
staticConfig:
# The targets specified by the static config.
targets:
[ - <target> ]
# Labels assigned to all metrics fetched from the targets.
labels:
[ <labelname>: <labelvalue> ... ]
# expSuffix is appended to all expression in this file.
expSuffix: <string>
# insert metricPrefix into metric name: <metricPrefix>_<raw_metric_name>
metricPrefix: <string>
# Metrics rule allow you to recompute queries.
metricsRules:
[ - <metric_rules> ]
```
#### <target>
```yaml
# The url of target exporter. the format should be complied with "java.net.URI"
url: <string>
# The path of root CA file.
sslCaFilePath: <string>
```
#### <metric_rules>
```yaml
# The name of rule, which combinates with a prefix 'meter_' as the index/table name in storage.
name: <string>
# MAL expression.
exp: <string>
```
To know more about MAL, please refer to [mal.md](../../concepts-and-designs/mal.md)
## Kafka Fetcher
The Kafka Fetcher pulls messages from the Kafka Broker to learn about what agent is delivered. Check the agent documentation for details. Typically, tracing segments, service/instance properties, JVM metrics, and meter system data are supported. Kafka Fetcher can work with gRPC/HTTP Receivers at the same time for adopting different transport protocols.
......
......@@ -11,13 +11,13 @@ HTTP protocol, with the formats [Kafka JSON](../../protocols/Log-Data-Protocol.m
or [HTTP JSON array](../../protocols/Log-Data-Protocol.md#http-api).
#### Filebeat
Filebeat supports using Kafka to transport logs. Open [kafka-fetcher](backend-fetcher.md#kafka-fetcher) and enable configs `enableNativeJsonLog`.
Filebeat supports using Kafka to transport logs. Open [kafka-fetcher](kafka-fetcher.md#kafka-fetcher) and enable configs `enableNativeJsonLog`.
Take the following filebeat config yaml as an example to set up Filebeat:
- [filebeat.yml](../../../../test/e2e-v2/cases/kafka/log/filebeat.yml)
#### Fluentd
Fluentd supports using Kafka to transport logs. Open [kafka-fetcher](backend-fetcher.md#kafka-fetcher) and enable configs `enableNativeJsonLog`.
Fluentd supports using Kafka to transport logs. Open [kafka-fetcher](kafka-fetcher.md#kafka-fetcher) and enable configs `enableNativeJsonLog`.
Take the following fluentd config file as an example to set up Fluentd:
- [fluentd.conf](../../../../test/e2e-v2/cases/kafka/log/fluentd.conf)
......@@ -37,7 +37,7 @@ Java agent provides toolkits for
to report logs through gRPC with automatically injected trace context.
[SkyWalking Satellite sidecar](https://github.com/apache/skywalking-satellite) is a recommended proxy/side that
forwards logs (including the use of Kafka MQ to transport logs). When using this, open [kafka-fetcher](backend-fetcher.md#kafka-fetcher)
forwards logs (including the use of Kafka MQ to transport logs). When using this, open [kafka-fetcher](kafka-fetcher.md#kafka-fetcher)
and enable configs `enableNativeProtoLog`.
Java agent provides toolkits for
......
# OpenTelemetry receiver
The OpenTelemetry receiver supports ingesting agent metrics by meter-system. The OAP can load the configuration at bootstrap.
If the new configuration is not well-formed, the OAP may fail to start up. The files are located at `$CLASSPATH/otel-<handler>-rules`.
E.g. The `oc` handler loads rules from `$CLASSPATH/otel-oc-rules`.
Supported handlers:
* `oc`: [OpenCensus](https://github.com/open-telemetry/opentelemetry-collector-contrib/blob/a08903f05d3a544f548535c222b1c205b9f5a154/exporter/opencensusexporter/README.md) gRPC service handler.
**Notice:** Set `SW_OTEL_RECEIVER=default` through system environment or change `receiver-otel/selector=${SW_OTEL_RECEIVER:default}` to activate the OpenTelemetry receiver.
The rule file should be in YAML format, defined by the scheme described in [prometheus-fetcher](./kafka-fetcher.md).
Note: `receiver-otel` only supports the `group`, `defaultMetricLevel`, and `metricsRules` nodes of the scheme due to its push mode.
To activate the `oc` handler and relevant rules of `istio`:
```yaml
receiver-otel:
// Change selector value to default, for activating the otel receiver.
selector: ${SW_OTEL_RECEIVER:default}
default:
enabledHandlers: ${SW_OTEL_RECEIVER_ENABLED_HANDLERS:"oc"}
enabledOcRules: ${SW_OTEL_RECEIVER_ENABLED_OC_RULES:"istio-controlplane"}
```
The receiver adds labels with `key = node_identifier_host_name` and `key = node_identifier_pid` to the collected data samples,
and values from `Node.identifier.host_name` and `Node.identifier.pid` defined in OpenCensus Agent Proto,
for identification of the metric data.
| Rule Name | Description | Configuration File | Data Source |
|----|----|-----|----|
|istio-controlplane| Metrics of Istio control panel | otel-oc-rules/istio-controlplane.yaml | Istio Control Panel -> OpenTelemetry Collector --OC format--> SkyWalking OAP Server |
|oap| Metrics of SkyWalking OAP server itself | otel-oc-rules/oap.yaml | SkyWalking OAP Server(SelfObservability) -> OpenTelemetry Collector --OC format--> SkyWalking OAP Server |
|vm| Metrics of VMs | otel-oc-rules/vm.yaml | Prometheus node-exporter(VMs) -> OpenTelemetry Collector --OC format--> SkyWalking OAP Server |
|k8s-cluster| Metrics of K8s cluster | otel-oc-rules/k8s-cluster.yaml | K8s kube-state-metrics -> OpenTelemetry Collector --OC format--> SkyWalking OAP Server |
|k8s-node| Metrics of K8s cluster | otel-oc-rules/k8s-node.yaml | cAdvisor & K8s kube-state-metrics -> OpenTelemetry Collector --OC format--> SkyWalking OAP Server |
|k8s-service| Metrics of K8s cluster | otel-oc-rules/k8s-service.yaml | cAdvisor & K8s kube-state-metrics -> OpenTelemetry Collector --OC format--> SkyWalking OAP Server |
\ No newline at end of file
# Prometheus Fetcher
Prometheus fetcher reads metrics from Prometheus endpoint, and transfer the metrics into SkyWalking native format for the MAL engine.
## Configuration file
Prometheus fetcher is configured via a configuration file. The configuration file defines everything related to fetching
services and their instances, as well as which rule files to load.
The OAP can load the configuration at bootstrap. If the new configuration is not well-formed, the OAP fails to start up. The files
are located at `$CLASSPATH/fetcher-prom-rules`.
The file is written in YAML format, defined by the scheme described below. Brackets indicate that a parameter is optional.
A full example can be found [here](../../../../oap-server/server-starter/src/main/resources/fetcher-prom-rules/self.yaml)
Generic placeholders are defined as follows:
* `<duration>`: This is parsed into a textual representation of a duration. The formats accepted are based on
the ISO-8601 duration format `PnDTnHnMn.nS` with days considered to be exactly 24 hours.
* `<labelname>`: A string matching the regular expression \[a-zA-Z_\]\[a-zA-Z0-9_\]*.
* `<labelvalue>`: A string of unicode characters.
* `<host>`: A valid string consisting of a hostname or IP followed by an optional port number.
* `<path>`: A valid URL path.
* `<string>`: A regular string.
```yaml
# How frequently to fetch targets.
fetcherInterval: <duration>
# Per-fetch timeout when fetching this target.
fetcherTimeout: <duration>
# The HTTP resource path on which to fetch metrics from targets.
metricsPath: <path>
#Statically configured targets.
staticConfig:
# The targets specified by the static config.
targets:
[ - <target> ]
# Labels assigned to all metrics fetched from the targets.
labels:
[ <labelname>: <labelvalue> ... ]
# expSuffix is appended to all expression in this file.
expSuffix: <string>
# insert metricPrefix into metric name: <metricPrefix>_<raw_metric_name>
metricPrefix: <string>
# Metrics rule allow you to recompute queries.
metricsRules:
[ - <metric_rules> ]
```
### <target>
```yaml
# The url of target exporter. the format should be complied with "java.net.URI"
url: <string>
# The path of root CA file.
sslCaFilePath: <string>
```
### <metric_rules>
```yaml
# The name of rule, which combinates with a prefix 'meter_' as the index/table name in storage.
name: <string>
# MAL expression.
exp: <string>
```
To know more about MAL, please refer to [mal.md](../../concepts-and-designs/mal.md)
## Active Fetcher Rules
Suppose you want to enable some `metric-custom.yaml` files stored at `fetcher-prom-rules`, append its name to `enabledRules` of
`prometheus-fetcher` as follows:
```yaml
prometheus-fetcher:
selector: ${SW_PROMETHEUS_FETCHER:default}
default:
enabledRules: ${SW_PROMETHEUS_FETCHER_ENABLED_RULES:"self,metric-custom"}
```
## Zipkin receiver
The Zipkin receiver makes the OAP server work as an alternative Zipkin server implementation. It supports Zipkin v1/v2 formats through HTTP service.
Make sure you use this with `SW_STORAGE=zipkin-elasticsearch` option to activate Zipkin storage implementation.
Once this receiver and storage are activated, SkyWalking's native traces would be ignored, and SkyWalking wouldn't analyze topology, metrics, and endpoint
dependency from Zipkin's trace.
Use the following config to activate it.
```yaml
receiver_zipkin:
selector: ${SW_RECEIVER_ZIPKIN:-}
default:
host: ${SW_RECEIVER_ZIPKIN_HOST:0.0.0.0}
port: ${SW_RECEIVER_ZIPKIN_PORT:9411}
contextPath: ${SW_RECEIVER_ZIPKIN_CONTEXT_PATH:/}
jettyMinThreads: ${SW_RECEIVER_ZIPKIN_JETTY_MIN_THREADS:1}
jettyMaxThreads: ${SW_RECEIVER_ZIPKIN_JETTY_MAX_THREADS:200}
jettyIdleTimeOut: ${SW_RECEIVER_ZIPKIN_JETTY_IDLE_TIMEOUT:30000}
jettyAcceptorPriorityDelta: ${SW_RECEIVER_ZIPKIN_JETTY_DELTA:0}
jettyAcceptQueueSize: ${SW_RECEIVER_ZIPKIN_QUEUE_SIZE:0}
```
NOTE: Zipkin receiver requires `zipkin-elasticsearch` storage implementation to be activated.
Read [this](backend-storage.md#elasticsearch-with-zipkin-trace-extension) doc to learn about Zipkin as a storage option.
......@@ -27,7 +27,12 @@ In Istio version 1.6.0+, if Istio is installed with [`demo` profile](https://ist
Note: Replace `<skywalking-oap.skywalking.svc:11800>` with the real address where SkyWalking OAP is deployed.
- Activate SkyWalking [Envoy Receiver](../backend/backend-receivers.md). This is activated by default.
- Activate SkyWalking Envoy Receiver. (activated in default)
```yaml
envoy-metric:
selector: ${SW_ENVOY_METRIC:default}
```
- Choose an ALS analyzer. There are two available analyzers for both HTTP access logs and TCP access logs: `k8s-mesh` and `mx-mesh`.
Set the system environment variables **SW_ENVOY_METRIC_ALS_HTTP_ANALYSIS** and **SW_ENVOY_METRIC_ALS_TCP_ANALYSIS**,
......
......@@ -9,7 +9,7 @@ Istio should be installed in the Kubernetes cluster. Simply follow the steps in
## Deploying SkyWalking backend
Follow the steps in [deploying backend in Kubernetes](../backend/backend-k8s.md) to install the OAP server in the Kubernetes cluster.
Refer to [OpenTelemetry receiver](../backend/backend-receivers.md#OpenTelemetry-receiver) to ingest metrics.
Refer to [OpenTelemetry receiver](../backend/opentelemetry-receiver.md) to ingest metrics.
`otel-receiver` is disabled by default. Set env var `SW_OTEL_RECEIVER` to `default` to enable it.
......
......@@ -8,4 +8,12 @@ It has these features:
See Client JS [official doc](https://github.com/apache/skywalking-client-js#quick-start) for more information.
Note: Make sure [`receiver-browser`](../backend/backend-receivers.md) is enabled. It is **ON** by default since version 8.2.0.
Note: Make sure receiver-browser is enabled. It is **ON** by default since version 8.2.0.
```yaml
receiver-browser:
selector: ${SW_RECEIVER_BROWSER:default} // This means activated.
default:
# The sample rate precision is 1/10000. 10000 means 100% sample in default.
sampleRate: ${SW_RECEIVER_BROWSER_SAMPLE_RATE:10000}
```
......@@ -85,10 +85,10 @@ catalog:
path: "/en/setup/backend/backend-cluster"
- name: "Choose Storage"
path: "/en/setup/backend/backend-storage"
- name: "Set Receivers"
path: "/en/setup/backend/backend-receivers"
- name: "Open Fetchers"
path: "/en/setup/backend/backend-fetcher"
- name: "Setup External Communication Channels"
path: "/en/setup/backend/backend-expose"
- name: "Kafka Fetcher"
path: "/en/setup/backend/Kafka-fetcher"
- name: "Advanced Deployment Options"
path: "/en/setup/backend/advanced-deployment"
- name: "Data Lifecycle. Time To Live (TTL)"
......@@ -111,18 +111,24 @@ catalog:
path: "/en/setup/backend/mq"
- name: "Uninstrumented Gateways"
path: "/en/setup/backend/uninstrumented-gateways"
- name: "Zipkin Trace"
path: "/en/setup/backend/zipkin-trace"
- name: "Metrics"
catalog:
- name: "OAL Scripts"
path: "/en/guides/backend-oal-scripts"
- name: "OpenTelemetry Metrics Analysis"
path: "/en/setup/backend/backend-receivers#opentelemetry-receiver"
- name: "OpenTelemetry Metrics"
path: "/en/setup/backend/opentelemetry-receiver"
- name: "Zabbix Metrics"
path: "/en/setup/backend/backend-zabbix"
- name: "Meter Analysis"
path: "/en/setup/backend/backend-meter"
- name: "Apdex Threshold"
path: "/en/setup/backend/apdex-threshold"
- name: "Spring Sleuth Metrics Analysis"
path: "/en/setup/backend/spring-sleuth-setup"
- name: "Prometheus Metrics"
path: "/en/setup/backend/prometheus-metrics"
- name: "Alarm"
path: "/en/setup/backend/backend-alarm"
- name: "Logging"
......
Markdown is supported
0% .
You are about to add 0 people to the discussion. Proceed with caution.
先完成此消息的编辑!
想要评论请 注册