SkyWalking leverages K8s kube-state-metrics (KSM) and cAdvisor for collecting metrics data from K8s. It leverages OpenTelemetry Collector to transfer the metrics to
[OpenTelemetry receiver](opentelemetry-receiver.md) and into the [Meter System](./../../concepts-and-designs/meter.md). This feature requires authorizing the OAP Server to access K8s's `API Server`.
## Data flow
1. K8s kube-state-metrics and cAdvisor collect metrics data from K8s.
2. OpenTelemetry Collector fetches metrics from kube-state-metrics and cAdvisor via Prometheus Receiver and pushes metrics to SkyWalking OAP Server via the OpenCensus GRPC Exporter.
3. The SkyWalking OAP Server access to K8s's `API Server` gets meta info and parses the expression with [MAL](../../concepts-and-designs/mal.md) to filter/calculate/aggregate and store the results.
3. The SkyWalking OAP Server access to K8s's `API Server` gets meta info and parses the expression with [MAL](../../concepts-and-designs/mal.md) to filter/calculate/aggregate and store the results.
2. cAdvisor is integrated into `kubelet` by default.
3. Set up [OpenTelemetry Collector ](https://opentelemetry.io/docs/collector/getting-started/#kubernetes). For details on Prometheus Receiver in OpenTelemetry Collector for K8s, refer to [here](https://github.com/prometheus/prometheus/blob/main/documentation/examples/prometheus-kubernetes.yml).
3. Set up [OpenTelemetry Collector ](https://opentelemetry.io/docs/collector/getting-started/#kubernetes). For details on Prometheus Receiver in OpenTelemetry Collector for K8s, refer to [here](https://github.com/prometheus/prometheus/blob/main/documentation/examples/prometheus-kubernetes.yml).
For a quick start, we have provided a complete example of configuration and recommended version; you can refer to [showcase](https://github.com/apache/skywalking-showcase/tree/main/deploy/platform/kubernetes/feature-kubernetes-monitor).
@@ -67,7 +67,7 @@ K8s Service as a `Service` in OAP and land on the `Layer: K8S_SERVICE`.
| Pod Restarts | | k8s_service_pod_status_restarts_total | The number of per container restarts related to the pods | K8s kube-state-metrics |
## Customizations
You can customize your own metrics/expression/dashboard panel.
The metrics definition and expression rules are found in `/config/otel-oc-rules/k8s-cluster.yaml,/config/otel-oc-rules/k8s-node.yaml, /config/otel-oc-rules/k8s-service.yaml`.
You can customize your own metrics/expression/dashboard panel.
The metrics definition and expression rules are found in `/config/otel-rules/k8s-cluster.yaml,/config/otel-rules/k8s-node.yaml, /config/otel-rules/k8s-service.yaml`.
The K8s Cluster dashboard panel configurations are found in `/config/ui-initialized-templates/k8s`.
The K8s Service dashboard panel configurations are found in `/config/ui-initialized-templates/k8s_service`.
SkyWalking leverages prometheus/mysqld_exporter for collecting metrics data from MySQL. It leverages OpenTelemetry Collector to transfer the metrics to
[OpenTelemetry receiver](opentelemetry-receiver.md) and into the [Meter System](./../../concepts-and-designs/meter.md).
[OpenTelemetry receiver](opentelemetry-receiver.md) and into the [Meter System](./../../concepts-and-designs/meter.md).
## Data flow
1. mysqld_exporter collect metrics data from MySQL.
2. OpenTelemetry Collector fetches metrics from mysqld_exporter via Prometheus Receiver and pushes metrics to SkyWalking OAP Server via the OpenCensus GRPC Exporter.
3. The SkyWalking OAP Server parses the expression with [MAL](../../concepts-and-designs/mal.md) to filter/calculate/aggregate and store the results.
3. The SkyWalking OAP Server parses the expression with [MAL](../../concepts-and-designs/mal.md) to filter/calculate/aggregate and store the results.
## Set up
## Set up
1. Set up [mysqld_exporter](https://github.com/prometheus/mysqld_exporter#using-docker).
2. Set up [OpenTelemetry Collector ](https://opentelemetry.io/docs/collector/getting-started/#docker). For details on Prometheus Receiver in OpenTelemetry Collector, refer to [here](../../../../test/e2e-v2/cases/mysql/prometheus-mysql-exporter/otel-collector-config.yaml).
2. Set up [OpenTelemetry Collector ](https://opentelemetry.io/docs/collector/getting-started/#docker). For details on Prometheus Receiver in OpenTelemetry Collector, refer to [here](../../../../test/e2e-v2/cases/mysql/prometheus-mysql-exporter/otel-collector-config.yaml).
@@ -22,7 +22,7 @@ MySQL monitoring provides monitoring of the status and resources of the MySQL se
| Max Connections | | meter_mysql_max_connections | The max number of connections. | mysqld_exporter|
| Innodb Buffer Pool Size | MB | meter_mysql_innodb_buffer_pool_size | The buffer pool size in Innodb engine | mysqld_exporter|
| Thread Cache Size | | meter_mysql_thread_cache_size | The size of thread cache | mysqld_exporter|
| Current QPS| | meter_mysql_qps | Queries Per Second | mysqld_exporter|
| Current QPS| | meter_mysql_qps | Queries Per Second | mysqld_exporter|
| Current TPS | | meter_mysql_tps | Transactions Per Second | mysqld_exporter|
| Commands Rate | | meter_mysql_commands_insert_rate <br/>meter_mysql_commands_select_rate<br/>meter_mysql_commands_delete_rate<br/>meter_mysql_commands_update_rate | The rate of total number of insert/select/delete/update executed by the current server | mysqld_exporter|
| Threads | | meter_mysql_threads_connected<br/>meter_mysql_threads_created<br/>meter_mysql_threads_cached<br/>meter_mysql_threads_running | The number of currently open connections(threads_connected) <br/> The number of threads created(threads_created) <br/> The number of threads in the thread cache(threads_cached) <br/> The number of threads that are not sleeping(threads_running) | mysqld_exporter|
...
...
@@ -32,8 +32,6 @@ MySQL monitoring provides monitoring of the status and resources of the MySQL se
## Customizations
You can customize your own metrics/expression/dashboard panel.
The metrics definition and expression rules are found in `/config/otel-oc-rules/mysql.yaml`.
You can customize your own metrics/expression/dashboard panel.
The metrics definition and expression rules are found in `/config/otel-rules/mysql.yaml`.
The MySQL dashboard panel configurations are found in `/config/ui-initialized-templates/mysql`.
SkyWalking leverages Prometheus node-exporter to collect metrics data from the VMs and leverages OpenTelemetry Collector to transfer the metrics to
[OpenTelemetry receiver](opentelemetry-receiver.md) and into the [Meter System](./../../concepts-and-designs/meter.md).
VM entity as a `Service` in OAP and on the `Layer: OS_LINUX`.
[OpenTelemetry receiver](opentelemetry-receiver.md) and into the [Meter System](./../../concepts-and-designs/meter.md).
VM entity as a `Service` in OAP and on the `Layer: OS_LINUX`.
## Data flow
1. The Prometheus node-exporter collects metrics data from the VMs.
2. The OpenTelemetry Collector fetches metrics from node-exporter via Prometheus Receiver and pushes metrics to the SkyWalking OAP Server via the OpenCensus gRPC Exporter.
3. The SkyWalking OAP Server parses the expression with [MAL](../../concepts-and-designs/mal.md) to filter/calculate/aggregate and store the results.
3. The SkyWalking OAP Server parses the expression with [MAL](../../concepts-and-designs/mal.md) to filter/calculate/aggregate and store the results.
## Setup
...
...
@@ -14,7 +14,7 @@ VM entity as a `Service` in OAP and on the `Layer: OS_LINUX`.
2. Setup [OpenTelemetry Collector ](https://opentelemetry.io/docs/collector/). This is an example for OpenTelemetry Collector configuration [otel-collector-config.yaml](../../../../test/e2e-v2/cases/vm/prometheus-node-exporter/otel-collector-config.yaml).
| Monitoring Panel | Unit | Metric Name | Description | Data Source |
...
...
@@ -32,9 +32,9 @@ VM entity as a `Service` in OAP and on the `Layer: OS_LINUX`.
| Network Status | | meter_vm_tcp_curr_estab<br/>meter_vm_tcp_tw<br/>meter_vm_tcp_alloc<br/>meter_vm_sockets_used<br/>meter_vm_udp_inuse | The number of TCPs established / TCP time wait / TCPs allocated / sockets in use / UDPs in use | Prometheus node-exporter |
| Filefd Allocated | | meter_vm_filefd_allocated | The number of file descriptors allocated | Prometheus node-exporter |
## Customizing
You can customize your own metrics/expression/dashboard panel.
The metrics definition and expression rules are found in `/config/otel-oc-rules/vm.yaml`.
## Customizing
You can customize your own metrics/expression/dashboard panel.
The metrics definition and expression rules are found in `/config/otel-rules/vm.yaml`.
The dashboard panel confirmations are found in `/config/ui-initialized-templates/os_linux`.
| receiver-zipkin | default | A receiver for Zipkin traces. | - | - | |
| - | - | sampleRate | The sample rate precision is 1/10000, should be between 0 and 10000 | SW_ZIPKIN_SAMPLE_RATE | 10000 |
| - | - | searchableTracesTags | Defines a set of span tag keys which are searchable. Multiple values are separated by commas. | SW_ZIPKIN_SEARCHABLE_TAG_KEYS | http.method |
The OpenTelemetry receiver supports ingesting agent metrics by meter-system. The OAP can load the configuration at bootstrap.
If the new configuration is not well-formed, the OAP may fail to start up. The files are located at `$CLASSPATH/otel-<handler>-rules`.
E.g. The `oc` handler loads rules from `$CLASSPATH/otel-oc-rules`.
If the new configuration is not well-formed, the OAP may fail to start up. The files are located at `$CLASSPATH/otel-rules`.
Supported handlers:
*`oc`: [OpenCensus](https://github.com/open-telemetry/opentelemetry-collector-contrib/blob/a08903f05d3a544f548535c222b1c205b9f5a154/exporter/opencensusexporter/README.md) gRPC service handler.
*`otlp`: [OpenTelemetry](https://github.com/open-telemetry/opentelemetry-collector/tree/main/exporter/otlpexporter) gRPC service handler.
**Notice:** Set `SW_OTEL_RECEIVER=default` through system environment or change `receiver-otel/selector=${SW_OTEL_RECEIVER:default}` to activate the OpenTelemetry receiver.
...
...
@@ -29,12 +29,12 @@ for identification of the metric data.
| Rule Name | Description | Configuration File | Data Source |
|----|----|-----|----|
|istio-controlplane| Metrics of Istio Control Plane | otel-oc-rules/istio-controlplane.yaml | Istio Control Plane -> OpenTelemetry Collector --OC format--> SkyWalking OAP Server |
|oap| Metrics of SkyWalking OAP server itself | otel-oc-rules/oap.yaml | SkyWalking OAP Server(SelfObservability) -> OpenTelemetry Collector --OC format--> SkyWalking OAP Server |
|vm| Metrics of VMs | otel-oc-rules/vm.yaml | Prometheus node-exporter(VMs) -> OpenTelemetry Collector --OC format--> SkyWalking OAP Server |
|k8s-cluster| Metrics of K8s cluster | otel-oc-rules/k8s-cluster.yaml | K8s kube-state-metrics -> OpenTelemetry Collector --OC format--> SkyWalking OAP Server |
|mysql| Metrics of MYSQL| otel-rules/mysql.yaml | prometheus/mysqld_exporter -> OpenTelemetry Collector --OC format--> SkyWalking OAP Server |
**Note**: You can also use OpenTelemetry exporter to transport the metrics to SkyWalking OAP directly. See [OpenTelemetry Exporter](./backend-meter.md#opentelemetry-exporter).