Merge branch '29142-add-prometheus-integration-documentation' into 'master'

Prometheus Service Integration Documentation

Closes #29142

See merge request !9954

doc/administration/monitoring/prometheus/index.md

@@ -3,10 +3,9 @@
 >**Notes:**
 - Prometheus and the various exporters listed in this page are bundled in the
   Omnibus GitLab package. Check each exporter's documentation for the timeline
-  they got added. For installations from source you will have to install
-  them yourself. Over subsequent releases additional GitLab metrics will be
-  captured.
-- Prometheus services are off by default but will be on starting with GitLab 9.0.
+  they got added. For installations from source you will have to install them
+  yourself. Over subsequent releases additional GitLab metrics will be captured.
+- Prometheus services are on by default with GitLab 9.0.
 - Prometheus and its exporters do not authenticate users, and will be available
   to anyone who can access them.
 
@@ -26,26 +25,25 @@ dashboard tool like [Grafana].
 ## Configuring Prometheus
 
 >**Note:**
-Available since Omnibus GitLab 8.16. For installations from source you'll
-have to install and configure it yourself.
+For installations from source you'll have to install and configure it yourself.
 
-To enable Prometheus:
+Prometheus and it's exporters are on by default, starting with GitLab 9.0.
+Prometheus will run as the `gitlab-prometheus` user and listen on
+`http://localhost:9090`. Each exporter will be automatically be set up as a
+monitoring target for Prometheus, unless individually disabled.
+
+To disable Prometheus and all of its exporters, as well as any added in the future:
 
 1. Edit `/etc/gitlab/gitlab.rb`
 1. Add or find and uncomment the following line, making sure it's set to `true`:
 
     ```ruby
-    prometheus['enable'] = true
+    prometheus['disable_all'] = true
     ```
 
 1. Save the file and [reconfigure GitLab][reconfigure] for the changes to
    take effect
 
-By default, Prometheus will run as the `gitlab-prometheus` user and listen on
-`http://localhost:9090`. If the [node exporter](#node-exporter) service
-has been enabled, it will automatically be set up as a monitoring target for
-Prometheus.
-
 ## Changing the port Prometheus listens on
 
 >**Note:**
@@ -71,16 +69,14 @@ To change the address/port that Prometheus listens on:
 
 ## Viewing performance metrics
 
-After you have [enabled Prometheus](#configuring-prometheus), you can visit
-`http://localhost:9090` for the dashboard that Prometheus offers by default.
+You can visit `http://localhost:9090` for the dashboard that Prometheus offers by default.
 
 >**Note:**
 If SSL has been enabled on your GitLab instance, you may not be able to access
 Prometheus on the same browser as GitLab due to [HSTS][hsts]. We plan to
 [provide access via GitLab][multi-user-prometheus], but in the interim there are
 some workarounds: using a separate browser for Prometheus, resetting HSTS, or
-having [Nginx proxy it][nginx-custom-config]. Follow issue [#27069] for more
-information.
+having [Nginx proxy it][nginx-custom-config].
 
 The performance data collected by Prometheus can be viewed directly in the
 Prometheus console or through a compatible dashboard tool.
@@ -96,6 +92,30 @@ Sample Prometheus queries:
 - **Data transmitted:** `irate(node_network_transmit_bytes[5m])`
 - **Data received:** `irate(node_network_receive_bytes[5m])`
 
+## Configuring Prometheus to monitor Kubernetes
+
+> Introduced in GitLab 9.0.
+
+If your GitLab server is running within Kubernetes, an option is now available
+to monitor the health of each node in the cluster. This is particularly helpful
+if your CI/CD environments run in the same cluster, and you would like enable
+[Prometheus integration][] to monitor them.
+
+When enabled, the bundled Prometheus server monitors Kubernetes and automatically
+[collects metrics][prometheus-cadvisor-metrics] from each Node in the cluster.
+
+To enable the Kubernetes monitoring:
+
+1. Edit `/etc/gitlab/gitlab.rb`
+1. Add or find and uncomment the following line:
+
+    ```ruby
+    prometheus['monitor_kubernetes'] = true
+    ```
+
+1. Save the file and [reconfigure GitLab][reconfigure] for the changes to
+   take effect
+
 ## Prometheus exporters
 
 There are a number of libraries and servers which help in exporting existing
@@ -143,5 +163,6 @@ The GitLab monitor exporter allows you to measure various GitLab metrics.
 [prom-grafana]: https://prometheus.io/docs/visualization/grafana/
 [scrape-config]: https://prometheus.io/docs/operating/configuration/#%3Cscrape_config%3E
 [reconfigure]: ../../restart_gitlab.md#omnibus-gitlab-reconfigure
-[#27069]: https://gitlab.com/gitlab-org/gitlab-ce/issues/27069
 [1261]: https://gitlab.com/gitlab-org/omnibus-gitlab/merge_requests/1261
+[prometheus integration]: ../../../user/project/integrations/prometheus.md
+[rometheus-cadvisor-metrics]: https://github.com/google/cadvisor/blob/master/docs/storage/prometheus.md

doc/install/requirements.md

@@ -108,7 +108,7 @@ use the CI features.
 
 ## Unicorn Workers
 
-It's possible to increase the amount of unicorn workers and this will usually help for to reduce the response time of the applications and increase the ability to handle parallel requests.
+It's possible to increase the amount of unicorn workers and this will usually help to reduce the response time of the applications and increase the ability to handle parallel requests.
 
 For most instances we recommend using: CPU cores + 1 = unicorn workers.
 So for a machine with 2 cores, 3 unicorn workers is ideal.
@@ -148,6 +148,16 @@ Sidekiq processes the background jobs with a multithreaded process.
 This process starts with the entire Rails stack (200MB+) but it can grow over time due to memory leaks.
 On a very active server (10,000 active users) the Sidekiq process can use 1GB+ of memory.
 
+## Prometheus and its exporters
+
+As of Omnibus GitLab 9.0, [Prometheus](https://prometheus.io) and its related
+exporters are enabled by default, to enable easy and in depth monitoring of
+GitLab. Approximately 200MB of memory will be consumed by these processes, with
+default settings.
+
+If you would like to disable Prometheus and it's exporters or read more information
+about it, check the [Prometheus documentation](../administration/monitoring/prometheus/index.md).
+
 ## Supported web browsers
 
 We support the current and the previous major release of Firefox, Chrome/Chromium, Safari and Microsoft browsers (Microsoft Edge and Internet Explorer 11).

doc/user/project/integrations/project_services.md

@@ -51,6 +51,7 @@ Click on the service links to see further configuration instructions and details
 | [Slack Notifications](slack.md) | Receive event notifications in Slack |
 | [Slack slash commands](slack_slash_commands.md) | Slack chat and ChatOps slash commands |
 | PivotalTracker | Project Management Software (Source Commits Endpoint) |
+| [Prometheus](prometheus.md) | Monitor the performance of your deployed apps |
 | Pushover | Pushover makes it easy to get real-time notifications on your Android device, iPhone, iPad, and Desktop |
 | [Redmine](redmine.md) | Redmine issue tracker |
 

doc/user/project/integrations/prometheus.md

+# Prometheus integration
+
+> [Introduced][ce-8935] in GitLab 9.0.
+
+GitLab offers powerful integration with [Prometheus] for monitoring your apps.
+Metrics are retrieved from the configured Prometheus server, and then displayed
+within the GitLab interface.
+
+Each project can be configured with its own specific Prometheus server, see the
+[configuration](#configuration) section for more details. If you have a single
+Prometheus server which monitors all of your infrastructure, you can pre-fill
+the settings page with a default template. To configure the template, see the
+[Services templates](services_templates.md) document.
+
+## Requirements
+
+Integration with Prometheus requires the following:
+
+1. GitLab 9.0 or higher
+1. Your app must be deployed on [Kubernetes][]
+1. Prometheus must be configured to collect Kubernetes metrics
+1. Each metric must be have a label to indicate the environment
+1. GitLab must have network connectivity to the Prometheus sever
+
+There are a few steps necessary to set up integration between Prometheus and
+GitLab.
+
+## Configuring Prometheus to collect Kubernetes metrics
+
+In order for Prometheus to collect Kubernetes metrics, you first must have a
+Prometheus server up and running. You have two options here:
+
+- If you installed Omnibus GitLab inside of Kubernetes, you can simply use the
+  [bundled version of Prometheus][promgldocs]. In that case, follow the info in the
+  [Omnibus GitLab section](#configuring-omnibus-gitlab-prometheus-to-monitor-kubernetes)
+  below.
+- If you are using GitLab.com or installed GitLab outside of Kubernetes, you
+  will likely need to run a Prometheus server within the Kubernetes cluster.
+  Once installed, the easiest way to monitor Kubernetes is to simply use
+  Prometheus' support for [Kubernetes Service Discovery][prometheus-k8s-sd].
+  In that case, follow the instructions on
+  [configuring your own Prometheus server within Kubernetes](#configuring-your-own-prometheus-server-within-kubernetes).
+
+### Configuring Omnibus GitLab Prometheus to monitor Kubernetes
+
+With Omnibus GitLab running inside of Kubernetes, you can leverage the bundled
+version of Prometheus to collect the required metrics.
+
+1. Read how to configure the bundled Prometheus server in the
+   [Administration guide][gitlab-prometheus-k8s-monitor].
+1. Now that Prometheus is configured, proceed on
+   [configuring the Prometheus project service in GitLab](#configuration-in-gitlab).
+
+### Configuring your own Prometheus server within Kubernetes
+
+Setting up and configuring Prometheus within Kubernetes is quick and painless.
+The Prometheus project provides an [official Docker image][prometheus-docker-image]
+which we can use as a starting point.
+
+To get started quickly, we have provided a [sample YML file][prometheus-yml]
+that can be used as a template. This file will create a `prometheus` **Namespace**,
+**Service**, **Deployment**, and **ConfigMap** in Kubernetes. You can upload
+this file to the Kubernetes dashboard using **+ Create** at the top right.
+
+![Deploy Prometheus](img/prometheus_yaml_deploy.png)
+
+Or use `kubectl`:
+
+```bash
+kubectl apply -f path/to/prometheus.yml
+```
+
+Once deployed, you should see the Prometheus service, deployment, and
+pod start within the `prometheus` namespace. The server will begin to collect
+metrics from each Kubernetes Node in the cluster, based on the configuration
+provided in the template.
+
+Since GitLab is not running within Kubernetes, the template provides external
+network access via a `NodePort` running on `30090`. This method allows access
+to be controlled using provider firewall rules, like within Google Compute Engine.
+
+Since a `NodePort` does not automatically have firewall rules created for it,
+one will need to be created manually to allow access. In GCP/GKE, you will want
+to confirm the Node that the Prometheus pod is running on. This can be done
+either by looking at the Pod in the Kubernetes dashboard, or by running:
+
+```bash
+kubectl describe pods -n prometheus
+```
+
+Next on GKE, we need to get the `tag` of the Node or VM Instance, so we can
+create an accurate firewall rule. The easiest way to do this is to go into the
+Google Cloud Platform Compute console and select the VM instance that matches
+the name of the Node gathered from the step above. In this case, the node tag
+needed is `gke-prometheus-demo-5d5ada10-node`. Also make a note of the
+**External IP**, which will be the IP address the Prometheus server is reachable
+on.
+
+![GCP Node Detail](img/prometheus_gcp_node_name.png)
+
+Armed with the proper Node tag, the firewall rule can now be created
+specifically for this node. To create the firewall rule, open the Google Cloud
+Platform Networking console, and select **Firewall Rules**.
+
+Create a new rule:
+
+- Specify the source IP range to match your desired access list, which should
+  include your GitLab server. A sample of GitLab.com's IP address range is
+  available [in this issue][gitlab.com-ip-range], but note that GitLab.com's IPs
+  are subject to change without prior notification.
+- Allowed protocol and port should be `tcp:30090`.
+- The target tags should match the Node tag identified earlier in this step.
+
+![GCP Firewall Rule](img/prometheus_gcp_firewall_rule.png)
+
+---
+
+Now that Prometheus is configured, proceed to
+[configure the Prometheus project service in GitLab](##configuration-in-gitlab).
+
+## Configuration in GitLab
+
+The actual configuration of Prometheus integration within GitLab is very simple.
+All you will need is the DNS or IP address of the Prometheus server you'd like
+to integrate with.
+
+1. Navigate to the [Integrations page](project_services.md#accessing-the-project-services)
+1. Click the **Prometheus** service
+1. Provide the base URL of the your server, for example `http://prometheus.example.com/`.
+   The **Test Settings** button can be used to confirm connectivity from GitLab
+   to the Prometheus server.
+
+![Configure Prometheus Service](img/prometheus_service_configuration.png)
+
+## Metrics and Labels
+
+GitLab retrieves performance data from two metrics, `container_cpu_usage_seconds_total`
+and `container_memory_usage_bytes`. These metrics are collected from the
+Kubernetes pods via Prometheus, and report CPU and Memory utilization of each
+container or Pod running in the cluster.
+
+In order to isolate and only display relevant metrics for a given environment
+however, GitLab needs a method to detect which pods are associated. To do that,
+GitLab will specifically request metrics that have an `environment` tag that
+matches the [$CI_ENVIRONMENT_SLUG][ci-environment-slug].
+
+If you are using [GitLab Auto-Deploy][autodeploy] and one of the methods of
+configuring Prometheus above, the `environment` will be automatically added.
+
+### GitLab Prometheus queries
+
+The queries utilized by GitLab are shown in the following table.
+
+| Metric | Query |
+| ------ | ----- |
+| Average Memory (MB) | `(sum(container_memory_usage_bytes{container_name="app",environment="$CI_ENVIRONMENT_SLUG"}) / count(container_memory_usage_bytes{container_name="app",environment="$CI_ENVIRONMENT_SLUG"})) /1024/1024` |
+| Average CPU Utilization (%) | `sum(rate(container_cpu_usage_seconds_total{container_name="app",environment="$CI_ENVIRONMENT_SLUG"}[2m])) / count(container_cpu_usage_seconds_total{container_name="app",environment="$CI_ENVIRONMENT_SLUG"}) * 100` |
+
+## Monitoring CI/CD Environments
+
+Once configured, GitLab will attempt to retrieve performance metrics for any
+environment which has had a successful deployment. If monitoring data was
+successfully retrieved, a metrics button will appear on the environment's
+detail page.
+
+![Environment Detail with Metrics](img/prometheus_environment_detail_with_metrics.png)
+
+Clicking on the metrics button will display a new page, showing up to the last
+8 hours of performance data. It may take a minute or two for data to appear
+after initial deployment.
+
+## Troubleshooting
+
+If the metrics button is not appearing, then one of a few issues may be
+occurring:
+
+- GitLab is not able to reach the Prometheus server. A test request can be sent
+  to the Prometheus server from the [Prometheus Service](#configuration-in-gitlab)
+  configuration screen.
+- No successful deployments have occurred to this environment.
+- Prometheus does not have performance data for this environment, or the metrics
+  are not labeled correctly. To test this, connect to the Prometheus server and
+  [run a query](#gitlab-prometheus-queries), replacing `$CI_ENVIRONMENT_SLUG`
+  with the name of your environment.
+
+[autodeploy]: ../../../ci/autodeploy/index.md
+[kubernetes]: https://kubernetes.io
+[prometheus-k8s-sd]: https://prometheus.io/docs/operating/configuration/#<kubernetes_sd_config>
+[prometheus]: https://prometheus.io
+[gitlab-prometheus-k8s-monitor]: ../../../administration/monitoring/prometheus/index.md#configuring-prometheus-to-monitor-kubernetes
+[prometheus-docker-image]: https://hub.docker.com/r/prom/prometheus/
+[prometheus-yml]:samples/prometheus.yml
+[gitlab.com-ip-range]: https://gitlab.com/gitlab-com/infrastructure/issues/434
+[ci-environment-slug]: https://docs.gitlab.com/ce/ci/variables/#predefined-variables-environment-variables
+[ce-8935]: https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/8935
+[promgldocs]: ../../../administration/monitoring/prometheus/index.md

doc/user/project/integrations/samples/prometheus.yml

+apiVersion: v1
+kind: Namespace
+metadata:
+  name: prometheus
+---
+apiVersion: v1
+kind: ConfigMap
+metadata:
+  name: prometheus
+  namespace: prometheus
+data:
+  prometheus.yml: |-
+      scrape_configs:
+      - job_name: 'kubernetes-nodes'
+        scheme: https
+        tls_config:
+          ca_file: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
+          insecure_skip_verify: true
+        bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token
+        kubernetes_sd_configs:
+        - role: node
+        metric_relabel_configs:
+        - source_labels: [pod_name]
+          target_label: environment
+          regex: (.+)-.+-.+
+          replacement: $1
+---
+apiVersion: v1
+kind: Service
+metadata:
+  name: prometheus
+  namespace: prometheus
+spec:
+  selector:
+    app: prometheus
+  ports:
+  - name: prometheus
+    protocol: TCP
+    port: 9090
+    nodePort: 30090
+  type: NodePort
+---
+apiVersion: extensions/v1beta1
+kind: Deployment
+metadata:
+  name: prometheus
+  namespace: prometheus
+spec:
+  replicas: 1
+  template:
+    metadata:
+      labels:
+        app: prometheus
+    spec:
+      containers:
+      - name: prometheus
+        image: prom/prometheus:latest
+        args:
+          - '-config.file=/prometheus-data/prometheus.yml'
+        ports:
+        - name: prometheus
+          containerPort: 9090
+        volumeMounts:
+        - name: data-volume
+          mountPath: /prometheus-data
+      volumes:
+      - name: data-volume
+        configMap:
+          name: prometheus