Added more info on how to setup monitoring (#600)

site/_data/sidebar.yaml

@@ -39,6 +39,8 @@ groups:
     endpoint: Kubernetes/#pulsar-on-google-container-engine
   - title: Pulsar on AWS
     endpoint: Kubernetes/#pulsar-on-amazon-web-services
+  - title: Monitoring
+    endpoint: Monitoring
 
 - title: Pulsar administration
   dir: admin
@@ -61,7 +63,7 @@ groups:
     endpoint: Stats
   - title: Modular load manager
     endpoint: ModularLoadManager
-    
+
 - title: Client libraries
   dir: clients
   docs:

site/docs/latest/deployment/InstanceSetup.md

@@ -171,40 +171,3 @@ To report the topic stats:
 ```shell
 $ bin/pulsar-admin persistent stats persistent://test/us-west/ns1/my-topic
 ```
-
-## Monitoring
-
-### Broker stats
-
-The [`pulsar-admin`](../../reference/CliTools#pulsar-admin) tool
-
-Pulsar {% popover broker %} metrics can be collected from brokers and exported in JSON format. There are two main types of metrics:
-
-* *Destination dumps*, which containing stats for each individual topic. They can be fetched using
-
-  ```shell
-  bin/pulsar-admin broker-stats destinations
-  ```
-
-* Broker metrics, containing broker info and topics stats aggregated at namespace
-  level:
-
-  ```shell
-  bin/pulsar-admin broker-stats monitoring-metrics
-  ```
-
-All the message rates are updated every 1min.
-
-### BookKeeper stats
-
-There are several stats frameworks that works with BookKeeper and that can be enabled by changing the `statsProviderClass` in `conf/bookkeeper.conf`.
-
-By following the instructions above, the `DataSketchesMetricsProvider` will be enabled. It features a very efficient way to compute latency quantiles, along with rates and counts.
-
-The stats are dumped every interval into a JSON file that is overwritten each time.
-
-```properties
-statsProviderClass=org.apache.bokkeeper.stats.datasketches.DataSketchesMetricsProvider
-dataSketchesMetricsJsonFileReporter=data/bookie-stats.json
-dataSketchesMetricsUpdateIntervalSeconds=60
-```

site/docs/latest/deployment/Monitoring.md

+---
+title: Monitoring
+---
+
+There are different ways to monitor a Pulsar cluster, exposing both metrics relative to the
+usage of topics and the overall health of the individual components of the cluster.
+
+## Collecting metrics
+
+### Broker stats
+
+The [`pulsar-admin`](../../reference/CliTools#pulsar-admin) tool
+
+Pulsar {% popover broker %} metrics can be collected from brokers and exported in JSON format. There are two main types of metrics:
+
+* *Destination dumps*, which containing stats for each individual topic. They can be fetched using
+
+  ```shell
+  bin/pulsar-admin broker-stats destinations
+  ```
+
+* Broker metrics, containing broker info and topics stats aggregated at namespace
+  level:
+
+  ```shell
+  bin/pulsar-admin broker-stats monitoring-metrics
+  ```
+
+All the message rates are updated every 1min.
+
+The aggregated broker metrics are also exposed in the [Prometheus](https://prometheus.io) format at:
+
+```shell
+http://$BROKER_ADDRESS:8080/metrics
+```
+
+### ZooKeeper stats
+
+The ZooKeeper server and clients that are shipped with Pulsar have been instrumented to expose
+detailed stats through Prometheus as well.
+
+```shell
+http://$ZK_SERVER:8080/metrics
+```
+
+### BookKeeper stats
+
+For BookKeeper you can configure the stats frameworks by changing the `statsProviderClass` in
+`conf/bookkeeper.conf`.
+
+By default, the default BookKeeper configuration included with Pulsar distribution will enable
+the Prometheus exporter.
+
+```shell
+http://$BOOKIE_ADDRESS:8000/metrics
+```
+
+For bookies, the default port is `8000` (instead of `8080`) and that can be configured by changing
+the `prometheusStatsHttpPort` in `conf/bookkeeper.conf`.
+
+## Configuring Prometheus
+
+You can configure Prometheus to collect and store the metrics data by following the Prometheus
+[Getting started](https://prometheus.io/docs/introduction/getting_started/) guide.
+
+When running on bare metal, you can provide the list of nodes that needs to be probed. When deploying
+in a Kubernetes cluster, the monitoring is automatically setup with the [provided](../../deployment/Kubernetes)
+instructions.
+
+## Dashboards
+
+When collecting time series statistics, the major problem is to make sure the number of dimensions
+attached to the data does not explode.
+
+For that reason we only collect time series of metrics aggregated at the namespace level.
+
+The
+
+### Pulsar per-topic dashboard
+
+The per-topic dashboard instructions are available at [Dashboard](../Dashboard).
+
+### Grafana
+
+You can use grafana to easily create dashboard driven by the data stored in Prometheus.
+
+There is a `pulsar-grafana` Docker image that is ready to use with the principal dashboards already
+in place. This is enabled by default when deploying Pulsar on Kubernetes.
+
+To use the dashboard manually:
+
+```shell
+docker run -p3000:3000 \
+        -e PROMETHEUS_URL=http://$PROMETHEUS_HOST:9090/ \
+        streamlio/pulsar-grafana:latest
+```

site/index.html

@@ -34,7 +34,7 @@ layout: default
           created at <a href="https://developer.yahoo.com/open-source/">Yahoo</a> and now part of the <a href="https://apache.org">Apache Software Foundation</a>.</p>
 
       <p class="button-group lead">
-        <a class="btn btn-pulsar btn-lg" href="/docs/latest" role="button">Read the docs</a>
+        <a class="btn btn-pulsar btn-lg" href="/docs/latest/getting-started/LocalCluster/" role="button">Read the docs</a>
         <a class="btn btn-github btn-lg" href="https://github.com/apache/incubator-pulsar" role="button">GitHub <i class="fa fa-github fa-lg"></i></a>
       </p>
     </div>