Metrics System

Spark uses Metrics - a Java library to measure the behaviour of the components.

org.apache.spark.metrics.source.Source is the top-level class for the metric registries in Spark. Sources expose their internal status.

Spark uses Metrics 3.1.0.

Note
 Metrics are only available for cluster modes, i.e. local mode turns metrics off.

Metrics System is available at http://localhost:4040/metrics/json/ (for the default setup of a Spark application).

$ http http://localhost:4040/metrics/json/
HTTP/1.1 200 OK
Cache-Control: no-cache, no-store, must-revalidate
Content-Length: 2359
Content-Type: text/json;charset=utf-8
Date: Sat, 29 Oct 2016 19:46:34 GMT
Server: Jetty(9.2.z-SNAPSHOT)
X-Frame-Options: SAMEORIGIN

{
    "counters": {
        "app-20161029211022-0000.driver.HiveExternalCatalog.fileCacheHits": {
            "count": 0
        },
        "app-20161029211022-0000.driver.HiveExternalCatalog.filesDiscovered": {
            "count": 0
        },
        "app-20161029211022-0000.driver.HiveExternalCatalog.partitionsFetched": {
            "count": 0
        }
    },
    ...
}
Note
 Separate MetricsSystems are created for driver, executor, external shuffleService, master, applications, worker, mesos_cluster. You can access the metrics using corresponding URLs for the services, e.g. 4040 for the driver, 8080 for Spark Standalone’s master and applications.
Note
 You have to use the trailing slash (/) to have the output.

Enable org.apache.spark.metrics.sink.JmxSink in conf/metrics.properties and use jconsole to access Spark metrics through JMX.

*.sink.jmx.class=org.apache.spark.metrics.sink.JmxSink
spark metrics jconsole.png
Figure 1. jconsole and JmxSink in spark-shell

Creating MetricsSystem Instance

When created, MetricsSystem creates a MetricsConfig and initializes it immediately.

createMetricsSystem Method

createMetricsSystem(
  instance: String,
  conf: SparkConf,
  securityMgr: SecurityManager): MetricsSystem

createMetricsSystem creates a MetricsSystem.

Note
 createMetricsSystem is executed when a SparkEnv is created for the driver and executors.

start Method

Caution
 FIXME

registerSource Method

Caution
 FIXME

report Method

Caution
 FIXME

Master

$ http http://192.168.1.4:8080/metrics/master/json/path
HTTP/1.1 200 OK
Cache-Control: no-cache, no-store, must-revalidate
Content-Length: 207
Content-Type: text/json;charset=UTF-8
Server: Jetty(8.y.z-SNAPSHOT)
X-Frame-Options: SAMEORIGIN

{
    "counters": {},
    "gauges": {
        "master.aliveWorkers": {
            "value": 0
        },
        "master.apps": {
            "value": 0
        },
        "master.waitingApps": {
            "value": 0
        },
        "master.workers": {
            "value": 0
        }
    },
    "histograms": {},
    "meters": {},
    "timers": {},
    "version": "3.0.0"
}

results matching ""

    No results matching ""