client private attribute is an instance of Client that YarnClientSchedulerBackend creates an instance of when it starts and uses to submit the Spark application.

client is also used to monitor the Spark application when YarnClientSchedulerBackend waits for the application.

client is stopped when YarnClientSchedulerBackend stops.

start is part of the SchedulerBackend Contract. It is executed when TaskSchedulerImpl starts.

It creates the internal client object and submits the Spark application to YARN ResourceManager. After the application is deployed to YARN and running, it starts the internal monitorThread state monitor thread. In the meantime it also calls the supertype’s start.

start sets spark.driver.appUIAddress as appUIAddress (of SparkUI) (and only if Spark’s web UI is enabled).

With DEBUG log level enabled you should see the following DEBUG message in the logs:

It then creates an instance of ClientArguments (using --arg [hostport] arguments).

It sets the parent’s totalExpectedExecutors to the initial number of executors.

It creates a Client object using the instance of ClientArguments and SparkConf.

The parent’s YarnSchedulerBackend.bindToYarn method is called with the current application id (being the result of calling Client.submitApplication) and None for the optional attemptId.

The parent’s YarnSchedulerBackend.start is called.

waitForApplication is executed that blocks until the application is running or an SparkException is thrown.

If spark.yarn.credentials.file is defined, YarnSparkHadoopUtil.get.startExecutorDelegationTokenRenewer(conf) is called.

A MonitorThread object is created (using asyncMonitorApplication) and started to asynchronously monitor the currently running application.

stop is part of the SchedulerBackend Contract.

It stops the internal helper objects, i.e. monitorThread and client as well as "announces" the stop to other services through Client.reportLauncherState. In the meantime it also calls the supertype’s stop.

stop makes sure that the internal client has already been created (i.e. it is not null), but not necessarily started.

stop stops the internal monitorThread using MonitorThread.stopMonitor method.

It then "announces" the stop using Client.reportLauncherState(SparkAppHandle.State.FINISHED).

Later, it passes the call on to the suppertype’s stop and, once the supertype’s stop has finished, it calls YarnSparkHadoopUtil.stopExecutorDelegationTokenRenewer followed by stopping the internal client.

Eventually, when all went fine, you should see the following INFO message in the logs:

waitForApplication is an internal (private) method that waits until the current application is running (using Client.monitorApplication).

If the application has FINISHED, FAILED, or has been KILLED, a SparkException is thrown with the following message:

You should see the following INFO message in the logs for RUNNING state:

asyncMonitorApplication internal method creates a separate daemon MonitorThread thread called "Yarn application state monitor".

MonitorThread internal class is to monitor a Spark application deployed to YARN in client mode.

When started, it calls the blocking Client.monitorApplication (with no application reports printed out to the console, i.e. logApplicationReport is disabled).

When the call to Client.monitorApplication has finished, it is assumed that the application has exited. You should see the following ERROR message in the logs:

That leads to stopping the current SparkContext (using SparkContext.stop).