JobListener and JobWaiter

Spark subscribes for job completion or failure events (after submitting a job to DAGScheduler) using JobListener trait.

The following are the job listeners used:

JobWaiter waits until DAGScheduler completes a job and passes the results of tasks to a resultHandler function.
ApproximateActionListener…FIXME

An instance of JobListener is used in the following places:

In ActiveJob as a listener to notify if tasks in this job finish or the job fails.
In DAGScheduler.handleJobSubmitted
In DAGScheduler.handleMapStageSubmitted
In JobSubmitted
In MapStageSubmitted

`JobListener` Contract

JobListener is a private[spark] contract with the following two methods:

private[spark] trait JobListener {
  def taskSucceeded(index: Int, result: Any)
  def jobFailed(exception: Exception)
}

A JobListener object is notified each time a task succeeds (by taskSucceeded) and when the whole job fails (by jobFailed).

JobWaiter

JobWaiter[T](
  dagScheduler: DAGScheduler,
  val jobId: Int,
  totalTasks: Int,
  resultHandler: (Int, T) => Unit)
extends JobListener

JobWaiter is a custom JobListener.

It is used when DAGScheduler submits a job or submits a map stage. You can use a JobWaiter to block until the job finishes executing or to cancel it.

While the methods execute, JobSubmitted and MapStageSubmitted events are posted that reference the JobWaiter.

As a JobListener, JobWaiter gets notified about task completions or failures, using taskSucceeded and jobFailed, respectively. When the total number of tasks (that equals the number of partitions to compute) equals the number of taskSucceeded, the JobWaiter instance is marked successful. A jobFailed event marks the JobWaiter instance failed.

JobListener and JobWaiter