ExecutorAllocationClient

ExecutorAllocationClient is a contract for clients to communicate with a cluster manager to request or kill executors.

Getting Executor Ids (getExecutorIds method)

getExecutorIds(): Seq[String]

getExecutorIds is a private[spark] method to calculate the identifiers of the executors in use.

Note	It is used when `SparkContext` calculates the executors in use and also when Spark Streaming manages executors.

Requesting Exact Number of Executors (requestTotalExecutors method)

requestTotalExecutors(
  numExecutors: Int,
  localityAwareTasks: Int,
  hostToLocalTaskCount: Map[String, Int]): Boolean

requestTotalExecutors is a private[spark] method to update the cluster manager with the exact number of executors desired. It returns whether the request has been acknowledged by the cluster manager (true) or not (false).

Note	It is used when: `SparkContext` requests executors (for coarse-grained scheduler backends only). `ExecutorAllocationManager` starts, does updateAndSyncNumExecutorsTarget, and addExecutors. Streaming `ExecutorAllocationManager` requests executors. `YarnSchedulerBackend` stops.

Requesting Additional Executors (requestExecutors method)

requestExecutors(numAdditionalExecutors: Int): Boolean

requestExecutors requests additional executors from a cluster manager and returns whether the request has been acknowledged by the cluster manager (true) or not (false).

Note	It is used when `SparkContext` requests additional executors (for coarse-grained scheduler backends only).

Requesting to Kill Single Executor (killExecutor method)

killExecutor(executorId: String): Boolean

killExecutor requests that a cluster manager to kill a single executor that is no longer in use and returns whether the request has been acknowledged by the cluster manager (true) or not (false).

Note	The default implementation simply calls killExecutors (with a single-element collection of executors to kill).

Note	It is used in: `ExecutorAllocationManager` to remove executor. `SparkContext` to request to kill executors. Streaming `ExecutorAllocationManager` to request to kill executors.

Requesting to Kill Executors (killExecutors method)

killExecutors(executorIds: Seq[String]): Boolean

killExecutors requests that a cluster manager to kill one or many executors that are no longer in use and returns whether the request has been acknowledged by the cluster manager (true) or not (false).

Note	Interestingly, it is only used for killExecutor.

ExecutorAllocationClient