log4j.logger.org.apache.spark.ExecutorAllocationManager=INFOExecutorAllocationManager — Allocation Manager for Spark Core
ExecutorAllocationManager is responsible for dynamically allocating and removing executors based on the workload.
It intercepts Spark events using the internal ExecutorAllocationListener that keeps track of the workload (changing the internal registries that the allocation manager uses for executors management).
It uses ExecutorAllocationClient, LiveListenerBus, and SparkConf (that are all passed in when ExecutorAllocationManager is created).
|
Tip
|
Enable Add the following line to Refer to Logging. |
addExecutors
|
Caution
|
FIXME |
removeExecutor
|
Caution
|
FIXME |
maxNumExecutorsNeeded method
|
Caution
|
FIXME |
Starting ExecutorAllocationManager (start method)
start(): Unitstart registers ExecutorAllocationListener (with LiveListenerBus) to monitor scheduler events and make decisions when to add and remove executors. It then immediately starts spark-dynamic-executor-allocation allocation executor that is responsible for the scheduling every 100 milliseconds.
|
Note
|
100 milliseconds for the period between successive scheduling is fixed, i.e. not configurable.
|
It requests executors using the input ExecutorAllocationClient. It requests spark.dynamicAllocation.initialExecutors.
|
Note
|
start is called while SparkContext is being created (with dynamic allocation enabled).
|
Scheduling Executors — schedule Method
schedule(): Unitschedule calls updateAndSyncNumExecutorsTarget to…FIXME
It then go over removeTimes to remove expired executors, i.e. executors for which expiration time has elapsed.
updateAndSyncNumExecutorsTarget
updateAndSyncNumExecutorsTarget(now: Long): IntupdateAndSyncNumExecutorsTarget…FIXME
If ExecutorAllocationManager is initializing it returns 0.
Resetting (reset method)
reset(): Unitreset resets ExecutorAllocationManager to its initial state, i.e.
-
initializing is enabled (i.e.
true). -
The currently-desired number of executors is set to the initial value.
-
The <<numExecutorsToAdd, ???> is set to
1. -
All executor pending to remove are cleared.
-
All ??? are cleared.
initialNumExecutors attribute
|
Caution
|
FIXME |
numExecutorsTarget attribute
|
Caution
|
FIXME |
numExecutorsToAdd attribute
numExecutorsToAdd attribute controls…FIXME
Stopping ExecutorAllocationManager — stop Method
stop(): Unitstop shuts down spark-dynamic-executor-allocation allocation executor.
|
Note
|
It waits 10 seconds for the complete termination. |
Internal Registries
executorsPendingToRemove registry
|
Caution
|
FIXME |
removeTimes registry
removeTimes keeps track of executors and their…FIXME
executorIds
|
Caution
|
FIXME |
spark-dynamic-executor-allocation Allocation Executor
spark-dynamic-executor-allocation allocation executor is a…FIXME
It is started…
It is stopped…