log4j.logger.org.apache.spark.deploy.yarn.YarnRMClient=INFOYarnRMClient
YarnRMClient is responsible for registering and unregistering a Spark application (in the form of ApplicationMaster) with YARN ResourceManager (and hence RM in the name). It is a mere wrapper for AMRMClient[ContainerRequest] that is started when registering ApplicationMaster (and never stopped explicitly!).
Besides being responsible for registration and unregistration, it also knows the application attempt identifiers and tracks the maximum number of attempts to register ApplicationMaster.
|
Tip
|
Enable Add the following line to Refer to Logging. |
Registering ApplicationMaster with YARN ResourceManager (register method)
To register ApplicationMaster (for a Spark application) with the YARN ResourceManager, Spark uses register.
register(
driverUrl: String,
driverRef: RpcEndpointRef,
conf: YarnConfiguration,
sparkConf: SparkConf,
uiAddress: String,
uiHistoryAddress: String,
securityMgr: SecurityManager,
localResources: Map[String, LocalResource]): YarnAllocatorregister instantiates YARN’s AMRMClient, initializes it (using conf input parameter) and starts immediately. It saves uiHistoryAddress input parameter internally for later use.
Figure 1. Registering ApplicationMaster with YARN ResourceManager
You should see the following INFO message in the logs (in stderr in YARN):
INFO YarnRMClient: Registering the ApplicationMasterIt then registers the application master on the local host with port 0 and uiAddress input parameter for the URL at which the master info can be seen.
The internal registered flag is enabled.
Ultimately, it creates a new YarnAllocator with the input parameters of register passed in and the just-created YARN AMRMClient.
Unregistering ApplicationMaster from YARN ResourceManager (unregister method)
unregister(status: FinalApplicationStatus, diagnostics: String = ""): Unitunregister unregisters the ApplicationMaster of a Spark application.
It basically checks that ApplicationMaster is registered and only when it is requests the internal AMRMClient to unregister.
unregister is called when ApplicationMaster wants to unregister.
Maximum Number of Attempts to Register ApplicationMaster (getMaxRegAttempts method)
getMaxRegAttempts(sparkConf: SparkConf, yarnConf: YarnConfiguration): IntgetMaxRegAttempts uses SparkConf and YARN’s YarnConfiguration to read configuration settings and return the maximum number of application attempts before ApplicationMaster registration with YARN is considered unsuccessful (and so the Spark application).
It reads YARN’s yarn.resourcemanager.am.max-attempts (available as YarnConfiguration.RM_AM_MAX_ATTEMPTS) or falls back to YarnConfiguration.DEFAULT_RM_AM_MAX_ATTEMPTS (which is 2).
The return value is the minimum of the configuration settings of YARN and Spark.
Getting ApplicationAttemptId of Spark Application (getAttemptId method)
getAttemptId(): ApplicationAttemptIdgetAttemptId returns YARN’s ApplicationAttemptId (of the Spark application to which the container was assigned).
Internally, it uses YARN-specific methods like ConverterUtils.toContainerId and ContainerId.getApplicationAttemptId.
getAmIpFilterParams
|
Caution
|
FIXME |