Executor Backend

Executor Backends

ExecutorBackend is a pluggable interface used by executors to send status updates about the different states of a task to a scheduler.

Figure 1. ExecutorBackends work on executors and communicate with driver

Caution

FIXME What is "a scheduler" in this context?

The interface comes with one method:

def statusUpdate(taskId: Long, state: TaskState, data: ByteBuffer)

It is effectively a bridge between the driver and an executor, i.e. there are two endpoints running.

Caution

FIXME What is cluster scheduler? Where is ExecutorBackend used?

Status updates include information about tasks, i.e. id, state, and data (as ByteBuffer).

At startup, an executor backend connects to the driver and creates an executor. It then launches and kills tasks. It stops when the driver orders so.

There are the following types of executor backends:

Caution

Executor Backends