UnifiedMemoryManager

Caution

UnifiedMemoryManager is the default MemoryManager since Spark 1.6 with onHeapStorageMemory being ??? and onHeapExecutionMemory being ???

Note	The other now legacy (before Spark 1.6) `MemoryManager` is `StaticMemoryManager`.

acquireStorageMemory method

Note	`acquireStorageMemory` is a part of the MemoryManager Contract.

acquireStorageMemory has two modes of operation per memoryMode, i.e. MemoryMode.ON_HEAP or MemoryMode.OFF_HEAP, for execution and storage pools, and the maximum amount of memory to use.

Caution

FIXME Where are they used?

In MemoryMode.ON_HEAP, onHeapExecutionMemoryPool, onHeapStorageMemoryPool, and maxOnHeapStorageMemory are used.

In MemoryMode.OFF_HEAP, offHeapExecutionMemoryPool, offHeapStorageMemoryPool, and maxOffHeapMemory are used.

Caution

FIXME What is the difference between them?

It makes sure that the requested number of bytes numBytes (for a block to store) fits the available memory. If it is not the case, you should see the following INFO message in the logs and the method returns false.

INFO Will not store [blockId] as the required space ([numBytes] bytes) exceeds our memory limit ([maxMemory] bytes)

If the requested number of bytes numBytes is greater than memoryFree in the storage pool, acquireStorageMemory will attempt to use the free memory from the execution pool.

Note	The storage pool can use the free memory from the execution pool.

It will take as much memory as required to fit numBytes from memoryFree in the execution pool (up to the whole free memory in the pool).

Ultimately, acquireStorageMemory requests the storage pool for numBytes for blockId.

acquireUnrollMemory method

Note	`acquireUnrollMemory` is a part of the MemoryManager Contract.

acquireUnrollMemory simply passes calls to acquireStorageMemory.

maxOnHeapStorageMemory method

Note	`maxOnHeapStorageMemory` is a part of the MemoryManager Contract.

Creating UnifiedMemoryManager Instance

private[spark] class UnifiedMemoryManager private[memory] (
  conf: SparkConf,
  val maxHeapMemory: Long,
  onHeapStorageRegionSize: Long,
  numCores: Int)

When an instance of UnifiedMemoryManager is created, it requires a SparkConf with the following numbers:

maxHeapMemory
onHeapStorageRegionSize
numCores

It makes sure that the sum of onHeapExecutionMemoryPool and onHeapStorageMemoryPool pool sizes is exactly the constructor’s maxHeapMemory.

It also makes sure that the sum of offHeapExecutionMemoryPool and offHeapStorageMemoryPool pool sizes is exactly maxOffHeapMemory.

Caution

FIXME Describe the pools

UnifiedMemoryManager