BlockTransferService

BlockTransferService is a contract for specialized ShuffleClient objects that can fetch and upload blocks in synchronously and asynchronously.

Note	`BlockTransferService` is a `private[spark] abstract class`.

Note	NettyBlockTransferService is the only available implementation of BlockTransferService Contract.

BlockTransferService Contract

Every BlockTransferService offers the following:

init that accepts BlockDataManager for storing or fetching blocks. It is assumed that the method is called before the service is considered fully operational.
```
init(blockDataManager: BlockDataManager): Unit
```
port the service listens to.
```
port: Int
```
hostName the service listens to.
```
hostName: String
```

uploadBlock to upload a block (of ManagedBuffer identified by BlockId) to a remote hostname and port.

uploadBlock(
  hostname: String,
  port: Int,
  execId: String,
  blockId: BlockId,
  blockData: ManagedBuffer,
  level: StorageLevel,
  classTag: ClassTag[_]): Future[Unit]

Synchronous (and hence blocking) fetchBlockSync to fetch one block blockId (that corresponds to the ShuffleClient parent’s asynchronous fetchBlocks).
```
fetchBlockSync(
  host: String,
  port: Int,
  execId: String,
  blockId: String): ManagedBuffer
```
fetchBlockSync is a mere wrapper around fetchBlocks to fetch one blockId block that waits until the fetch finishes.
Synchronous (and hence blocking) uploadBlockSync to upload a block (of ManagedBuffer identified by BlockId) to a remote hostname and port.
```
uploadBlockSync(
  hostname: String,
  port: Int,
  execId: String,
  blockId: BlockId,
  blockData: ManagedBuffer,
  level: StorageLevel,
  classTag: ClassTag[_]): Unit
```
uploadBlockSync is a mere wrapper around uploadBlock that waits until the upload finishes.

BlockTransferService

BlockTransferService

BlockTransferService Contract

NettyBlockTransferService - Netty-Based BlockTransferService

results matching ""

No results matching ""