replay(
logData: InputStream,
sourceName: String,
maybeTruncated: Boolean = false): UnitReplayListenerBus
ReplayListenerBus is a custom SparkListenerBus that can replay JSON-encoded SparkListenerEvent events from a stream and post them to listeners.
|
Note
|
ReplayListenerBus is used in FsHistoryProvider.
|
|
Note
|
ReplayListenerBus is a private[spark] class in org.apache.spark.scheduler package.
|
Replaying JSON-encoded SparkListenerEvents from Stream (replay method)
replay reads JSON-encoded SparkListenerEvent events from logData (one event per line) and posts them to all registered SparkListenerInterface listeners.
replay uses JsonProtocol.sparkEventFromJson to convert JSON-encoded events to SparkListenerInterface listeners objects.
|
Note
|
replay uses jackson from json4s library to parse the AST for JSON.
|
When there is an exception parsing a JSON event, you may see the following WARN message in the logs (for the last line) or a JsonParseException.
WARN Got JsonParseException from log file $sourceName at line [lineNumber], the file might not have finished writing cleanly.Any other non-IO exceptions end up with the following ERROR messages in the logs:
ERROR Exception parsing Spark event log: [sourceName]
ERROR Malformed line #[lineNumber]: [currentLine]|
Note
|
The sourceName input argument is only used for messages.
|