DEBUG WindowedDStream: Using partition aware union for windowing at [time]WindowedDStreams
WindowedDStream (aka windowed stream) is an internal DStream with dependency on the parent stream.
|
Note
|
It is the result of window operators. |
windowDuration has to be a multiple of the parent stream’s slide duration.
slideDuration has to be a multiple of the parent stream’s slide duration.
|
Note
|
When windowDuration or slideDuration are not multiples of the parent stream’s slide duration, Exception is thrown.
|
The parent’s RDDs are automatically changed to be persisted at StorageLevel.MEMORY_ONLY_SER level (since they need to last longer than the parent’s slide duration for this stream to generate its own RDDs).
Obviously, slide duration of the stream is given explicitly (and must be a multiple of the parent’s slide duration).
parentRememberDuration is extended to cover the parent’s rememberDuration and the window duration.
compute method always returns a RDD, either PartitionerAwareUnionRDD or UnionRDD, depending on the number of the partitioners defined by the RDDs in the window. It uses slice operator on the parent stream (using the slice window of [now - windowDuration + parent.slideDuration, now]).
If only one partitioner is used across the RDDs in window, PartitionerAwareUnionRDD is created and you should see the following DEBUG message in the logs:
Otherwise, when there are multiple different partitioners in use, UnionRDD is created and you should see the following DEBUG message in the logs:
DEBUG WindowedDStream: Using normal union for windowing at [time]|
Tip
|
Enable Add the following line to |