Caching

Caution
 FIXME

You can use CACHE TABLE [tableName] to cache tableName table in memory. It is an eager operation which is executed as soon as the statement is executed.

sql("CACHE TABLE [tableName]")

You could use LAZY keyword to make caching lazy.

Caching Dataset — cache Method

cache(): this.type

cache merely passes the calls to no-argument persist method.

Persisting Dataset — persist Method

persist(): this.type
persist(newLevel: StorageLevel): this.type

persist caches the Dataset using the default storage level MEMORY_AND_DISK or newLevel.

Internally, persist requests the CacheManager to cache the query (that is accessible through SharedState of the current SparkSession).

Unpersisting Dataset — unpersist Method

unpersist(blocking: Boolean): this.type

unpersist uncache the Dataset possibly by blocking the call.

Internally, unpersist requests the CacheManager to uncache the query.

results matching ""

    No results matching ""