Builder — Building SparkSession with Fluent API

Builder — Building SparkSession with Fluent API

Builder is the fluent API to build a fully-configured SparkSession.

Table 1. Builder Methods
Method	Description
enableHiveSupport	Enables Hive support.

import org.apache.spark.sql.SparkSession
val spark: SparkSession = SparkSession.builder
  .master("local[*]")
  .appName("My Spark Application")
  .getOrCreate

You can use the fluent design pattern to set the various properties of a SparkSession that opens a session to Spark SQL.

Note	You can have multiple `SparkSession`s in a single Spark application.

`config` method

Caution

FIXME

Enabling Hive Support — `enableHiveSupport` Method

When creating a SparkSession, you can enable Hive support using enableHiveSupport method.

enableHiveSupport(): Builder

enableHiveSupport enables Hive support.

Internally, enableHiveSupport makes sure that the Hive classes are on CLASSPATH, i.e. Spark SQL’s org.apache.spark.sql.hive.HiveSessionState and org.apache.hadoop.hive.conf.HiveConf, and sets spark.sql.catalogImplementation property to hive.

Builder — Building SparkSession with Fluent API