SparkPlanner — Query Planner
SparkPlanner
is a concrete QueryPlanner (indirectly through extending SparkStrategies) that allows for extraStrategies
, i.e. a collection of additional SparkStrategy transformations (beside the ones defined in SparkPlanner
itself).
SparkPlanner
requires a SparkContext, a SQLConf, and a collection of Strategy
objects (as extraStrategies
) when created.
SparkPlanner
defines numPartitions
method that returns the value of spark.sql.shuffle.partitions for the number of partitions to use for joins and aggregations.
strategies
collection uses the constructor’s extraStrategies
and the predefined collection of Strategy
objects.
Among the SparkStrategy
objects are FileSourceStrategy and JoinSelection.