SparkPlanner — Query Planner

SparkPlanner is a concrete QueryPlanner (indirectly through extending SparkStrategies) that allows for extraStrategies, i.e. a collection of additional SparkStrategy transformations (beside the ones defined in SparkPlanner itself).

SparkPlanner requires a SparkContext, a SQLConf, and a collection of Strategy objects (as extraStrategies) when created.

SparkPlanner defines numPartitions method that returns the value of spark.sql.shuffle.partitions for the number of partitions to use for joins and aggregations.

strategies collection uses the constructor’s extraStrategies and the predefined collection of Strategy objects.

Among the SparkStrategy objects are FileSourceStrategy and JoinSelection.

results matching ""

    No results matching ""