Expression TreeNode

An Expression in Spark SQL 2.0’s Catalyst represents a query expression that can be SQL or non-SQL. It is described by the Expression contract.

Table 1. Catalyst’s Top-Level Expressions
Name	Scala Kind	Behaviour	Example
Unevaluable	trait	Cannot be evaluated, i.e. `eval` and `doGenCode` throw an `UnsupportedOperationException`	`CurrentDatabase`
Nondeterministic	trait
NonSQLExpression	trait
ExpectsInputTypes	trait
CodegenFallback	trait
NamedExpression	trait
UnaryExpression	abstract class
BinaryExpression	abstract class
TernaryExpression	abstract class
LeafExpression	abstract class

Expression Contract

The contract of an expression in Spark SQL is described by the Expression abstract class.

abstract class Expression extends TreeNode[Expression]

A Expression is a TreeNode that obeys the following contract:

It may or may not be foldable.
It may or may not be deterministic.
It may or may not be nullable.
It uses references.
It can be evaluated to a JVM object given a InternalRow.
It can genCode to produce a ExprCode.
It can doGenCode to produce a ExprCode.
It may or may not be resolved.
It is of dataType data type.
It may or may not have childrenResolved.
It has a canonicalized representation.
It may or may not be semanticEquals given another Expression.
It has a semanticHash.
It can be checkInputDataTypes.
It has a prettyName.
It has a sql representation.

`Unevaluable` Expression

Unevaluable expressions cannot be evaluated, i.e. eval and doGenCode methods throw an UnsupportedOperationException.

Note	`Unevaluable` is a Scala trait.

`Nondeterministic` Expression

Nondeterministic expressions are non-deterministic and non-foldable, i.e. deterministic and foldable properties are disabled (i.e. false). They require explicit initialization before evaluation.

Nondeterministic expressions have two additional methods:

initInternal for internal initialization (called before eval)
evalInternal to evaluate a InternalRow into a JVM object.

Note	`Nondeterministic` is a Scala trait.

Nondeterministic expressions have the additional initialized flag that is enabled (i.e. true) after the other additional initInternal method has been called.

Examples of Nondeterministic expressions are InputFileName, MonotonicallyIncreasingID, SparkPartitionID functions and the abstract RDG (that is the base for Rand and Randn functions).

Note	`Nondeterministic` expressions are the target of `PullOutNondeterministic` logical plan rule.

`NonSQLExpression` Expression

NonSQLExpression expressions are expressions that have no SQL representation.

Internally, NonSQLExpression makes for all Attributes in a tree to be PrettyAttributes.

Note	`NonSQLExpression` is a Scala trait with the `sql` method being `final` (i.e. non-overridable).

`ExpectsInputTypes` Expression

ExpectsInputTypes expressions are…TK

Note	`ExpectsInputTypes` is a Scala trait.

`CodegenFallback` Expression

CodegenFallback expressions are…TK

Note	`CodegenFallback` is a Scala trait.

`UnaryExpression` Expression

UnaryExpression expressions are…TK

Note	`UnaryExpression` is an abstract class.

`BinaryExpression` Expression

BinaryExpression expressions are…TK

Note	`BinaryExpression` is an abstract class.

`TernaryExpression` Expression

TernaryExpression expressions are…TK

Note	`TernaryExpression` is an abstract class.

`LeafExpression` Expression

LeafExpression expressions are Catalyst expressions with no children, i.e. children method returns always an empty collection.

Note	`LeafExpression` is an abstract class.

`NamedExpression` Expression

NamedExpression expressions are Catalyst expressions that can later be referenced in the dataflow graph.

Note	`NamedExpression` is a Scala trait.

Expression TreeNode