StreamSourceProvider — Streaming Source Provider

StreamSourceProvider trait defines a contract for objects that can create a streaming Source for a format (e.g. text file) or system (e.g. Apache Kafka) by their names.

Note
 StreamSourceProvider is used when DataSource is requested for the name and schema of a Source or creates one.
Table 1. Streaming Source Providers
Name Description

TextSocketSourceProvider

Creates TextSocketSources (for socket format).

KafkaSourceProvider

Creates KafkaSourceProvider (for kafka format).

StreamSourceProvider Contract

StreamSourceProvider contract defines two methods:

  1. sourceSchema that returns the name and schema of the Source.

  2. createSource that produces a streaming Source (to continually read data).

sourceSchema Method

sourceSchema(
  sqlContext: SQLContext,
  schema: Option[StructType],
  providerName: String,
  parameters: Map[String, String]): (String, StructType)

createSource Method

createSource(
  sqlContext: SQLContext,
  metadataPath: String,
  schema: Option[StructType],
  providerName: String,
  parameters: Map[String, String]): Source

results matching ""

    No results matching ""