Revision history [back]

To use multiple output sinks with Spark structured streaming in Scala, you can follow these steps:

1. Define your input data source, for example, a socket source:

val inputStream = spark.readStream
  .format("socket")
  .option("host", "localhost")
  .option("port", "9999")
  .load()

1. Apply transformations to your input stream as needed:

val transformedStream = ...

1. Define your output sinks, for example, a console sink and a file sink:

val consoleSink = transformedStream.writeStream
  .format("console")
  .outputMode("append")
  .start()

val fileSink = transformedStream.writeStream
  .format("parquet")
  .option("path", "/path/to/parquet/files")
  .option("checkpointLocation", "/path/to/checkpoint")
  .outputMode("append")
  .start()

1. Start your output sinks:

consoleSink.awaitTermination()
fileSink.awaitTermination()

Note that you can define as many output sinks as you need by repeating steps 3 and 4 with different output formats and options. Also, make sure to set different checkpoint locations for each sink to avoid conflicts.