Here is one approach to transforming a dictionary presented as a string into a structured DataFrame in Scala:
import spray.json._
val jsonString = "{ \"name\":\"John\", \"age\":30, \"city\":\"New York\" }"
val jsonMap = jsonString.parseJson.convertTo[Map[String, Any]]
import org.apache.spark.rdd.RDD
import org.apache.spark.sql.SparkSession
val spark = SparkSession.builder.appName("DictionaryStringToDataFrame").getOrCreate()
val rdd = spark.sparkContext.parallelize(Seq(jsonMap))
import org.apache.spark.sql.DataFrame
val df = spark.createDataFrame(rdd)
df.printSchema()
df.show()
The output should look something like this:
root
|-- age: integer (nullable = false)
|-- city: string (nullable = true)
|-- name: string (nullable = true)
+---+--------+----+
|age| city|name|
+---+--------+----+
| 30|New York|John|
+---+--------+----+
Please start posting anonymously - your entry will be published after you log in or create a new account. This space is reserved only for answers. If you would like to engage in a discussion, please instead post a comment under the question or an answer that you would like to discuss
Asked: 2021-07-01 11:00:00 +0000
Seen: 11 times
Last updated: Sep 18 '21
What does "waiting for handler commit" mean in relation to the slow writes experienced in MySQL 8?
How is the Node.js Digital Ocean MySQL database related?
What is the process of integrating a Python task queue in a Flask application?
What is the difference between indexing in Elasticsearch and MongoDB?