Ask Your Question
1

How can I add several columns in Spark while using the same WHEN condition?

asked 2023-05-31 19:46:56 +0000

pufferfish gravatar image

edit retag flag offensive close merge delete

1 Answer

Sort by ยป oldest newest most voted
0

answered 2023-05-31 20:16:01 +0000

djk gravatar image

You can use the selectExpr method in Spark to add multiple columns with the same WHEN condition. Here's an example:

val df = Seq((1, "A"), (2, "B"), (3, "C")).toDF("id", "letter")

val newDf = df.selectExpr("*",
  "CASE WHEN id < 2 THEN 'Low' ELSE 'High' END as id_category",
  "CASE WHEN id < 2 THEN 'Small' ELSE 'Big' END as id_size")

newDf.show()

In this example, we're adding two columns (id_category and id_size) with the same WHEN condition (id < 2). The selectExpr method allows us to pass in multiple expressions separated by commas, which are evaluated and added as new columns to the DataFrame.

edit flag offensive delete link more

Your Answer

Please start posting anonymously - your entry will be published after you log in or create a new account. This space is reserved only for answers. If you would like to engage in a discussion, please instead post a comment under the question or an answer that you would like to discuss

Add Answer


Question Tools

Stats

Asked: 2023-05-31 19:46:56 +0000

Seen: 13 times

Last updated: May 31 '23