There are a few things that could cause an error when using xcom_pull(param) with DatabricksSubmitRunOperator:
The parameter 'param' may not exist in the output of the Databricks job specified in the DatabricksSubmitRunOperator. Double-check that the parameter exists and has been correctly named.
The Databricks job may not be outputting the correct information for the parameter specified. Double-check that the information being outputted matches the format expected by xcom_pull(param).
The Databricks job may not be executing correctly, preventing any output from being generated. Check the logs of the Databricks job to see if there are any errors or issues that need to be resolved before xcom_pull(param) can work properly.
There may be a configuration issue with the Airflow environment or the Databricks cluster that is preventing xcom_pull(param) from working correctly. Check the configuration settings and make sure that Airflow and Databricks are properly integrated and set up to work with each other.
Asked: 2022-12-15 11:00:00 +0000
Seen: 7 times
Last updated: Jul 19 '21