I'm trying to get the path to spark.worker.dir
for the current sparkcontext
.
If I explicitly set it as a config param
, I can read it back out of SparkConf
, but is there anyway to access the complete config
(including all defaults) using PySpark
?