Quantcast
Channel: Active questions tagged config - Stack Overflow
Viewing all articles
Browse latest Browse all 5049

Is there a spark-defaults.conf when installed with pip install pyspark

$
0
0

I installed pyspark with pip.I code in jupyter notebooks. Everything works fine but not I got a java heap space error when exporting a large .csv file.Here someone suggested editing the spark-defaults.config. Also in the spark documentation, it says

"Note: In client mode, this config must not be set through the SparkConf directly in your application, because the driver JVM has already started at that point. Instead, please set this through the --driver-memory command line option or in your default properties file."

But I'm afraid there is no such file when installing pyspark with pip.I'm I right? How do I solve this?

Thanks!


Viewing all articles
Browse latest Browse all 5049

Trending Articles



<script src="https://jsc.adskeeper.com/r/s/rssing.com.1596347.js" async> </script>