I installed pyspark
with pip
.I code in jupyter
notebooks. Everything works fine but not I got a java
heap space error when exporting a large .csv
file.Here someone suggested editing the spark-defaults.config
. Also in the spark documentation, it says
"Note: In client mode, this config must not be set through the SparkConf directly in your application, because the driver JVM has already started at that point. Instead, please set this through the --driver-memory command line option or in your default properties file."
But I'm afraid there is no such file when installing pyspark
with pip
.I'm I right? How do I solve this?
Thanks!