I am trying to submit a spark job in cluster mode on AWS EMR. The spark application requires to parse an application.conf file for app configurations. I submitted the config file using spark-submit --files s3://bucket/application.conf ...The application configuration is being parsed as follows.
val confFile = SparkFiles.get("application.conf")val config = ConfigFactory.parseFile(new File(confFile))The SparkFiles.get("application.conf") is able to return the file path, but the ConfigFactory is not able to parse the file properly and throws the exception No configuration setting found for key..., when trying to fetch a specific configuration.Please let me know what is wrong here.