spark.yarn.submit.file.replication: The default HDFS replication (usually 3) HDFS replication level for the files uploaded into HDFS for the application. These include things like the Spark jar, the app jar, and any distributed cache files/archives. spark.yarn.stagingDir: Current user's home directory in the filesystemSpark environment provides a command to execute the application file, be it in Scala or Java(need a Jar format), Python and R programming file. The command is, $ spark-submit --master <url> <SCRIPTNAME>.py. I'm running spark in windows 64bit architecture system with JDK 1.8 version. P.S find a screenshot of my terminal window. Code snippetTo do so, specify the spark properties spark.kubernetes.driver.podTemplateFile and spark.kubernetes.executor.podTemplateFile to point to local files accessible to the spark-submit process. To allow the driver pod access the executor pod template file, the file will be automatically mounted onto a volume in the driver pod when it’s created. rdd = sc.textFile ("file:///path/to/file") If your file isn’t already on all nodes in the cluster, you can load it locally on the driver without going through Spark and then call parallelize to distribute the contents to workers. Take care to put file:// in front and the use of "/" or "\" according to OS. Share.command options. You specify spark-submit options using the form --option value instead of --option=value . (Use a space instead of an equals sign.) Option. Description. class. For Java and Scala applications, the fully qualified classname of the class containing the main method of the application. For example, org.apache.spark.examples.SparkPi. spark.yarn.submit.file.replication: The default HDFS replication (usually 3) HDFS replication level for the files uploaded into HDFS for the application. These include things like the Spark jar, the app jar, and any distributed cache files/archives. spark.yarn.stagingDir: Current user's home directory in the filesystemSep 7, 2016 · I want to load a property config file when submit a spark job, so I can load the proper config due to different environment, such as a test environment or a product environment. But I don't know where to put the properties file, here is the code loading the properties file: This mode is preferred for Production Run of a Spark Applications or Jobs. Client mode - In client mode, the driver run will run in the local machine (your laptop\desktop terminal). This mode is used for Testing , Debugging or To Test Issue Fixes of a Spark Application or job. However although the the driver runs locally but all the executors ... rdd = sc.textFile ("file:///path/to/file") If your file isn’t already on all nodes in the cluster, you can load it locally on the driver without going through Spark and then call parallelize to distribute the contents to workers. Take care to put file:// in front and the use of "/" or "\" according to OS. Share.The most basic steps to configure the key stores and the trust store for a Spark Standalone deployment mode is as follows: Generate a key pair for each node. Export the public key of the key pair to a file on each node. Import all exported public keys into a single trust store.In case if you wanted to run a PySpark application using spark-submit from a shell, use the below example. Specify the .py file you wanted to run and you can also specify the .py, .egg, .zip file to spark submit command using --py-files option for any dependencies. ./bin/spark-submit \ --master yarn \ --deploy-mode cluster \ wordByExample.py.This will let you create an .egg file which is similar to java jar file. You can then specify the path of this egg file using --py-files. spark-submit --py-files path_to_egg_file path_to_spark_driver_file. Create zip files (example- abc.zip) containing all your dependencies.The second precedence goes to spark-submit options. Finally, properties specified in spark-defaults.conf file. When you are setting jars in different places, remember the precedence it takes. Use spark-submit with --verbose option to get more details about what jars Spark has used. 2.1 Add jars to the classpath using –jar OptionIn case if you wanted to run a PySpark application using spark-submit from a shell, use the below example. Specify the .py file you wanted to run and you can also specify the .py, .egg, .zip file to spark submit command using --py-files option for any dependencies. ./bin/spark-submit \ --master yarn \ --deploy-mode cluster \ wordByExample.py. For Python, you can use the --py-files argument of spark-submit to add .py, .zip or .egg files to be distributed with your application. If you depend on multiple Python files we recommend packaging them into a .zip or .egg. Launching Applications with spark-submit. Once a user application is bundled, it can be launched using the bin/spark ... We are using Spark 2.3.0 on Yarn in pseudo distributed mode. We need to query a postgres table from spark whose configurations are defined in a properties file. I passed the property file using --files attribute of spark submit. To read the file in my code I simply used java.util.Properties.PropertiesReader class.Apr 19, 2023 · Python manager for spark-submit jobs. Spark-submit. TL;DR: Python manager for spark-submit jobs Description. This package allows for submission and management of Spark jobs in Python scripts via Apache Spark's spark-submit functionality. So, I am trying to redirect the output of an apache spark-submit command to text file but some output fails to populate file. Here is the command I am using: spark-submit something.py > results.txt I can see the output in the terminal but I do not see it in the file. What am I forgetting or doing wrong here? Edit: If I useOct 16, 2017 · Spark-submit can't locate local file. Ask Question Asked 5 years, 10 months ago. Modified 5 years, 10 months ago. Viewed 8k times 2 I've written a very simple python ... Jul 2, 2020 · I have a pyspark code stored both on the master node of an AWS EMR cluster and in an s3 bucket that fetches over 140M rows from a MySQL database and stores the sum of a column back in the log files on s3. When I spark-submit the pyspark code on the master node, the job gets completed successfully and the output is stored in the log files on the ... Usage: spark-submit --status [submission ID] --master [spark://...] Usage: spark-submit run-example [options] example-class [example args] As you can see in the first Usage spark-submit requires <app jar | python file>. The app jar argument is a Spark application's jar with the main object (SimpleApp in your case). You can build the app jar ...2. In my Spark job I read some additional data from resources files. Some example Resources.getResource ("/more-data") It works great locally, and when I run from spark-submit master=local [*] I only to need to add --conf=spark.driver.extraClassPath=moredata. Moving to cluster mode (Yarn) it is no longer able to find the folder.You can pass the arguments from the spark-submit command and then access them in your code in the following way, sys.argv[1] will get you the first argument, sys.argv[2] the second argument and so on.But when I copy the same to my properties file: spark.class MyClass spark.master spark://my_master spark.files test.config spark.jars build/jars/MyProject.jar, build/jars/Config.jar On trying to use this file with spark-submit, I get an error: java.lang.IllegalArgumentException: Missing application resourceSo, I am trying to redirect the output of an apache spark-submit command to text file but some output fails to populate file. Here is the command I am using: spark-submit something.py > results.txt I can see the output in the terminal but I do not see it in the file. What am I forgetting or doing wrong here? Edit: If I useFor a comprehensive list of all configurations that can be passed with spark-submit, just run spark-submit --help. In this link provided by @suj1th, they say that: configuration values explicitly set on a SparkConf take the highest precedence, then flags passed to spark-submit, then values in the defaults file.1 You could put the file on a network mount that is accessible by all the nodes on the cluster. This way you can just read from that mount in your driver program. You could expose a simple endpoint that returns the file. This way your driver program can make an http call. – Alex Naspo Jan 20, 2016 at 20:39 True enough, @AlexNaspo, but redundant. Once application is built, spark-submit command is called to submit the application to run in a Spark environment. Use --jars option. To add JARs to a Spark job, --jars option can be used to include JARs on Spark driver and executor classpaths. If multiple JAR files need to be included, use comma to separate them. The following is an example:Submit Spark workload by submitting Spark batch applications by using the cluster management console, RESTful APIs, or the CLI. A Spark batch application is launched by only the spark-submit command from the following ways: cluster management console (immediately or by scheduling the submission). ascd Spark application RESTful APIs. Yeah I added another parameter. It was Spark-submit --py-files wheelfile driver.py This driver was calling the function inside wheelfile. But then this driver and wheel are in same location essentially. What is the use of wheel then? Because if I run the command with spark-submit driver.py . Then also its the same Right?? –spark.yarn.submit.file.replication: The default HDFS replication (usually 3) HDFS replication level for the files uploaded into HDFS for the application. These include things like the Spark jar, the app jar, and any distributed cache files/archives. spark.yarn.stagingDir: Current user's home directory in the filesystemMar 26, 2017 · The easiest way to set some config: spark.conf.set ("spark.sql.shuffle.partitions", 500). Where spark refers to a SparkSession, that way you can set configs at runtime. It's really useful when you want to change configs again and again to tune some spark parameters for specific queries. Share. For the 5th process I am using a spark-submit command as this process needs to leverage spark because of the size of the data being processed. I am running into issues with JDBC and Kerberos Authnetication with the spark-submit command. The Oracle @Configuration is the same for all of these processes. It works fine and authenticates fine with a ...When you wanted to spark-submit a PySpark application (Spark with Python), you need to specify the .py file you wanted to run and specify the .egg file or .zip file for dependency libraries. Below are some of the options & configurations specific to run pyton (.py) file with spark submit. besides these, you can also use most of the options ... The Spark shell and spark-submit tool support two ways to load configurations dynamically. The first is command line options, such as --master, as shown above. spark-submit can accept any Spark property using the --conf/-c flag, but uses special flags for properties that play a part in launching the Spark application.Apr 15, 2020 · The spark-submit job will setup and configure Spark as per our instructions, execute the program we pass to it, then cleanly release the resources that were being used. A simply Python program passed to spark-submit might look like this: """ spark_submit_example.py An example of the kind of script we might want to run. The modules and functions ... Dec 8, 2017 · This is a JSON protocol to submit Spark application, to submit Spark application to cluster manager, we should use HTTP POST request to send above JSON protocol to Livy Server: curl -H "Content-Type: application/json" -X POST -d ‘<JSON Protocol>’ <livy-host>:<port>/batches. As you can see most of the arguments are the same, but there still ... Oct 23, 2020 · Yeah I added another parameter. It was Spark-submit --py-files wheelfile driver.py This driver was calling the function inside wheelfile. But then this driver and wheel are in same location essentially. What is the use of wheel then? Because if I run the command with spark-submit driver.py . Then also its the same Right?? – This mode is preferred for Production Run of a Spark Applications or Jobs. Client mode - In client mode, the driver run will run in the local machine (your laptop\desktop terminal). This mode is used for Testing , Debugging or To Test Issue Fixes of a Spark Application or job. However although the the driver runs locally but all the executors ... When you wanted to spark-submit a PySpark application (Spark with Python), you need to specify the .py file you wanted to run and specify the .egg file or .zip file for dependency libraries. Below are some of the options & configurations specific to run pyton (.py) file with spark submit. besides these, you can also use most of the options ... For a comprehensive list of all configurations that can be passed with spark-submit, just run spark-submit --help. In this link provided by @suj1th, they say that: configuration values explicitly set on a SparkConf take the highest precedence, then flags passed to spark-submit, then values in the defaults file. 2. In my case I am using Spark (2.1.1) and for the processing I need to connect to Kafka (using kerberos, therefore a keytab). When submitting the job I can pass the keytab with --keytab and --principal options. The main drawback is that the keytab will no be send to the distributed cache (or at least be available to the executors) so it will fail.spark.yarn.submit.file.replication: The default HDFS replication (usually 3) HDFS replication level for the files uploaded into HDFS for the application. These include things like the Spark jar, the app jar, and any distributed cache files/archives. 0.8.1: spark.yarn.stagingDir: Current user's home directory in the filesystem For a comprehensive list of all configurations that can be passed with spark-submit, just run spark-submit --help. In this link provided by @suj1th, they say that: configuration values explicitly set on a SparkConf take the highest precedence, then flags passed to spark-submit, then values in the defaults file.6,344 24 92 174 Add a comment 2 Answers Sorted by: 8 You can use --properties-file which should include parameters with starting keyword spark like spark.driver.memory 5g spark.executor.memory 10g And command should look like:The most basic steps to configure the key stores and the trust store for a Spark Standalone deployment mode is as follows: Generate a key pair for each node. Export the public key of the key pair to a file on each node. Import all exported public keys into a single trust store. Setting the spark-submit flags is one of the ways to dynamically supply configurations to the SparkContext object that is instantiated in the driver. spark-submit can also read configuration values set in the conf/spark-defaults.conf file which you can set using EMR configuration options when creating your cluster and, although not recommended, ...Jun 4, 2017 · Usage: spark-submit --status [submission ID] --master [spark://...] Usage: spark-submit run-example [options] example-class [example args] As you can see in the first Usage spark-submit requires <app jar | python file>. The app jar argument is a Spark application's jar with the main object (SimpleApp in your case). You can build the app jar ... java.io.FileNotFoundException for a file sent in Spark-submit --files. 1. How to pass arguments to spark-submit using docker. 0. Running Scala Jar with Spark-Submit. 4.3 Answers. No, spark-submit --files option doesn't support sending folder, but you can put all your files in a zip, use that file in --files list. You can use SparkFiles.get (filename) in your spark job to load the file, explode it and use exploded files. 'filename' doesn't need to be absolute path, just filename does it.Oct 1, 2020 · I have four python files , out of four files 1 file has spark entry code defined and that file drives and calls rest other python files . for now I have provided four python files with --py-files option in spark submit command , but instead of submitting this way I want to create zip file and pack these all four python files and submit with ... The second precedence goes to spark-submit options. Finally, properties specified in spark-defaults.conf file. When you are setting jars in different places, remember the precedence it takes. Use spark-submit with --verbose option to get more details about what jars Spark has used. 2.1 Add jars to the classpath using –jar OptionThis will let you create an .egg file which is similar to java jar file. You can then specify the path of this egg file using --py-files. spark-submit --py-files path_to_egg_file path_to_spark_driver_file. Create zip files (example- abc.zip) containing all your dependencies.spark.yarn.submit.file.replication: The default HDFS replication (usually 3) HDFS replication level for the files uploaded into HDFS for the application. These include things like the Spark jar, the app jar, and any distributed cache files/archives. 0.8.1: spark.yarn.stagingDir: Current user's home directory in the filesystem In case if you wanted to run a PySpark application using spark-submit from a shell, use the below example. Specify the .py file you wanted to run and you can also specify the .py, .egg, .zip file to spark submit command using --py-files option for any dependencies. ./bin/spark-submit \ --master yarn \ --deploy-mode cluster \ wordByExample.py.For deploy-mode cluster. As previous answers mentioned, if you want to pass env variable to spark master, you want to use: --conf spark.yarn.appMasterEnv.FOO=bar // pass bar value to FOO variable --conf spark.yarn.appMasterEnv.FOO=$ {FOO} // passing current FOO env variable --conf spark.yarn.appMasterEnv.FOO2=bar2 // multiple variables are ... As suspected, the two options ( sc.addFile and --files) are not equivalent, and this is (admittedly very subtly) hinted at the documentation (emphasis added): addFile (path, recursive=False) Add a file to be downloaded with this Spark job on every node. --files FILES. Comma-separated list of files to be placed in the working directory of each ...For Python, you can use the --py-files argument of spark-submit to add .py, .zip or .egg files to be distributed with your application. If you depend on multiple Python files we recommend packaging them into a .zip or .egg. Launching Applications with spark-submit. Once a user application is bundled, it can be launched using the bin/spark ... The Spark shell and spark-submit tool support two ways to load configurations dynamically. The first is command line options, such as --master, as shown above. spark-submit can accept any Spark property using the --conf/-c flag, but uses special flags for properties that play a part in launching the Spark application.It turned out that since I'm submitting my application in client mode, then the machine I run the spark-submit command from will run the driver program and will need to access the module files. I added my module to the PYTHONPATH environment variable on the node I'm submitting my job from by adding the following line to my .bashrc file (or ...Actually When using spark-submit, the application jar along with any jars included with the --jars option will be automatically transferred to the cluster. Your extra jars could be added to --jars, they will be copied to cluster automatically. please refer to "Advanced Dependency Management" section in below link:2. When using spark-submit with --master yarn-cluster, the application JAR file along with any JAR file included with the --jars option will be automatically transferred to the cluster. URLs supplied after --jars must be separated by commas. That list is included in the driver and executor classpaths.This is a JSON protocol to submit Spark application, to submit Spark application to cluster manager, we should use HTTP POST request to send above JSON protocol to Livy Server: curl -H "Content-Type: application/json" -X POST -d ‘<JSON Protocol>’ <livy-host>:<port>/batches. As you can see most of the arguments are the same, but there still ...For Python, you can use the --py-files argument of spark-submit to add .py, .zip or .egg files to be distributed with your application. If you depend on multiple Python files we recommend packaging them into a .zip or .egg. Launching Applications with spark-submit. Once a user application is bundled, it can be launched using the bin/spark ...For Python, you can use the --py-files argument of spark-submit to add .py, .zip or .egg files to be distributed with your application. If you depend on multiple Python files we recommend packaging them into a .zip or .egg. Launching Applications with spark-submit. Once a user application is bundled, it can be launched using the bin/spark ... Oct 1, 2020 · I have four python files , out of four files 1 file has spark entry code defined and that file drives and calls rest other python files . for now I have provided four python files with --py-files option in spark submit command , but instead of submitting this way I want to create zip file and pack these all four python files and submit with ... Nov 9, 2017 · As suspected, the two options ( sc.addFile and --files) are not equivalent, and this is (admittedly very subtly) hinted at the documentation (emphasis added): addFile (path, recursive=False) Add a file to be downloaded with this Spark job on every node. --files FILES. Comma-separated list of files to be placed in the working directory of each ... for me, run spark on yarn,just add --files log4j.properties makes everything ok. 1. make sure the directory where you run spark-submit contains file "log4j.properties". 2. run spark-submit ... --files log4j.properties. let's see why this work. 1.spark-submit will upload log4j.properties to hdfs like this Submit Spark workload by submitting Spark batch applications by using the cluster management console, RESTful APIs, or the CLI. A Spark batch application is launched by only the spark-submit command from the following ways: cluster management console (immediately or by scheduling the submission). ascd Spark application RESTful APIs. The spark-submit command is a utility to run or submit a Spark or PySpark application program (or job) to the cluster by specifying options and configurations, the application you are submitting can be written in Scala, Java, or Python (PySpark). spark-submit command supports the following. Apr 21, 2017 · It turned out that since I'm submitting my application in client mode, then the machine I run the spark-submit command from will run the driver program and will need to access the module files. I added my module to the PYTHONPATH environment variable on the node I'm submitting my job from by adding the following line to my .bashrc file (or ... The spark-submit job will setup and configure Spark as per our instructions, execute the program we pass to it, then cleanly release the resources that were being used. A simply Python program passed to spark-submit might look like this: """ spark_submit_example.py An example of the kind of script we might want to run. The modules and functions ...Teams. Q&A for work. Connect and share knowledge within a single location that is structured and easy to search. Learn more about TeamsMar 16, 2017 · spark-submit --class Eventhub --master yarn --deploy-mode cluster --executor-memory 1024m --executor-cores 4 --files app.conf spark-hdfs-assembly-1.0.jar --conf "app.conf" I was looking a way to put all these flags in file to pass to spark-submit to make my spark-submit command simple liek this Pass system property to spark-submit and read file from classpath or custom path. 2 adding external property file to classpath in spark. 0 ...But configuration file is imported in some other python file that is not entry point for spark application . I want to write spark submit command in pyspark , but I am not sure how to provide multiple files along configuration file with spark submit command when configuration file is not python file but text file or ini file.When you wanted to spark-submit a PySpark application (Spark with Python), you need to specify the .py file you wanted to run and specify the .egg file or .zip file for dependency libraries. Below are some of the options & configurations specific to run pyton (.py) file with spark submit. besides these, you can also use most of the options ... May 12, 2020 · Spark on Kubernetes doesn't support submitting locally stored files with spark-submit. 2. When using spark-submit with --master yarn-cluster, the application JAR file along with any JAR file included with the --jars option will be automatically transferred to the cluster. URLs supplied after --jars must be separated by commas. That list is included in the driver and executor classpaths.One straightforward method is to use script options such as --py-files or the spark.submit.pyFiles configuration, but this functionality cannot cover many cases, such as installing wheel files or when the Python libraries are dependent on C and C++ libraries such as pyarrow and NumPy. This blog post introduces how to control Python dependencies ...The easiest way to set some config: spark.conf.set ("spark.sql.shuffle.partitions", 500). Where spark refers to a SparkSession, that way you can set configs at runtime. It's really useful when you want to change configs again and again to tune some spark parameters for specific queries. Share.Mar 1, 2019 · I have a Java-spark code that reads certain properties files. These properties are being passed with spark-submit like: spark-submit --master yarn \\ --deploy-mode cluster \\ --files /home/aiman/
As suspected, the two options ( sc.addFile and --files) are not equivalent, and this is (admittedly very subtly) hinted at the documentation (emphasis added): addFile (path, recursive=False) Add a file to be downloaded with this Spark job on every node. --files FILES. Comma-separated list of files to be placed in the working directory of each .... Theme wzgdmt
May 5, 2016 · I have a CSV file "test.csv" that I'm trying to have copied to all nodes on the cluster. I have a 4 node apache-spark 1.5.2 standalone cluster. There are 4 workers where one node also acts has mas... On Kubernetes I'm having an issue: files uploaded via --files can't be read by Spark Driver. On Yarn, as described in many answers I can read those files using Source.fromFile(filename) . But I can't read files in Spark on Kubernetes.Mar 23, 2017 · I am currently running spark 2.1.0. I have worked most of the time in PYSPARK shell, but I need to spark-submit a python file(similar to spark-submit jar in java) . Using PySpark Native Features ¶. PySpark allows to upload Python files ( .py ), zipped Python packages ( .zip ), and Egg files ( .egg ) to the executors by one of the following: Setting the configuration setting spark.submit.pyFiles. Setting --py-files option in Spark scripts. Directly calling pyspark.SparkContext.addPyFile () in applications.Referencing here and here, I expect that I should be able to change the name by which a file is referenced in Spark by using an octothorpe - that is, if I call spark-submit --files local-file-name.json#spark-file-name.json, I should then be able to reference the file as spark-file-name.json. However, this doesn't appear to be the case:2. In my Spark job I read some additional data from resources files. Some example Resources.getResource ("/more-data") It works great locally, and when I run from spark-submit master=local [*] I only to need to add --conf=spark.driver.extraClassPath=moredata. Moving to cluster mode (Yarn) it is no longer able to find the folder.Spark environment provides a command to execute the application file, be it in Scala or Java(need a Jar format), Python and R programming file. The command is, $ spark-submit --master <url> <SCRIPTNAME>.py. I'm running spark in windows 64bit architecture system with JDK 1.8 version. P.S find a screenshot of my terminal window. Code snippetUsing PySpark Native Features ¶. PySpark allows to upload Python files ( .py ), zipped Python packages ( .zip ), and Egg files ( .egg ) to the executors by one of the following: Setting the configuration setting spark.submit.pyFiles. Setting --py-files option in Spark scripts. Directly calling pyspark.SparkContext.addPyFile () in applications.This mode is preferred for Production Run of a Spark Applications or Jobs. Client mode - In client mode, the driver run will run in the local machine (your laptop\desktop terminal). This mode is used for Testing , Debugging or To Test Issue Fixes of a Spark Application or job. However although the the driver runs locally but all the executors ...Dec 12, 2021 · These config files will give information to Spark about the EMR cluster like which is the master node, resource manager, and hive metastore to connect to on running spark-submit. Store the config ... Now that I am trying to run this on a cluster, I think I need to use spark-submit, to which I thought would be: spark-submit --conf spark.neo4j.bolt.password=Stuffffit --packages neo4j-contrib:neo4j-spark-connector:2.0.0-M2,graphframes:graphframes:0.2.0-spark2.0-s_2.11 -i neo4jsparkCluster.scala but it does not like the .scala file, somehow ...Usage: spark-submit --status [submission ID] --master [spark://...] Usage: spark-submit run-example [options] example-class [example args] As you can see in the first Usage spark-submit requires <app jar | python file>. The app jar argument is a Spark application's jar with the main object (SimpleApp in your case). You can build the app jar ...I have an AWS CLI cluster creation command that I am trying to modify so that it enables my driver and executor to work with a customized log4j.properties file. With Spark stand-alone clusters I have successfully used the approach of using the --files <log4j.file> switch together with setting -Dlog4j.configuration=<log4j.file> specified via ...command options. You specify spark-submit options using the form --option value instead of --option=value . (Use a space instead of an equals sign.) Option. Description. class. For Java and Scala applications, the fully qualified classname of the class containing the main method of the application. For example, org.apache.spark.examples.SparkPi.1. I am using spark 2.4.1 version and java8. I am trying to load external property file while submitting my spark job using spark-submit. As I am using below TypeSafe to load my property file. <groupId>com.typesafe</groupId> <artifactId>config</artifactId> <version>1.3.1</version>. In my code I am using.6,344 24 92 174 Add a comment 2 Answers Sorted by: 8 You can use --properties-file which should include parameters with starting keyword spark like spark.driver.memory 5g spark.executor.memory 10g And command should look like:I am trying to submit a spark job using 'gcloud dataproc jobs submit spark'. To connect to ES cluster I need to pass the truststore path. The job is successful if I copy the truststore file to all the worker nodes and give the absolute path as below:Now that I am trying to run this on a cluster, I think I need to use spark-submit, to which I thought would be: spark-submit --conf spark.neo4j.bolt.password=Stuffffit --packages neo4j-contrib:neo4j-spark-connector:2.0.0-M2,graphframes:graphframes:0.2.0-spark2.0-s_2.11 -i neo4jsparkCluster.scala but it does not like the .scala file, somehow ....