The Spark Service
The Spark service is available in two versions: Spark and Spark (Standalone). The previously available Spark service, which runs Spark in standalone mode, has been renamed Spark (Standalone). The Spark (Standalone) service has its own runtime roles: Master and Worker. The current Spark service runs Spark as a YARN application. Both services have a History Server role. In secure clusters, Spark applications can only run on YARN. Cloudera recommends that you use the Spark service.
You can install Spark through the Cloudera Manager installation wizard using parcels and have the Spark service added and started as part of the first run installation wizard. See Installing Spark.
If you elect not to add the Spark service using the installation wizard, you can use the Add Service wizard to create the service. The wizard automatically configures dependent services and the Spark service. See Adding a Service for instructions.
Testing the Spark Service
val file = sc.textFile("hdfs://namenode:8020/path/to/input") val counts = file.flatMap(line => line.split(" ")).map(word => (word, 1)).reduceByKey(_ + _) counts.saveAsTextFile("hdfs://namenode:8020/output")
To submit Spark applications to YARN, use the --master yarn flag when you start spark-shell. To see information about the running Spark shell application, go to Spark History Server UI at http://spark_history_server:18088 or the YARN applications page in the Cloudera Manager Admin Console.
If you are running the Spark (Standalone) service, you can see the Spark shell application, and its executors, and logs in the Spark Master UI, by default at http://spark_master:18080.
For more information on running Spark applications, see Running Spark Applications.
Adding the Spark History Server Role
- Go to the Spark service.
- Click the Instances tab.
- Click the Add Role Instances button.
- Select a host in the column under History Server, then click OK.
- Click Continue.
- Check the checkbox next to the History Server role.
- Select and click Start.
- Click Close when the action completes.