Installation Steps For Spark
Downloading and Installation Steps for Spark:
Step 1 : Make sure whether Java is installed Correctly:
Open Terminal and type.
java -version
If Java is already installed correctly on your system, you will see the following output on your screen:
Java version "1.8.0_71"
Java(TM) SE Runtime Environment (build 1.8.0_71-b13)
Java HotSpot(TM) Client VM (build 25.0-b02, mixed mode).
If java is not installed on your system, then install it.
Ubuntu 14.04 :
- sudo add-apt-repository ppa:openjdk-r/ppa
- sudo apt-get update
- sudo apt-get install openjdk-8-jdk
- sudo apt-get install openjdk-8-jre
Ubuntu 16.04 :
- sudo apt-get install openjdk-8-jdk
- sudo apt-get install openjdk-8-jre
Step 2 : Make sure whether Scala is installed in your system:
Installing Scala language is very necesssary before installing Spark as it is important for implementation. Following command will verify the version of Scala in your system.
Scala language is used to implement Spark. So we need to verify the Scala installation by using the following command.
$ scala -version
$ scala -version
If Scala application is already installed on your system correctly then , you will get the following output on the screen as :
Scala code runner version 2.11.6 — Copyright 2002-2013, LAMP/EPFL
If you don’t have Scala, then install it.
Download the latest version of Scala from (https://www.scala-lang.org/download/).We are currently using scala-2.11.6 version. After downloading, you will be able to find the Scala tar file in the download folder.
Follow the given steps to install Scala
- Extract the Scala tar file using following command
tar xvf scala-2.11.6.tgz - Move Scala software files by using the following commands , to its respective directory (/usr/local/scala).
mv scala-2.11.6 /usr/local/scala - Dont forget to set path in Environment.
Set PATH for Scala using following command
$ export PATH = $PATH:/usr/local/scala/bin
or you can also add path in ~/.bashrc file by using the following command.
open in terminal and type sudo vim ~/.bashrc
and paste the path and don't forget to source it by running . .bashrc (or) source ~/.bashrc
$ scala -version
You have now successfully Installed scala if you see the scala version in your screen.
Step 3 : Downloading Apache Spark:
After finishing with the installation of Java and Scala successfully on your system then, Download the latest version of Spark by visiting following site (https://spark.apache.org/downloads.html)
After Downloading you can find a Spark tar file in the download folder.
Follow the below steps given below correctly for installing Apache Spark.
Extracting Spark tar file using following command
tar xvf spark-2.3.0-bin-hadoop2.7.tgz
Move Spark files by using the following commands , to its respective directory (/usr/local/spark)
mv spark-2.3.0-bin-hadoop2.7 /usr/local/spark
Configuration of Spark Environment.
Dont forget to set path in Environment.
Set PATH for Spark using following command
$ export PATH = $PATH:/usr/local/spark/bin
or you can also add path in ~/.bashrc file by using the following command.
open in terminal and type sudo vim ~/.bashrc
and paste the path and don't forget to source it by running . .bashrc (or) source ~/.bashrc
Step 4: Verify the Installation of Spark application on your system:
By Running the Following command will open Spark shell.
$ spark-shell
If spark is installed successfully then you will be getting the following output.
$ export PATH = $PATH:/usr/local/spark/bin
or you can also add path in ~/.bashrc file by using the following command.
open in terminal and type sudo vim ~/.bashrc
and paste the path and don't forget to source it by running . .bashrc (or) source ~/.bashrc
Step 4: Verify the Installation of Spark application on your system:
By Running the Following command will open Spark shell.
$ spark-shell
If spark is installed successfully then you will be getting the following output.
Note: If your going to connect Spark and Hive,Then Don't Forget To Copy the 'hive-site.xml' from hive conf directory to Spark conf Directory(/usr/local/spark/conf).
In Spark Add mysql jar file in PATH
----------------------------------------------------------
- /usr/local/spark/jars
This Command will show the connector jar path
This Command will show the connector jar path
ll /usr/share/java/mysql-connector-java.jar
Add the Mysql Jar File Path
cp /usr/share/java/mysql-connector-java.jar /usr/local/spark/jars
- cp spark-env.sh.template spark-env.sh
- vim spark-env.sh
# add the line
SPARK_CLASSPATH=/usr/share/java/mysql-connector-java.jar
Thanks,
Have Fun!!!!
Please Leave your Comment Below If you Have Any Clarifications Regarding Installations or Any Suggestions.
Thanks,
Have Fun!!!!
Please Leave your Comment Below If you Have Any Clarifications Regarding Installations or Any Suggestions.
Comments
Post a Comment