Installation Steps For Spark

By -

Downloading  and Installation Steps for Spark:


Step 1 : Make sure whether  Java is installed Correctly:
             
Open Terminal and type.                    
             
              java -version

If Java is already installed correctly on your system, you will  see the following output on your screen:
             
             Java version "1.8.0_71"
             Java(TM) SE Runtime Environment (build 1.8.0_71-b13)
             Java HotSpot(TM) Client VM (build 25.0-b02, mixed mode).

If java is not installed on your system, then install it.

 

        Ubuntu 14.04 :

- sudo add-apt-repository ppa:openjdk-r/ppa

- sudo apt-get update

- sudo apt-get install openjdk-8-jdk

- sudo apt-get install openjdk-8-jre



Ubuntu 16.04 :

- sudo apt-get install openjdk-8-jdk
- sudo apt-get install openjdk-8-jre

Step 2 : Make sure whether  Scala is installed in your system:


Installing Scala language is very necesssary before installing Spark as it is important for implementation. Following command will verify the version of Scala  in your system.

Scala language is used to implement Spark. So we need to verify the Scala installation by using the following command.
            $  scala -version

If Scala application is already installed on your system correctly then , you will  get  the following output on the screen as :

Scala code runner version 2.11.6 — Copyright 2002-2013, LAMP/EPFL

If you don’t have Scala, then install it.

Download the latest version of Scala from (https://www.scala-lang.org/download/).We are currently using scala-2.11.6 version. After downloading, you will be able to find the Scala tar file in the download folder.

Follow the given steps to install Scala 

  • Extract the Scala tar file using following command
           tar xvf scala-2.11.6.tgz
  • Move Scala software files by using the following commands , to its respective directory (/usr/local/scala).
           mv scala-2.11.6 /usr/local/scala
  • Dont forget to set path in Environment.
    Set PATH for Scala using following command
           $ export PATH = $PATH:/usr/local/scala/bin
    or you can also add path in  ~/.bashrc file by using the following command.
    open  in terminal and type sudo vim ~/.bashrc
    and paste the path and don't forget to source it by running . .bashrc (or) source ~/.bashrc
It is very important to verify the installation of Scala once again.
     $ scala -version
You have now successfully Installed scala if you see the scala version in your screen.

Step 3 : Downloading Apache Spark:
After finishing with the installation of Java and Scala successfully on your system then, Download the latest version of Spark by visiting following site (https://spark.apache.org/downloads.html)

After Downloading you can find a Spark tar file in the download folder.
Follow the below steps given below  correctly for installing  Apache Spark.

Extracting Spark tar file using following command
       tar xvf spark-2.3.0-bin-hadoop2.7.tgz

Move Spark  files by using the following commands , to its respective directory (/usr/local/spark)
       mv spark-2.3.0-bin-hadoop2.7 /usr/local/spark
Configuration of Spark Environment.
   
Dont forget to set path in Environment.
   Set PATH for Spark using following command
       $ export PATH = $PATH:/usr/local/spark/bin
   or you can also add path in  ~/.bashrc file by using the following command.
   open  in terminal and type sudo vim ~/.bashrc
   and paste the path and don't forget to source it by running . .bashrc (or) source ~/.bashrc
Step 4: Verify the Installation of Spark application on your  system:
By Running the Following command will open Spark shell.

         $ spark-shell 
If spark is installed successfully then you will be getting the following output.


Note: If your going to connect Spark and Hive,Then Don't Forget To Copy the 'hive-site.xml' from hive conf directory to Spark conf Directory(/usr/local/spark/conf).

In Spark Add mysql jar file in PATH
 ---------------------------------------------------------- 
 - /usr/local/spark/jars

This Command will show the connector jar path
ll /usr/share/java/mysql-connector-java.jar

Add the Mysql Jar File Path

cp /usr/share/java/mysql-connector-java.jar  /usr/local/spark/jars
 - cp  spark-env.sh.template  spark-env.sh
 - vim spark-env.sh
# add the line
SPARK_CLASSPATH=/usr/share/java/mysql-connector-java.jar



Thanks,
Have Fun!!!!

Please Leave your Comment Below If you Have Any Clarifications Regarding Installations or Any Suggestions.

Comments

Post a Comment

Popular posts from this blog

Quick Setup Steps For Installing & Integrating Spark,Hive,Hadoop Processess Together !!!

By -
Image
If you wish For Indepth Understanding go to my Previous Blog and Install All  Separately. You can Install all the Packages Separately or Install together using the following Commands. SSH :                                                                            #For Interacting With All   - ssh-keygen -t rsa -P ""  - cat $HOME/.ssh/id_rsa.pub >> $HOME/.ssh/authorized_keys  - # ssh-add -l # Not Working the run the command  - ssh localhost Install and Configuration for Spark, Hive and Hadoop: Make a separate Directory for Installing All the Packages you can use 'mkdir' command to create a directory   - mkdir /home/user/Dilli/Installed/All_Setup  - cd /home/user/Dilli/Installed/All_Setup  - java -version  - readlink -f /usr/bin/java | sed "s:bin/java::"  - wget https://archive.apache.org/dist/hadoop/core/hadoop-2.9.0/hadoop-2.9.0.tar.gz  - tar -xzvf hadoop-2.9.0.tar.gz  - mv hadoop-2.9.0 hadoop  - wget http://www-eu
Read more

Twitter Analysis Using R

By -
Image
PREREQUISITES: You have already installed  R  and are using  RStudio . In order to extract tweets, you will need a Twitter application and hence a Twitter account. If you don’t have a Twitter account, please  sign up . Use your Twitter login ID and password to sign in at  Twitter Developers . 1.  STEPS TO CREATE A TWITTER APPLICATION a.  Navigate to My Applications in the upper right hand corner. b.  Navigate to My Applications in the upper right hand corner. c.  Create a new application. d.  Fill out the new app form. Names should be unique, i.e., no one else should have used this name for their Twitter app. Give   a brief description of the app. You can change this later on if needed. Enter   your website or blog address.   Callback URL can be left blank. Once you’ve done this, make sure you’ve read the “Developer Rules Of The Road” blurb, check the “Yes, I agree” box, fill in the CAPTCHA and click the “Create Your Twitter Application” button. e.  
Read more

Best Practices for Apache Hive Performance Tuning

By -
If you’re wondering how to scale Apache Hive, here are the best ways to make the most of Hive performance. 1. Partitioning Tables :                                      Hive partitioning is an effective method to improve the query performance on larger tables. 2 . Compress map/reduce output : By enabling compression at various Levels we can achieve the performance improvement in Hive Queries.                                                     Inside Hive-site.xml Add these Properties               <property>                   <name> mapred.compress.map.output </name>                   < value> true </value>               </property>               <property>                   <name> mapred.output.compress </name>                   <value> true </value>             </property> 3 . Map join  :             We can improve the performance of joins by enabling Auto Convert Ma
Read more

Performance Improvement in Hive (Properties in HIVE -SITE.XML)

By -
There are list of Properties Available to increase the Performance in hive. Use the Properties Appropriate to your Need. <?xml version="1.0" encoding="UTF-8" standalone="no"?> <?xml-stylesheet type="text/xsl" href="configuration.xsl"?> <configuration> <property> <name>javax.jdo.option.ConnectionURL</name> <value>jdbc:mysql://localhost:3306/metastore_db_3?createDatabaseIfNotExist=true&amp;useSSL=false</value> <description>JDBC connect string for a JDBC metastore</description> </property> <property> <name>javax.jdo.option.ConnectionDriverName</name> <value>com.mysql.jdbc.Driver</value> <description>Driver class name for a JDBC metastore</description> </property> <property> <name>javax.jdo.option.ConnectionUserName</name> <value>hiveuser</value>
Read more

Installation Steps For MySql Server (version 5.7.23) For Centos

By -
Image
MySQL is an open source free relational database management system (RDBMS) released under GNU (General Public License). It is used to run multiple databases on any single server by providing multi-user access to each created database. Below Steps are provided for  installing MySQL 5.7.23 version on RHEL/CentOS 7/6/5 and Fedora 23/22/21 using MySQL Yum repository via YUM utility. Step 1: Adding the MySQL Yum Repository  We will use official MySQL Yum software repository, which will provides RPM packages for installing the latest version of MySQL  server, client, MySQL Utilities, MySQL Workbench, Connector/ODBC, and Connector/Python for the RHEL/CentOS 7/6/5 and Fedora 23-21. Now download and add the following MySQL Yum repository to your respective Linux distribution system’s repository list to install the latest version of MySQL.             ---------------------------------- On RHEL/CentOS 7 ----------------------------------------------------------------          
Read more

Installation Steps for Hive

By -
Hive is a SQL Database which uses various servers like thrift, Derby, MySQL to maintain the metastore. Here in this setup MySQL server is configured as a base to maintain the metastore of Hive. SSH : -------  - ssh-keygen -t rsa -P ""  - cat $HOME/.ssh/id_rsa.pub >> $HOME/.ssh/authorized_keys  - # ssh-add -l # Not Working the run the command  - ssh localhost   Needed Software’s - Java ,Hadoop,MySql Server Before Installing Hive you make sure all the dependent Software's are correctly Installed .If it is not Installed then you can Refer my Previous Blog to Install them. After Successfully Installing the Softwares correctly!!!! Download Hive from below Repository: https://archive.apache.org/dist/hive The following steps are required for installing Hive on your PC. Let us assume the Hive archive is downloaded onto the /Downloads directory . Here, we download Hive archive named “ ap ache-hive-2.3.0-bin.tar.gz ". The fol
Read more

Facebook Data Analysis Using R

By -
Image
Hey Folks, Today We gonna Learn to Analyse Facebook Data using R . Before Going to concept, makesure you Have R Installed in your System, If Not Proceed the Following Steps to Install. Step 1 – Install R  1. Download the R installer from: https://cran.r-project.org/ 2. Run the installer. Default settings are fine. If you do not have admin rights on your laptop, then ask your local IT support. In that case, it is important that you also ask them to give you full permissions to the R directories. Without this, you will not be able to install additional packages later. Step 2 – Install RStudio 1. Download RStudio: https://www.rstudio.com/products/rstudio/download/ 2. Once the installation of R has completed successfully (and not before), run the RStudio installer. 3. If you do not have administrative rights on your laptop, step 2 may fail. Ask your IT Support or download a pre-built zip archive of RStudio which doesn’t need installing.    a. Download the appropriate a
Read more