Quick Setup Steps For Installing & Integrating Spark,Hive,Hadoop Processess Together !!!

By -

If you wish For Indepth Understanding go to my Previous Blog and Install All  Separately.

You can Install all the Packages Separately or Install together using the following Commands.

SSH :                                                                            #For Interacting With All

  - ssh-keygen -t rsa -P ""
 - cat $HOME/.ssh/id_rsa.pub >> $HOME/.ssh/authorized_keys
 - # ssh-add -l # Not Working the run the command
 - ssh localhost


Install and Configuration for Spark, Hive and Hadoop:

Make a separate Directory for Installing All the Packages

you can use 'mkdir' command to create a directory 

 - mkdir /home/user/Dilli/Installed/All_Setup
 - cd /home/user/Dilli/Installed/All_Setup
 - java -version
 - readlink -f /usr/bin/java | sed "s:bin/java::"

 - wget https://archive.apache.org/dist/hadoop/core/hadoop-2.9.0/hadoop-2.9.0.tar.gz
 - tar -xzvf hadoop-2.9.0.tar.gz
 - mv hadoop-2.9.0 hadoop

 - wget http://www-eu.apache.org/dist/hive/hive-3.0.0/apache-hive-3.0.0-bin.tar.gz
 - tar -xzvf apache-hive-3.0.0-bin.tar.gz
 - mv apache-hive-3.0.0-bin hive

 - cd
 - wget https://archive.apache.org/dist/spark/spark-2.3.0/spark-2.3.0-bin-hadoop2.7.tgz
 - tar -xzvf spark-2.3.0-bin-hadoop2.7.tgz
 - mv spark-2.3.0-bin-hadoop2.7 spark

#if you dint Install Scite
Then Install it using the following Command.
 - sudo apt-get install scite

- scite .bashrc
        # Add the Following to the file
# Set JAVA_HOME (we will also configure JAVA_HOME directly for Hadoop later on)

export JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64

# Set Hadoop-related environment variables
export HADOOP_HOME=/home/user/Dilli/Installed/All_Setup/hadoop
export PATH=$PATH:$HADOOP_HOME/bin

export HADOOP_CONF=/home/user/Dilli/Installed/All_Setup/hadoop/conf
export PATH=$PATH:$HADOOP_CONF/bin

export HADOOP_PREFIX=/home/user/Dilli/Installed/All_Setup/hadoop
export PATH=$PATH:$HADOOP_PREFIX/bin

# Set Spark-related environment variables
export SPARK_CONF=/home/user/Dilli/Installed/All_Setup/spark/conf
export PATH=$PATH:$SPARK_CONF/bin

export SPARK_HOME=/home/user/Dilli/Installed/All_Setup/spark
export PATH=$PATH:$SPARK_HOME/bin

# Set HBase-related environment variables
export HIVE_HOME=/home/user/Dilli/Installed/All_Setup/hive
export PATH=$PATH:$HIVE_HOME/bin


 -  . .bashrc                                        # Don't Forget to do this step

 - sudo apt-get install mysql-server                 #For Connecting With Mysql

 - sudo apt-get install libmysql-java


Hadoop Configurations:

- cd /home/user/Dilli/Installed/All_Setup/hadoop/etc/hadoop

 - vim core-site.xml
# Add the following,
<configuration>
<property>
<name>fs.defaultFS</name>
<value>hdfs://localhost:8020</value>
</property>
</configuration>


 - vim hdfs-site.xml
# Add the following,
<configuration>
<property>
<name>dfs.replication</name>
<value>1</value>
</property>

<property>
<name>dfs.blocksize</name>
<value>268435456</value>
</property>
<property>
<name>dfs.namenode.datanode.registration.ip-hostname-check</name>
<value>false</value>
</property>
<property>
<name>dfs.datanode.data.dir</name>
<value>/home/user/Dilli/Installed/All_Setup/process_data/data</value>
</property>
<property>
<name>dfs.namenode.name.dir</name>
<value>/home/user/Dilli/Installed/All_Setup/process_data/name</value>
</property>
<property>
<name>dfs.namenode.secondary.http-address</name>
<value>localhost:50090</value>
</property>
</configuration>


 - vim hadoop-env.sh
# Change the 'JAVA_HOME' path
export JAVA_HOME="/usr/lib/jvm/java-8-openjdk-amd64"


Hive Configurations:


- cd /home/Dilli/Installed_Softwares/All_Setup/hive/conf

# To Improve Performance you can Add Some More Properties to Speedup the Process
   to speed up visit my post "Performance Improvement in Hive (Properties in HIVE -SITE.XML)"
   and add necessary properties that are needed for your Process.While Mentioning the Database Metastore specify the correct name.

 - vim hive-site.xml
# create a 'hive-site.xml' and add the following,

<?xml version="1.0" encoding="UTF-8" standalone="no"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
<configuration>
<property>
  <name>javax.jdo.option.ConnectionURL</name>
  <value>jdbc:mysql://localhost:3306/metastore_db_3?createDatabaseIfNotExist=true&amp;useSSL=false</value>
  <description>JDBC connect string for a JDBC metastore</description>
</property>

<property>
  <name>javax.jdo.option.ConnectionDriverName</name>
  <value>com.mysql.jdbc.Driver</value>
  <description>Driver class name for a JDBC metastore</description>
</property>

<property>
  <name>javax.jdo.option.ConnectionUserName</name>
  <value>hiveuser</value>
  <description>username to use against metastore database</description>
</property>

<property>
  <name>javax.jdo.option.ConnectionPassword</name>
  <value>hivepassword</value>
  <description>password to use against metastore database</description>
</property>

  <property>
    <name>hive.metastore.schema.verification</name>
    <value>false</value>
    <description>
    </description>
  </property>

  <property>
    <name>hive.metastore.schema.verification.record.version</name>
    <value>true</value>
    <description>
      When true the current MS version is recorded in the VERSION table. If this is disabled and verification is
       enabled the MS will be unusable.
    </description>
  </property>

<property>
    <name>hive.execution.engine</name>
    <value>mr</value>
    <description>
      Expects one of [mr, tez, spark].
      Chooses execution engine. Options are: mr (Map reduce, default), tez, spark. While MR
      remains the default engine for historical reasons, it is itself a historical engine
      and is deprecated in Hive 2 line. It may be removed without further warning.
    </description>
  </property>
</configuration>


Copy hive-site.xml:

 
  # Copy 'hive-site.xml' from '/home/user/Dilli/Installed/All_Setup/hive/conf' path to '/home/user/Dilli/Installed/All_Setup/spark/conf' path

   - cp /home/user/Dilli/Installed/All_Setup/hive/conf/hive-site.xml /home/user/Dilli/Installed/All_Setup/spark/conf/hive-site.xml


Hadoop/ Spark Configuration - Data node/ Slave Node address in slaves files:

 - vim /home/user/Dilli/Installed/All_Setup/hadoop/etc/hadoop/slaves
localhost

 - vim /home/user/Dilli/Installed/All_Setup/spark/conf/slaves
localhost


Create the directories to store stuff on name node and datanode and assign ownership
-----------------------------------
- cd /home/user/Dilli/Installed/All_Setup/
- mkdir process_data
- cd process_data

# NameNode
- mkdir name
- chown -R user:user name        #save based on your user   ex: if user is root then use root:user

#Datanode
- mkdir data
- chown -R user:user data



Create Hive user in mysql :

 - mysql -u root -proot123
CREATE USER 'hiver'@'%' IDENTIFIED BY 'hivepassword';
GRANT all on *.* to 'hiver'@localhost identified by 'hivepassword';
flush privileges;

Create metastore in mysql:

 Make sure you point out Correct Schema if you use hive 3 use the below schema or else point to the correct schema available.

 - cd /home/user/Dilli/Installed/All_Setup/hive/scripts/metastore/upgrade/mysql

 - mysql -u hiver -phivepassword
create database metastore_db_3;
use metastore_db_3;
source hive-schema-3.0.0.mysql.sql
exit;

 Hive Conf Add mysql jar file in PATH:

 - ll /usr/share/java/mysql-connector-java.jar

 - cp /usr/share/java/mysql-connector-java.jar /home/user/Dilli/Installed/All_Setup/hive/lib

 Spark Conf Add mysql jar file in PATH:

 - cd /home/user/Dilli/Installed/All_Setup/spark/conf
 - cp /usr/share/java/mysql-connector-java.jar /home/user/Dilli/Installed/All_Setup/spark/jars
 - cp  spark-env.sh.template  spark-env.sh

 - vim spark-env.sh
# add the line
SPARK_CLASSPATH=/usr/share/java/mysql-connector-java.jar

 - cp log4j.properties.template log4j.properties


If Hive database not showing then Add the following line in 'hive.sh' file

 - cd /home/user/Dilli/Installed/All_Setup/hive/bin

 - vim hive
for f in ${SPARK_HOME}/jars/*.jar; do
     CLASSPATH=${CLASSPATH}:$f;
done
# reference url: http://stackoverflow.com/questions/38345447/apache-hive-exception-noclassdeffounderror-scala-collection-iterable.


Hadoop name node format:

 - cd /home/user/Dilli/Installed/All_Setup/hadoop
 - hadoop namenode -format
 - sbin/start-all.sh
 - sbin/start-dfs.sh
 - spark-shell
 - hive
 - cd ../spark/sbin/
 - ./start-all.sh
 - jps

You will see Output Something Like this...











If Data Node is Not Starting then remove the temp files using the below step,

Make sure Hadoop is Stopped.

sudo rm -r /home/user/Dilli/Installed/All_Setup/process_data/data/current


That's it you have successfully Installed All the Three Setups and Integrated Together

Thanks,
Have Fun!!!!

Please Leave your Comment Below If you Have Any Clarifications Regarding Installations or Any Suggestions.









Comments

Post a Comment

Popular posts from this blog

Twitter Analysis Using R

By -
Image
PREREQUISITES: You have already installed  R  and are using  RStudio . In order to extract tweets, you will need a Twitter application and hence a Twitter account. If you don’t have a Twitter account, please  sign up . Use your Twitter login ID and password to sign in at  Twitter Developers . 1.  STEPS TO CREATE A TWITTER APPLICATION a.  Navigate to My Applications in the upper right hand corner. b.  Navigate to My Applications in the upper right hand corner. c.  Create a new application. d.  Fill out the new app form. Names should be unique, i.e., no one else should have used this name for their Twitter app. Give   a brief description of the app. You can change this later on if needed. Enter   your website or blog address.   Callback URL can be left blank. Once you’ve done this, make sure you’ve read the “Developer Rules Of The Road” blurb, check the “Yes, I agree” box, fill in the CAPTCHA and click the “Create Your Twitter Application” button. e.  
Read more

Best Practices for Apache Hive Performance Tuning

By -
If you’re wondering how to scale Apache Hive, here are the best ways to make the most of Hive performance. 1. Partitioning Tables :                                      Hive partitioning is an effective method to improve the query performance on larger tables. 2 . Compress map/reduce output : By enabling compression at various Levels we can achieve the performance improvement in Hive Queries.                                                     Inside Hive-site.xml Add these Properties               <property>                   <name> mapred.compress.map.output </name>                   < value> true </value>               </property>               <property>                   <name> mapred.output.compress </name>                   <value> true </value>             </property> 3 . Map join  :             We can improve the performance of joins by enabling Auto Convert Ma
Read more

Performance Improvement in Hive (Properties in HIVE -SITE.XML)

By -
There are list of Properties Available to increase the Performance in hive. Use the Properties Appropriate to your Need. <?xml version="1.0" encoding="UTF-8" standalone="no"?> <?xml-stylesheet type="text/xsl" href="configuration.xsl"?> <configuration> <property> <name>javax.jdo.option.ConnectionURL</name> <value>jdbc:mysql://localhost:3306/metastore_db_3?createDatabaseIfNotExist=true&amp;useSSL=false</value> <description>JDBC connect string for a JDBC metastore</description> </property> <property> <name>javax.jdo.option.ConnectionDriverName</name> <value>com.mysql.jdbc.Driver</value> <description>Driver class name for a JDBC metastore</description> </property> <property> <name>javax.jdo.option.ConnectionUserName</name> <value>hiveuser</value>
Read more

Installation Steps For MySql Server (version 5.7.23) For Centos

By -
Image
MySQL is an open source free relational database management system (RDBMS) released under GNU (General Public License). It is used to run multiple databases on any single server by providing multi-user access to each created database. Below Steps are provided for  installing MySQL 5.7.23 version on RHEL/CentOS 7/6/5 and Fedora 23/22/21 using MySQL Yum repository via YUM utility. Step 1: Adding the MySQL Yum Repository  We will use official MySQL Yum software repository, which will provides RPM packages for installing the latest version of MySQL  server, client, MySQL Utilities, MySQL Workbench, Connector/ODBC, and Connector/Python for the RHEL/CentOS 7/6/5 and Fedora 23-21. Now download and add the following MySQL Yum repository to your respective Linux distribution system’s repository list to install the latest version of MySQL.             ---------------------------------- On RHEL/CentOS 7 ----------------------------------------------------------------          
Read more

Installation Steps for Hive

By -
Hive is a SQL Database which uses various servers like thrift, Derby, MySQL to maintain the metastore. Here in this setup MySQL server is configured as a base to maintain the metastore of Hive. SSH : -------  - ssh-keygen -t rsa -P ""  - cat $HOME/.ssh/id_rsa.pub >> $HOME/.ssh/authorized_keys  - # ssh-add -l # Not Working the run the command  - ssh localhost   Needed Software’s - Java ,Hadoop,MySql Server Before Installing Hive you make sure all the dependent Software's are correctly Installed .If it is not Installed then you can Refer my Previous Blog to Install them. After Successfully Installing the Softwares correctly!!!! Download Hive from below Repository: https://archive.apache.org/dist/hive The following steps are required for installing Hive on your PC. Let us assume the Hive archive is downloaded onto the /Downloads directory . Here, we download Hive archive named “ ap ache-hive-2.3.0-bin.tar.gz ". The fol
Read more

Facebook Data Analysis Using R

By -
Image
Hey Folks, Today We gonna Learn to Analyse Facebook Data using R . Before Going to concept, makesure you Have R Installed in your System, If Not Proceed the Following Steps to Install. Step 1 – Install R  1. Download the R installer from: https://cran.r-project.org/ 2. Run the installer. Default settings are fine. If you do not have admin rights on your laptop, then ask your local IT support. In that case, it is important that you also ask them to give you full permissions to the R directories. Without this, you will not be able to install additional packages later. Step 2 – Install RStudio 1. Download RStudio: https://www.rstudio.com/products/rstudio/download/ 2. Once the installation of R has completed successfully (and not before), run the RStudio installer. 3. If you do not have administrative rights on your laptop, step 2 may fail. Ask your IT Support or download a pre-built zip archive of RStudio which doesn’t need installing.    a. Download the appropriate a
Read more

Installation Steps For Spark

By -
Image
Downloading  and Installation Steps for Spark: Step 1 : Make sure whether  Java is installed Correctly:               Open Terminal and type.                                                  java -version If Java is already installed correctly on your system, you will  see the following output on your screen:                            Java version "1.8.0_71"              Java(TM) SE Runtime Environment (build 1.8.0_71-b13)              Java HotSpot(TM) Client VM (build 25.0-b02, mixed mode). If java is not installed on your system, then install it.           Ubuntu 14.04 : - sudo add-apt-repository ppa:openjdk-r/ppa - sudo apt-get update - sudo apt-get install openjdk-8-jdk - sudo apt-get install openjdk-8-jre Ubuntu 16.04 : - sudo ap
Read more