Installer for Ubuntu

Installer for Ubuntu

  • Please create a new user with sudo permission
    • username – hduser
  • Login to the system with the new user
  • Setup ssh server
    • sudo apt-get install ssh
  • Uncheck the option in Startup Application
  • Screen Shot 2017-01-30 at 11.56.14 AM
  • Screen Shot 2017-01-30 at 11.56.43 AM
  • RESTART BOX
  • Install JPS –
    • Please ensure ‘jps’ command is available. If not please type following to install it
      • sudo apt-get install  openjdk-7-jdk
  • Download the package
    • https://drive.google.com/open?id=0B3z01aLb6U-JNWFoUEpYMV81Q2s
    • Check it’s md5 sum
      • md5sum woir_workshop.tar.gz
      • MD5 ( woir_workshop.tar.gz) = 3dc3efcf732c4222c31ca8dac28c47c1
    • Unpack it in the home directory.
      • tar xvfz woir_workshop.tar.gz
  • Setup the environment
    • add following to you in your .bachrc (/home/hduser/.bachrc) in the last
    • source sourceme
  • Install python packages
    • sudo apt-get install python-pip
    • sudo pip install pika
    • sudo apt-get install erlang
  • Cleanup the directory
    • rm -rf ~/data_mongodb
    • mkdir -p ~/data_mongodb
  • Create symbolic link to the JAVA
    • ln -s ~/jdk1.8.0_112    ~/JAVA
  • Generate key
    • ssh-keygen -t rsa -P “”
    • Sample o/p

jagdeep@jagdeep-VirtualBox:~$  ssh-keygen -t rsa -P “”
Generating public/private rsa key pair.
Enter file in which to save the key (/home/jagdeep/.ssh/id_rsa):
Your identification has been saved in /home/jagdeep/.ssh/id_rsa.
Your public key has been saved in /home/jagdeep/.ssh/id_rsa.pub.
The key fingerprint is:
24:b2:dd:ec:cb:5f:d4:cc:3a:c6:f1:95:6e:bc:9a:26 jagdeep@jagdeep-VirtualBox
The key’s randomart image is:
+–[ RSA 2048]—-+
|                 |
|                 |
|    . . .        |
|     + =     +  .|
|    . . S   o +..|
|       .   o +o. |
|        .   * .+ |
|       . . E oo .|
|        o.. oo.. |
+—————–+

  • Copy the Key

jagdeep@jagdeep-VirtualBox:~$ cat /home/$USER/.ssh/id_rsa.pub >> /home/$USER/.ssh/authorized_keys

  • Validate
    jagdeep@jagdeep-VirtualBox:~$ ssh localhost

You should be able to login without any password requirement.

If it ask for the password – please don’t proceed – rather call me.

 

 

  • Modify following files ( replace amar with the user name you have created above )-
    • vi  ~/stock-logstash.conf_1.csv
    • hadoop-2.6.0/etc/hadoop/core-site.xml
    • hadoop-2.6.0/etc/hadoop/hadoop-env.sh
    • hadoop-2.6.0/etc/hadoop/hdfs-site.xml
    • Sample output

core-site.xml:  <value>/home/amar/app/hadoop/tmp</value>
hadoop-env.sh:export JAVA_HOME=/home/amar/JAVA
hdfs-site.xml:   <value>file:/home/amar/hadoop_store/hdfs/namenode</value>
hdfs-site.xml:   <value>file:/home/amar/hadoop_store/hdfs/datanode</value

  • Start Hadoop
    • ./start_hadoop.sh ( first time we will have to enter ‘yes’ )
    • Sample o/plocalhost: starting namenode, logging to /home/jagdeep/hadoop-2.6.0/logs/hadoop-jagdeep-namenode-jagdeep-VirtualBox.out
      localhost: starting datanode, logging to /home/jagdeep/hadoop-2.6.0/logs/hadoop-jagdeep-datanode-jagdeep-VirtualBox.out
      Starting secondary namenodes [0.0.0.0]
      The authenticity of host ‘0.0.0.0 (0.0.0.0)’ can’t be established.
      ECDSA key fingerprint is 8c:f5:a0:bb:63:c0:0e:36:50:cc:4a:c0:60:4c:f6:b5.
      Are you sure you want to continue connecting (yes/no)? yes
      0.0.0.0: Warning: Permanently added ‘0.0.0.0’ (ECDSA) to the list of known hosts.
  • If required format the HDFS partition ( don’t do it unless instructed)
    • hadoop namenode -format