Installer for Ubuntu

• Please create a new user with sudo permission
  • username – hduser
• Login to the system with the new user
• Setup ssh server
  • sudo apt-get install ssh
• Uncheck the option in Startup Application
• 
• 
• RESTART BOX
• Install JPS –
  • Please ensure ‘jps’ command is available. If not please type following to install it
    • sudo apt-get install  openjdk-7-jdk
• Download the package
  • https://drive.google.com/open?id=0B3z01aLb6U-JNWFoUEpYMV81Q2s
  • Check it’s md5 sum
    • md5sum woir_workshop.tar.gz
    • MD5 ( woir_workshop.tar.gz) = 3dc3efcf732c4222c31ca8dac28c47c1
  • Unpack it in the home directory.
    • tar xvfz woir_workshop.tar.gz
• Setup the environment
  • add following to you in your .bachrc (/home/hduser/.bachrc) in the last
  • source sourceme
• Install python packages
  • sudo apt-get install python-pip
  • sudo pip install pika
  • sudo apt-get install erlang
• Cleanup the directory
  • rm -rf ~/data_mongodb
  • mkdir -p ~/data_mongodb
• Create symbolic link to the JAVA
  • ln -s ~/jdk1.8.0_112    ~/JAVA
• Generate key
  • ssh-keygen -t rsa -P “”
  • Sample o/p

jagdeep@jagdeep-VirtualBox:~$  ssh-keygen -t rsa -P “”
Generating public/private rsa key pair.
Enter file in which to save the key (/home/jagdeep/.ssh/id_rsa):
Your identification has been saved in /home/jagdeep/.ssh/id_rsa.
Your public key has been saved in /home/jagdeep/.ssh/id_rsa.pub.
The key fingerprint is:
24:b2:dd:ec:cb:5f:d4:cc:3a:c6:f1:95:6e:bc:9a:26 jagdeep@jagdeep-VirtualBox
The key’s randomart image is:
+–[ RSA 2048]—-+
|                 |
|                 |
|    . . .        |
|     + =     +  .|
|    . . S   o +..|
|       .   o +o. |
|        .   * .+ |
|       . . E oo .|
|        o.. oo.. |
+—————–+

• Copy the Key

jagdeep@jagdeep-VirtualBox:~$ cat /home/$USER/.ssh/id_rsa.pub >> /home/$USER/.ssh/authorized_keys

• Validate
  jagdeep@jagdeep-VirtualBox:~$ ssh localhost

You should be able to login without any password requirement.

If it ask for the password – please don’t proceed – rather call me.

• Modify following files ( replace amar with the user name you have created above )-
  • vi  ~/stock-logstash.conf_1.csv
  • hadoop-2.6.0/etc/hadoop/core-site.xml
  • hadoop-2.6.0/etc/hadoop/hadoop-env.sh
  • hadoop-2.6.0/etc/hadoop/hdfs-site.xml
  • Sample output

core-site.xml:  <value>/home/amar/app/hadoop/tmp</value>
hadoop-env.sh:export JAVA_HOME=/home/amar/JAVA
hdfs-site.xml:   <value>file:/home/amar/hadoop_store/hdfs/namenode</value>
hdfs-site.xml:   <value>file:/home/amar/hadoop_store/hdfs/datanode</value

• Start Hadoop
  • ./start_hadoop.sh ( first time we will have to enter ‘yes’ )
  • Sample o/plocalhost: starting namenode, logging to /home/jagdeep/hadoop-2.6.0/logs/hadoop-jagdeep-namenode-jagdeep-VirtualBox.out
    localhost: starting datanode, logging to /home/jagdeep/hadoop-2.6.0/logs/hadoop-jagdeep-datanode-jagdeep-VirtualBox.out
    Starting secondary namenodes [0.0.0.0]
    The authenticity of host ‘0.0.0.0 (0.0.0.0)’ can’t be established.
    ECDSA key fingerprint is 8c:f5:a0:bb:63:c0:0e:36:50:cc:4a:c0:60:4c:f6:b5.
    Are you sure you want to continue connecting (yes/no)? yes
    0.0.0.0: Warning: Permanently added ‘0.0.0.0’ (ECDSA) to the list of known hosts.
• If required format the HDFS partition ( don’t do it unless instructed)
  • hadoop namenode -format