Installer for Ubuntu
- Please create a new user with sudo permission
- username – hduser
- Login to the system with the new user
- Setup ssh server
- sudo apt-get install ssh
- Uncheck the option in Startup Application
- RESTART BOX
- Install JPS –
- Please ensure ‘jps’ command is available. If not please type following to install it
- sudo apt-get install openjdk-7-jdk
- Please ensure ‘jps’ command is available. If not please type following to install it
- Download the package
- https://drive.google.com/open?id=0B3z01aLb6U-JNWFoUEpYMV81Q2s
- Check it’s md5 sum
- md5sum woir_workshop.tar.gz
- MD5 ( woir_workshop.tar.gz) = 3dc3efcf732c4222c31ca8dac28c47c1
- Unpack it in the home directory.
- tar xvfz woir_workshop.tar.gz
- Setup the environment
- add following to you in your .bachrc (/home/hduser/.bachrc) in the last
- source sourceme
- Install python packages
- sudo apt-get install python-pip
- sudo pip install pika
- sudo apt-get install erlang
- Cleanup the directory
- rm -rf ~/data_mongodb
- mkdir -p ~/data_mongodb
- Create symbolic link to the JAVA
- ln -s ~/jdk1.8.0_112 ~/JAVA
- Generate key
- ssh-keygen -t rsa -P “”
- Sample o/p
jagdeep@jagdeep-VirtualBox:~$ ssh-keygen -t rsa -P “”
Generating public/private rsa key pair.
Enter file in which to save the key (/home/jagdeep/.ssh/id_rsa):
Your identification has been saved in /home/jagdeep/.ssh/id_rsa.
Your public key has been saved in /home/jagdeep/.ssh/id_rsa.pub.
The key fingerprint is:
24:b2:dd:ec:cb:5f:d4:cc:3a:c6:f1:95:6e:bc:9a:26 jagdeep@jagdeep-VirtualBox
The key’s randomart image is:
+–[ RSA 2048]—-+
| |
| |
| . . . |
| + = + .|
| . . S o +..|
| . o +o. |
| . * .+ |
| . . E oo .|
| o.. oo.. |
+—————–+
- Copy the Key
jagdeep@jagdeep-VirtualBox:~$ cat /home/$USER/.ssh/id_rsa.pub >> /home/$USER/.ssh/authorized_keys
- Validate
jagdeep@jagdeep-VirtualBox:~$ ssh localhost
You should be able to login without any password requirement.
If it ask for the password – please don’t proceed – rather call me.
- Modify following files ( replace amar with the user name you have created above )-
- vi ~/stock-logstash.conf_1.csv
- hadoop-2.6.0/etc/hadoop/core-site.xml
- hadoop-2.6.0/etc/hadoop/hadoop-env.sh
- hadoop-2.6.0/etc/hadoop/hdfs-site.xml
- Sample output
core-site.xml: <value>/home/amar/app/hadoop/tmp</value>
hadoop-env.sh:export JAVA_HOME=/home/amar/JAVA
hdfs-site.xml: <value>file:/home/amar/hadoop_store/hdfs/namenode</value>
hdfs-site.xml: <value>file:/home/amar/hadoop_store/hdfs/datanode</value
- Start Hadoop
- ./start_hadoop.sh ( first time we will have to enter ‘yes’ )
- Sample o/plocalhost: starting namenode, logging to /home/jagdeep/hadoop-2.6.0/logs/hadoop-jagdeep-namenode-jagdeep-VirtualBox.out
localhost: starting datanode, logging to /home/jagdeep/hadoop-2.6.0/logs/hadoop-jagdeep-datanode-jagdeep-VirtualBox.out
Starting secondary namenodes [0.0.0.0]
The authenticity of host ‘0.0.0.0 (0.0.0.0)’ can’t be established.
ECDSA key fingerprint is 8c:f5:a0:bb:63:c0:0e:36:50:cc:4a:c0:60:4c:f6:b5.
Are you sure you want to continue connecting (yes/no)? yes
0.0.0.0: Warning: Permanently added ‘0.0.0.0’ (ECDSA) to the list of known hosts.
- If required format the HDFS partition ( don’t do it unless instructed)
- hadoop namenode -format