Mongo Db Tutorial

Mongo Db Tutorial

Start MongoDB

  • Enter following commands

1.) cd 
2.) /home/hduser/start_mongodb.sh
Wait for approx 30 seconds
3). type mongo
you should see mongodb prompt as below
> db

 

In case if you have difficulty in connecting – try setting following –

 

export LC_ALL=en_US.UTF-8
export LANG=en_US.UTF-8
export LANGUAGE=en_US.UTF-8
Check the DBs already existing in your setup:

> show dbs
local  0.078GB
woir   0.078GB

Delete all the DBs existing to start your tutorial from scratch. To delete / drop a database, make sure you have selected the database and then do this:

> use woir
> db.dropDatabase()
{"dropped" : "scores", "ok" : 1 }

To select a database:

> use sports
switched to db sports

To find out the currently selected database:

> db.getName()
sports

To delete / drop a database, make sure you have selected the database and then do this:

> use sports
> db.dropDatabase()
{"dropped" : "scores", "ok" : 1 }

Create and select DB again

> use sports
switched to db sports

To see the collections in a databases:

> show collections

or

> db.getCollectionNames()

Let’s create a new database named “sports” – it is lazy creation:

> use sports
switched to db sports
> db.information.insert({name:"Andre Adams",year:1975} )

That above commands resulted in the creation of the new database named “sports”. In the process a new collection called “information” was also created in the database.

Create / add data in MongoDB

To create documents in a collection:

> db.information.insert({name:"Brendon McCullum",year:1981})
> db.information.insert({name:"Carl Cachopa",year:1986})

The two commands above added two more documents to the collection named “information” in the “sports” database.

You could have done the same thing using the save command instead of the insert command.

> db.information.save({name:"Chris Cairns",year:1970}) 
> db.information.save({name:"Chris Harris",year:1969})

The difference between insert and save is this:

insert will always add a new document.
save does an insert if there is no _id key in the object it receives, else it does an update.

To read data from a collection

> db.information.find()
> db.information.find()
{ "_id" : ObjectId("5894174f3b9f8fd81c8b9768"), "name" : "Andre Adams", "year" : 1975 }
{ "_id" : ObjectId("589417583b9f8fd81c8b9769"), "name" : "Brendon McCullum", "year" : 1981 }
{ "_id" : ObjectId("5894175e3b9f8fd81c8b976a"), "name" : "Carl Cachopa", "year" : 1986 }
{ "_id" : ObjectId("5894176c3b9f8fd81c8b976b"), "name" : "Chris Cairns", "year" : 1970 }
{ "_id" : ObjectId("589417723b9f8fd81c8b976c"), "name" : "Chris Harris", "year" : 1969 }

To limit it to just two:

> db.information.find().limit(2)
{ "_id" : ObjectId("5894174f3b9f8fd81c8b9768"), "name" : "Andre Adams", "year" : 1975 }
{ "_id" : ObjectId("589417583b9f8fd81c8b9769"), "name" : "Brendon McCullum", "year" : 1981 }

Similar to using find().limit(1),  equivalent –

>db.information.findOne()
{
"_id" : ObjectId("5894174f3b9f8fd81c8b9768"),
"name" : "Andre Adams",
"year" : 1975
}

What if you want to conditionally find documents?

> db.information.find({year:{$lt:1981}})
{ "_id" : ObjectId("5894174f3b9f8fd81c8b9768"), "name" : "Andre Adams", "year" : 1975 }
{ "_id" : ObjectId("5894176c3b9f8fd81c8b976b"), "name" : "Chris Cairns", "year" : 1970 }
{ "_id" : ObjectId("589417723b9f8fd81c8b976c"), "name" : "Chris Harris", "year" : 1969 }

$lt is one of the many conditional operators in MongoDB. Here are the rest.

$lt - ' $lte - ' $gte - '>='
$ne - '!='
$in - 'is in array'
$nin - '! in array'

And how do we do an ‘equal to’ (==) query? Just match the value for the queried key:

> db.information.find({year:1992})
{ "_id" : ObjectId("5894174f3b9f8fd81c8b9768"), "name" : "Andre Adams", "year" : 1975 }

We can even use regular expressions in our queries!

> db.information.find({name:{$regex: /Chri|dre/i}})
{ "_id" : ObjectId("5894174f3b9f8fd81c8b9768"), "name" : "Andre Adams", "year" : 1975 }
{ "_id" : ObjectId("5894176c3b9f8fd81c8b976b"), "name" : "Chris Cairns", "year" : 1970 }
{ "_id" : ObjectId("589417723b9f8fd81c8b976c"), "name" : "Chris Harris", "year" : 1969 }

Now let’s try a more complicated use of regex in a query.

> var names = ['Andre', 'Chri']
> names = names.join('|');
> var re = new RegExp(names, 'i')
> db.information.find({name:{$regex: re}})
{ "_id" : ObjectId("5894174f3b9f8fd81c8b9768"), "name" : "Andre Adams", "year" : 1975 }
{ "_id" : ObjectId("5894176c3b9f8fd81c8b976b"), "name" : "Chris Cairns", "year" : 1970 }
{ "_id" : ObjectId("589417723b9f8fd81c8b976c"), "name" : "Chris Harris", "year" : 1969 }

What if you want to get only some fields in the result?

> db.information.find({year:{'$lt':1994}}, {name:true})
{"_id" : ObjectId("5894174f3b9f8fd81c8b9768"), "name" : "Andre Adams" }
{ "_id" : ObjectId("589417583b9f8fd81c8b9769"), "name" : "Brendon McCullum" }
{ "_id" : ObjectId("5894175e3b9f8fd81c8b976a"), "name" : "Carl Cachopa" }
{ "_id" : ObjectId("5894176c3b9f8fd81c8b976b"), "name" : "Chris Cairns" }
{ "_id" : ObjectId("589417723b9f8fd81c8b976c"), "name" : "Chris Harris" }
To get more than one exclusive fields 
 db.information.find({year:{'$lt':1994}}, {name:true, father'sname:true}). 

The _id field is always returned by default.

To hide it use following –

> db.information.find({year:{'$lt':1994}}, {_id:false})

Try to hide name from the o/p

> db.information.find({year:{'$lt':1994}}, {name:false})
{ "_id" : ObjectId("5894174f3b9f8fd81c8b9768"), "year" : 1975 }
{ "_id" : ObjectId("589417583b9f8fd81c8b9769"), "year" : 1981 }
{ "_id" : ObjectId("5894175e3b9f8fd81c8b976a"), "year" : 1986 }
{ "_id" : ObjectId("5894176c3b9f8fd81c8b976b"), "year" : 1970 }
{ "_id" : ObjectId("589417723b9f8fd81c8b976c"), "year" : 1969 }

BTW field inclusion and exclusion cannot be used together.

Dot Notation -: so let’s create some documents in a collection named ‘articles’:

> db.articles.insert({title:'The Amazingness of MongoDB', meta:{author:'Mike Vallely', date:1321958582668, likes:23, tags:['mongo', 'amazing', 'mongodb']}, comments:[{by:'Steve', text:'Amazing article'}, {by:'Dave', text:'Thanks a ton!'}]})

> db.articles.insert({title:'Mongo Business', meta:{author:'Chad Muska', date:1321958576231, likes:10, tags:['mongodb', 'business', 'mongo']}, comments:[{by:'Taylor', text:'First!'}, {by:'Rob', text:'I like it'}]})

> db.articles.insert({title:'MongoDB in Mongolia', meta:{author:'Ghenghiz Khan', date:1321958598538, likes:75, tags:['mongo', 'mongolia', 'ghenghiz']}, comments:[{by:'Alex', text:'Dude, it rocks'}, {by:'Steve', text:'The best article ever!'}]})

Note the dot notation

> db.articles.find({'meta.author':'Chad Muska'})
> db.articles.find({'meta.likes':{$gt:10}})

Searching array:

> db.articles.find({'meta.tags':'mongolia'})

When the key is an array, the database looks for the object right in the array.

> db.articles.find({'comments.by':'Steve'})

Refer to array indexes:

> db.articles.find({'comments.0.by':'Steve'})

Always remember that a quoted number is a string, and is not the same as the actual number. For example:

> db.employee.find({salary:100})

is totally different from

> db.employee.find({salary:'100'})

Try following two –

> db.information.find('this.year > 1971 && this.name != "Andre Adams"')

 

> db.information.find({year:{$gt:1971}, name:{$ne:'Andre Adams'}})

Both above are same.

MongoDB has another operator called $where using which you can perform SQL’s WHERE-like operations.

> db.information.find({$where: 'this.year > 1971'})
{ "_id" : ObjectId("5894174f3b9f8fd81c8b9768"), "name" : "Andre Adams", "year" : 1975 }
{ "_id" : ObjectId("589417583b9f8fd81c8b9769"), "name" : "Brendon McCullum", "year" : 1981 }
{ "_id" : ObjectId("5894175e3b9f8fd81c8b976a"), "name" : "Carl Cachopa", "year" : 1986 }

and

> db.information.find({name:'Andre Adams', $where: 'this.year > 1970'})
{ "_id" : ObjectId("5894174f3b9f8fd81c8b9768"), "name" : "Andre Adams", "year" : 1975 }

 

Update data in MongoDB

Document replacement-

> db.information.update({name:"Andre Adams"}, {father:'Stephen Adams'})

 

Appending –

 

> db.information.update({name:"Andre Adams"}, {'$set':{father:'Stephen Adams', players:['Crickets', 'Hockey']}})

Update an array

> db.information.update({name:"Andre Adams"}, {'$push':{players:'Kabbadi'}})
> db.information.update({name:"Andre Adams"}, {'$push':{players:'Tennis'}})

Eh, we need to remove something from the cast array. We do it this way:

db.information.update({name:"Andre Adams"}, {'$pull':{players:'Kabbadi'}})

Delete data in MongoDB

 

> db.information.update({name:'Andre Adams'}, {$unset:{players:1}})

Delete a field from all the document of a collection:

> db.information.update({$unset:{players:1}}, false, true)

The false parameter if for upsert option, true is for multiple option. We set multiple option to true because we want to delete them all from the collection.

Delete all documents from a collection with name Andre Adams

> db.information.remove({name:'Andre Adams'})

Drop all the documents

> db.information.remove()

The above command truncates the collection.

Delete / drop a collection-

> db.information.drop()

To delete a database select the database and call the db.dropDatabase() on it:

> use information
> db.dropDatabase()
{"dropped" : "information", "ok" : 1 }

Get the count of Documents / Records

 

> db.information.count({})

This will return the total number of documents in the collection named information with the value of year more than 1970:

> db.information.count({year:{$gt:1990})

 

 

Screen Shot 2017-02-03 at 11.32.13 AM

RabbitMQ

RabbitMQ

RabbitMQ

  • Open terminal and check the all files
ls /home/amar
  • Start the RabbitMQ
/home/amar/start_rabbitmq.sh
  • You should see something similar as below

rabbit

 

  • Open the link
 http://localhost:15672/
  • You should see something similar as below

rabbitrr

  • Enter the username and password as guest
  • You should see something similar as below

memu

 

  • Click on the queues tab  and create the queues
  • You should see something similar as below

cccccccccccco

  • Queues have different parameters and arguments depending on how they were created. The features column show the parameters that belong to the queue. It could be features like Durable queue (which ensure that RabbitMQ will never lose the queue), Message TTL (which tells how long a message published to a queue can live before it is discarded), Auto expire (which tells how long a queue can be unused for before it is automatically deleted), Max length (which tells how many (ready) messages a queue can contain before it starts to drop them) and Max length bytes (which tells the total body size for ready messages a queue can contain before it starts to drop them).
  • You can also create a queue from this view.
  • If you press on any chosen queue from the list of queues, all information about the queue are shown like in the pictures that follow below.
  • You should see something similar as below

ooooooooo

  • The first two charts include the same information as the overview, but it just shows the number of queued messages and the message rates for that specific queue.
Admin
  • From the Admin view it is possible to add users and change user permissions. You can set up vhosts, policies, federation and shovels.

fhfhf

 

addddd

Exchanges
  • An exchange receives messages from producers and pushes them to queues. The exchange must know exactly what to do with a message it receives. All exchanges can be listed from the exchange tab. Virtual host shows the vhost for the exchange, type is the exchange type such as direct, topic, headers, fanout. Features show the parameters for the exchange (e.g. D stand for durable, and AD for auto-delete). Features and types can be specified when the exchange is created. In this list there are some amq.* exchanges and the default (unnamed) exchange. These are created by default.

exexex

 

Example

  • This example show how you can create a queue “example-queue” and an exchange called example.exchange.

adddnew

exchangeeee

 

  • The exchange and the queue are connected by a binding called “pdfprocess”. Messages published to the exchange with the routing key “pdfprocess” will end up in the queue.

ghghghghhgg

pbpbpbbpbp

 

  • If you want to stop the RabbitMQ
  • First logout from the RabbitMQ after that run this command

/home/amar/stop_rabbitmq.sh

 

 

 

ActiveMQ

ActiveMQ

ActiveMQ

  • Open terminal and check the all files
ls /home/woir
  • Start the Activemq
/home/woir/stop_all.sh
/home/woir/start_activemq.sh
  • You should see something similar as below

start

 

activemq

 

  • Click on the Manage ActiveMQ broker
  • Enter the username and password as admin
  • You should see something similar as below

login

 

  • Click on the Queues tab
  • After clicking on queues tab you will see all list of queus
  • You should see something similar as below

queues

 

  • If you want to create a new queue you can create
  • Enter the new Queue name in the above input box and click on create button
  • From Queues List page, You can click on any queue’s name, then you can see the all the messages from  that queue’s name
  • You should see something similar as below

messages

Create a Message manually from web

  • If the destination queue exists, then from the queue list page, click the Send To link. If the destination queue does not exists, you can click Send from the menu, and give a name of the destination, the queue will be create automatically when sending message.

send

 

  • Type the text content, then click send 

send msg

 

  • Click on the message Id, then you can see the details of the message. A message is composed by message header and message body.
  • You should see something similar as below

mssg _details

 

Delete a specified message in a Queue

  • From the message list of the Queue, click Delete to remove message.

delete

 

Delete all messages of a queue

  • From the Queue list, click Purge.

all delete

Deleting the Queue

  • From the Queue list, click on Delete .

queue delete

 

Generating messages programmatically

Producers
Send message to queue “woir”
  • Send one message to queue name “woir”
curl -u admin:admin -d "body='Hello World 1 '" "http://localhost:8161/api/message/woir?type=queue&clientId=consumerA"
  • Send second message
curl -u admin:admin -d "body='Hello World 2'" "http://localhost:8161/api/message/woir?type=queue&clientId=consumerA"

Send third message

curl -u admin:admin -d "body='Hello World 3'" "http://localhost:8161/api/message/woir?type=queue&clientId=consumerA"
Send message to queue “college”

Send one message to queue name “college”

curl -u admin:admin -d "body='Hello World 1 '" "http://localhost:8161/api/message/college?type=queue&clientId=consumerA"
  • Send second message
curl -u admin:admin -d "body='Hello World 2'" "http://localhost:8161/api/message/college?type=queue&clientId=consumerA"
  • Send third message
    curl -u admin:admin -d "body='Hello World 3'" "http://localhost:8161/api/message/college?type=queue&clientId=consumerA"
  • Check the messages from GUI ( You should see something similar as below)

msg create

Consumers

Read Messages from “woir” queue

From command line type following –

  • Read First Message
curl -u admin:admin -X delete "http://localhost:8161/api/message/woir?type=queue&clientId=consumerA"
  • Read Second Message
curl -u admin:admin -X delete "http://localhost:8161/api/message/woir?type=queue&clientId=consumerA"

 

  • Read Third Message

 

  • curl -u admin:admin -X delete "http://localhost:8161/api/message/woir?type=queue&clientId=consumerA"

Read Messages from “college” queue

From command line type following –

  • Read First Message
curl -u admin:admin -X delete "http://localhost:8161/api/message/college?type=queue&clientId=consumerA"
  • Read Second Message
curl -u admin:admin -X delete "http://localhost:8161/api/message/college?type=queue&clientId=consumerA"
  • Read Third Message
  • curl -u admin:admin -X delete "http://localhost:8161/api/message/college?type=queue&clientId=consumerA"
Observe the Order of receiving messages – they must come in the FIFO order.
  • You should see something similar as below

consumer

  • For stopping the ActiveMQ run this command
/home/woir/stop_activemq.sh

 

  • Hands-on interaction
    • sudo pip install stompy
  • Paste following code in your home area ( say p2.py )
  • Run command –
    • python p2.py -q “/queue/test” -P -n 1 -m “WOIR Message One”   ( producing messages  to queue)
    • python p2.py -q “/queue/test” -c -n 5
    • python p2.py -q “/topic/test” -P -n 1 -m “WOIR Message One”   ( producing messages to topic )
    • python p2.py -q “/topic/test” -c -n 5

 

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
#!/usr/bin/env python
import sys
import itertools
from stompy import Client as StompClient
from itertools import count
from optparse import OptionParser

def callback_func(message):
	print("Using Callback Function")
	print(message.headers.get('message-id'))
	print(message.body)
	stomp.ack(message)

def consume(queue, num=None, callback=None):

	stomp.subscribe(queue, ack="client")

	def _handle_message(frame):
		print(frame.headers.get("message-id"))
		print(frame.body)
		stomp.ack(frame)

	# if num is not set, iterate forever.
	it = xrange(0, num) if num else itertools.count()

	try:
		for i in it:
			if callback:
				stomp.get(callback=callback_func)
			else:
				frame = stomp.get()
				_handle_message(frame)
	finally:
		stomp.disconnect()


def produce(queue, num=1000, message="Hello Friend"):

	for i in xrange(0, num):
		print("Message #%d for message = %s " % (i, message))
		this_frame = stomp.put("Testing %d for %s" % (i,message),
							   destination=queue,
							   persistent=True)
		print("Receipt: %s" % this_frame.headers.get("receipt-id"))

	stomp.disconnect()


if __name__ == '__main__':
	parser = OptionParser()
	parser.add_option('-H', '--host', action='store', default="localhost",
					  type='string', dest='host', help='hostname')
	parser.add_option('-p', '--port', action='store',
					  type='int', dest='port', help='port', default=61613)
	parser.add_option('-q', '--queue', action='store',
					  type='string', dest='queue', help='destination queue')
	parser.add_option('-P', '--produce', action='store_true',
					  default=False, dest='produce', help='produce messages')
	parser.add_option('-c', '--consume', action='store_true',
					  default=False, dest='consume', help='consume messages')
	parser.add_option('-C', '--use-callback', action='store_true',
					  default=False, dest='callback',
					  help='send retrieved message to python callable')
	parser.add_option('-n', '--number', action='store',
					  type='int', dest='number', default="100",
					  help='produce or consume NUMBER messages')
	parser.add_option('-m', '--message', action='store',
					  type='string', dest='message', default="Hello friend",
					  help='Type the message you want to send')

	options, args = parser.parse_args()

	if not options.queue:
		print("Queue name is required!")
		parser.print_help()
		sys.exit(1)

	stomp = StompClient(options.host, options.port)
	# optional connect keyword args "username" and "password" like so:
	# stomp.connect(username="user", password="pass")
	stomp.connect()

	if options.produce:
		print("PRODUCING")
		#produce("/topic/hello", options.number, options.message)
		produce(options.queue, options.number, options.message)
	elif options.consume:
		consume(options.queue, options.number, options.callback)
		#consume("/topic/hello", options.number, options.callback)
Installer for Ubuntu

Installer for Ubuntu

  • Please create a new user with sudo permission
    • username – hduser
  • Login to the system with the new user
  • Setup ssh server
    • sudo apt-get install ssh
  • Uncheck the option in Startup Application
  • Screen Shot 2017-01-30 at 11.56.14 AM
  • Screen Shot 2017-01-30 at 11.56.43 AM
  • RESTART BOX
  • Install JPS –
    • Please ensure ‘jps’ command is available. If not please type following to install it
      • sudo apt-get install  openjdk-7-jdk
  • Download the package
    • https://drive.google.com/open?id=0B3z01aLb6U-JNWFoUEpYMV81Q2s
    • Check it’s md5 sum
      • md5sum woir_workshop.tar.gz
      • MD5 ( woir_workshop.tar.gz) = 3dc3efcf732c4222c31ca8dac28c47c1
    • Unpack it in the home directory.
      • tar xvfz woir_workshop.tar.gz
  • Setup the environment
    • add following to you in your .bachrc (/home/hduser/.bachrc) in the last
    • source sourceme
  • Install python packages
    • sudo apt-get install python-pip
    • sudo pip install pika
    • sudo apt-get install erlang
  • Cleanup the directory
    • rm -rf ~/data_mongodb
    • mkdir -p ~/data_mongodb
  • Create symbolic link to the JAVA
    • ln -s ~/jdk1.8.0_112    ~/JAVA
  • Generate key
    • ssh-keygen -t rsa -P “”
    • Sample o/p

jagdeep@jagdeep-VirtualBox:~$  ssh-keygen -t rsa -P “”
Generating public/private rsa key pair.
Enter file in which to save the key (/home/jagdeep/.ssh/id_rsa):
Your identification has been saved in /home/jagdeep/.ssh/id_rsa.
Your public key has been saved in /home/jagdeep/.ssh/id_rsa.pub.
The key fingerprint is:
24:b2:dd:ec:cb:5f:d4:cc:3a:c6:f1:95:6e:bc:9a:26 jagdeep@jagdeep-VirtualBox
The key’s randomart image is:
+–[ RSA 2048]—-+
|                 |
|                 |
|    . . .        |
|     + =     +  .|
|    . . S   o +..|
|       .   o +o. |
|        .   * .+ |
|       . . E oo .|
|        o.. oo.. |
+—————–+

  • Copy the Key

jagdeep@jagdeep-VirtualBox:~$ cat /home/$USER/.ssh/id_rsa.pub >> /home/$USER/.ssh/authorized_keys

  • Validate
    jagdeep@jagdeep-VirtualBox:~$ ssh localhost

You should be able to login without any password requirement.

If it ask for the password – please don’t proceed – rather call me.

 

 

  • Modify following files ( replace amar with the user name you have created above )-
    • vi  ~/stock-logstash.conf_1.csv
    • hadoop-2.6.0/etc/hadoop/core-site.xml
    • hadoop-2.6.0/etc/hadoop/hadoop-env.sh
    • hadoop-2.6.0/etc/hadoop/hdfs-site.xml
    • Sample output

core-site.xml:  <value>/home/amar/app/hadoop/tmp</value>
hadoop-env.sh:export JAVA_HOME=/home/amar/JAVA
hdfs-site.xml:   <value>file:/home/amar/hadoop_store/hdfs/namenode</value>
hdfs-site.xml:   <value>file:/home/amar/hadoop_store/hdfs/datanode</value

  • Start Hadoop
    • ./start_hadoop.sh ( first time we will have to enter ‘yes’ )
    • Sample o/plocalhost: starting namenode, logging to /home/jagdeep/hadoop-2.6.0/logs/hadoop-jagdeep-namenode-jagdeep-VirtualBox.out
      localhost: starting datanode, logging to /home/jagdeep/hadoop-2.6.0/logs/hadoop-jagdeep-datanode-jagdeep-VirtualBox.out
      Starting secondary namenodes [0.0.0.0]
      The authenticity of host ‘0.0.0.0 (0.0.0.0)’ can’t be established.
      ECDSA key fingerprint is 8c:f5:a0:bb:63:c0:0e:36:50:cc:4a:c0:60:4c:f6:b5.
      Are you sure you want to continue connecting (yes/no)? yes
      0.0.0.0: Warning: Permanently added ‘0.0.0.0’ (ECDSA) to the list of known hosts.
  • If required format the HDFS partition ( don’t do it unless instructed)
    • hadoop namenode -format

 

Steps to Verify Hadoop/Elasticsearch/ActiveMQ/Cassandra Installation in One Box Setup VM

Steps to Verify Hadoop/Elasticsearch/ActiveMQ/Cassandra Installation in One Box Setup VM

Hadoop Installation

  •     Start Virtual Box, choose the machine you prepared in earlier step and click on the “Start” button ( green colour ).

screen-shot-2016-12-09-at-11-08-22-am

  • Please login as user Hadoop ( user id hduser), if asked for please enter password ‘abcd1234’

Screen Shot 2017-01-27 at 12.41.46 AM

  • Click on the Ubuntu on the top-left corner and look for terminal and click on the terminal

screen-shot-2016-12-09-at-11-10-51-am

  • Once the terminal is up and running it should look similar to following –

Screen Shot 2017-01-27 at 12.42.28 AM

  • Go to home directory and take a look on the directory presents
    • cd /home/hduser
    • ‘pwd’ command should show path as ‘/home/hduser’.
    • execute ‘ls -lart’ to take a look on the files and directory in general.
  • Close already running applications
    • /home/hduser/stop_all.sh
  • Start hadoop
    • /home/hduser/start_hadoop.sh
  • Confirm that service is running successfully or not
    • run ‘jps’ – you should see something similar to following –

screen-shot-2016-12-09-at-12-44-15-pm

  • Run wordcount program by using following command –
    • /home/hduser/run_helloword.sh
  • At the end you should see something similar –

screen-shot-2016-12-09-at-11-44-33-am

  • Check if the output files have been generated
  • hadoop dfs -ls /user/hduser/output     – you should see something similar to below screenshot

screen-shot-2016-12-09-at-11-46-35-am

  • Get the contents of the output files ( similar to following ) –
    • hadoop dfs -cat /user/hduser/output/part-r-00000

screen-shot-2016-12-09-at-11-48-22-am

  • Finally shutdown the hadoop services
    • /home/hduser/stop_hadoop.sh

Elasticsearch Installation

  • Close already running applications
    • /home/hduser/stop_all.sh
  • Start Elasticsearch –
    • /home/hduser/start_elasticsearch.sh
    • tail /home/hduser/elastic123.log
      • You should see some messages ( it should not have any ERROR ) in the last you may something similar –

screen-shot-2016-12-09-at-12-22-59-pm

  • Verify Elasticsearch instance
    • Open browser ( firefox )
    • goto http://localhost:9200
    • You should see following output

screen-shot-2016-12-09-at-12-26-20-pm

 

  • Start Kibana –
    • /home/hduser/start_kibana.sh
    • tail /home/hduser/kibana123.log
  • You should see some messages ( it should not have any ERROR ) in the last you may something similar –

 

Screen Shot 2017-01-27 at 1.04.45 AM

  • Verify Kibana instance
    • Open browser ( firefox )
    • goto http://localhost:5601/app/kibana#
    • You should see similar output

screen-shot-2016-12-16-at-1-26-42-am

  • Shutdown Elasticsearch and Kibana
    • /home/hduser/stop_elasticsearch.sh
    • /home/hduser/stop_kibana.sh

ActiveMQ Installation

  • Close already running applications
    • /home/hduser/stop_all.sh
  • Start ActiveMQ
    • /home/hduser/start_activemq.sh
  • Run validation test – send messages
    • cd /home/hduser/activemq-5.14.3
    • /home/hduser/activemq-5.14.3/send_message.sh
  • See following output on the screen

Screen Shot 2017-01-27 at 1.19.50 AM

  • Continue – receive messages
    • cd /home/hduser/activemq-5.14.3
    • /home/hduser/activemq-5.14.3/receive_message.sh
  • See following output on the screen

Screen Shot 2017-01-27 at 1.24.01 AM

  • Stop ActiveMQ
    • /home/hduser/stop_activemq.sh

 

Steps to Install VM for workshop

Steps to Install VM for workshop

  1. Install Virtual Box ( follow the steps ) on your computer.
  2. Once the Virtual Box is installed  the VM ( please see the license terms and conditions to use it for any commercial/business purpose) can be downloaded from the following link –
    • https://drive.google.com/open?id=0B2vqFbCIJR_USXBzNVZYZGloOVU
  3. The downloaded image name will be ‘woir.vdi’.
  4. Create a New Virtual Machine in VirtualBox using the uncompressed VDI file as the Hard Drive.
    • Run VirtualBox
    • Click the “New” button
    • Enter the name “Ubuntu-Vasvi”;
    • Select “Linux” with the OS Type dropdown
    • Select “Next”
    • On the “Memory” panel choose around 4 gb memory and click “Next”
    • On the Virtual Hard Disk” panel select “Existing” – this opens the VirtualBox Virtual Disk Manager”
    • Select the “Add” button.
    • Select the “harddisk which you have downloaded” ( in this case it should be Vijayawada.vdi) file.
    • Click “Select”
    • Click “Next”
    • Click “Finished”
    • Click RUN to Start the VM ( you should see Ubuntu running )
    • Use username as woir and password as abcd1234 whenever required.
Steps to Install one box setup of Hadoop

Steps to Install one box setup of Hadoop

  1. Install Virtual Box ( follow the steps ) on your computer.
  2. Once the Virtual Box is installed Download the VM ( please see the license terms and conditions to use it for any commercial/business purpose).
  3. The downloaded image name will be ‘Ubuntu-cvr-2.vdi’.
  4. Create a New Virtual Machine in VirtualBox using the uncompressed VDI file as the Hard Drive.
    • Run VirtualBox
    • Click the “New” button
    • Click “Next”
    • Enter the name “Ubuntu-CVR”;
    • Select “Linux” with the OS Type dropdown
    • Select “Next”
    • On the “Memory” panel choose around 4 gb memory and click “Next”
    • On the Virtual Hard Disk” panel select “Existing” – this opens the VirtualBox Virtual Disk Manager”
    • Select the “Add” button.
    • Select the “harddisk which you have downloaded” ( in this case it should be Ubuntu-cvr-2.vdi) file.
    • Click “Select”
    • Click “Next”
    • Click “Finished”
    • Click RUN to Start the VM ( you should see Ubuntu running )