S3 Storage

S3 tools available to browse S3 storage

  • S3 Backup – Windows desktop application that makes it trivial for everyone to use Amazon’s impressive infrastructure for remote backup and secure online file storage.
  • Jets3t – Toolkit for Amazon’s S3 online storage service.
  • Sync2S3 – Synchronizes your files with the Amazon (S3),providing you with a secure and affordable backup solutions.
  • SME Storage – Access files from anywhere.
  • Transmit – FTP/SFTP application for Mac.
  • S3Sync – Consist of S3syncs and S3cmds. Ruby program that allows control of Amazon S3 account with shell commands.
  • Bucket Explorer – User Interfaces for Amazon S3.
  • Backup Manager – Command-line tool for Linux.

s31

  • First login in your AWS account.
  • Choose S3 storage from the Services.

s32

  • After clicking on the S3 storage you will see the page as shown below.

s33

  • Click on create bucket button you will see the popup as shown in below image.

s35

  • In the above image you can see four steps follow all the steps.
  • In step1 Enter bucket name as amar2017f-eb and select regioon as Asia pacific(Mumbai)

s36

  • After selecting the region click on next button.
  • You will see the list of  step 2.

s37

  • From that list select versioning and check the suspend versioning and click on save button.

s38

  • After saving the versioning click on next button.

s39

  • In step 3 follow the above image to fill the details.
  • After that click on next button.

s310

  • Click on create button
  • Now you can see the new bucket Name is amar2017f-eb

s313

  • Double click on the new bucket amar2017f-eb you will see same as below image.

s315

  • Create folder Name as Workshop

s314

 

  • Double click on workshop folder you will see empty like below image.

s316

  • Now click on upload button and upload any pdf file.

s317

  • You will see four steps to upload the file, follow the all steps from below images.

s318 s319 s320 s321

  • If you want to delete the uploaded file first tic that file and click on More tab you will get dropdown as shown in the below image.

s322

  • From that dropdown select delete one you willthe page as shown below

s323

  • click on delete button your bucket will become empty.

s324

  • Now the below images showing that how to copy/upload a file in existing folder  using command lines.

s325 s326 s327

  • The above command lines copied two files in the amar2017 folder as show in below.

s328

  • Now check/tic the two files.

s329

  • To delete that files click on More tab.

s330

  • From that More tab select delete link you will see  the image  same as below image.
  • Now click on Delete button that will delete the checked files.

s331

  • Now click on the lifecycle tab you will see the page asshown below.

s332

  • click get started button.
  • This Lifecycle rule is for setting the time to delete the files from bucket.
  • For creating the life cycle rule we have to give rule name.
  • Here rule name is Workshop-ananthapur, click next
  • Follow below images to fill all the steps.

s333 s334 s335

  • The above image is showing that the file is should be detele after 7 days from start of upload.

s336

  • Save the lifecycle rule.

s337

 

s338

Posted in AWS

Elasticsearch/Logstash/Kibana in AWS

Creating a Elasticsearch/Logstash/Kibana in AWS

  • First login your Account
  • Access to AWS services.
  • Select the Elasticsearch under the Analytics as showing in the below.

Screenshot (109)

 

  • After selectting that you will see page as shown in below

Screenshot (110)

  • Click on get started button
  • You will see the page as shown n below

Screenshot (111)

  • Follow all four steps
  • Filll the details,enter the Elasticsearch domain name as ananthapur-demo
  • Elasticseaarch version as 5.1.
  • Then click next button as shown in the below image

Screenshot (112)

  • In the step-2 enter the Instance count as 2.
  • Instance type as t2.small elasticsearch as shown in the below image

Screenshot (113)

  • check the Enable Dedicated Master
  • Select the Dedicated Master Instance Type as t2.small elasticsearch
  • select Dedicated Master Instance Count as 3(default) and
  • Check the Enable zone awareness as shown in the below image..

Screenshot (114)

  • Select storage type as EBS
  • Select EBS volume type as General Purpose(SSD)
  • Select the  EBS volume size as 08:00 UTC
  • As shown in the below image

Screenshot (115)

  • After that select the Automated snapshot start hour as 00:00 UTC(default)
  • Under Advanced Options select default options
  • After that click on next button as shown in the below image.

Screenshot (116)

  • In the step 3 Set the Dmain access polocy as Allow open access to the domain.

Screenshot (117)

  • After selecting that you will see the confirm popup click on Ok button as shown in the beow image.

Screenshot (118)

  • After clicking on ok button youwill see the access polocy code.
  • Then click on confirm and create button as shown in below.

Screenshot (119)

  • If you want to edit the access polocy code you can edit if you dont want to edit click on the next button as shown in the below image.

Screenshot (120)

  • After that you will get the message as you have successfully created Elasticsearch domain

Screenshot (121)

  • It takes 10 minutes time to start the domain

Screenshot (123)

  • Now you can see the Dmain status as Active

Screenshot (124)

  • Click on the link Endpoint to confirm if Elasticsearchis up or not – the o/p similar to below Image will ensure that Elasticsearch is up and running.

Screenshot (126)

  • After that click on the kibana link you will see images as shon below

Screenshot (128)

Screenshot (129)

  • The above image is Kibana Dashboard
  • Enter the Index Name as Stack1.

Screenshot (133)

  • Select Time-field name as DATE

Screenshot (134)

  • After selecting the Date field click on create button you will see the image as shown below.

Screenshot (135)

  • After successfully created dashboard click on the Discover tab it is in the left pane of dasshboard.

Screenshot (136)

  • In above page you can check the data on which date do you want, the below image is showing data from last 2 years.

Screenshot (137)

  • Click on visualize tab it is in left pane of dashboard.
  • From visualized tab click on any type of chart which type do you want to see as shown in the below Image.

Screenshot (138)

 

  • After complition of your tasks come back to your Elasticsearch domain
  • You will the option Delete Elasticsearch domain as shown in the below Image

Screenshot (139)

  • Click on Delete Domain button you will see the popup and again click on delete button.

Screenshot (140)

  • The below Image is showing the ananthapur-demo is being deleted.

Screenshot (141)

 

Posted in AWS
Running an external jar from Aws Hadoop

Running an external jar from Aws Hadoop

Hadoop AWS

  • First select the services  and click on EMR from Analytics.

Screenshot (45)

 

  • Then click on the add cluster.

Screenshot (46)

 

  • Fill the Details of Cluster.
  1. Cluster name as Ananthapur-jntu
  2. Here we are checking the Logging
  3. Browse the s3 folder with the amar2017/feb
  4. Launch mode should be Step Extension
  5. After that select step type as custom jar and click on configure.
  6. The below image is showing the details.

Screenshot (49)

  • After clicking on the configure button we will see the popup like shown below
  1. Name as Custom JAR
  2. Jar location should be s3://amar2017/inputJar/wcount.jar
  3. Fill the Arguments with org.myorg.wordcount, s3://amar2017/deleteme.txt, s3://amar2017/output3
  4. Select the Action on failure as Terminate cluster
  5. Then click on add button.
  6. How to fill the details as shown below.

 

Screenshot (48)

  • Software configuration
  1. Select vendor as Amazon
  2. select Release as emr-5.3.1
  • Hardware Configuration
  1. select instance type as m1.medium
  2. And number of instanses as 3
  • Security and access
  1. Permissions checking as default
  2. After that click on create cluster button
  • Details as shown in the below image

Screenshot (50)

  • You will see the Cluster Ananthapur is Starting as shown below.

Screenshot (51)

  • The below image is showing that in cluster list Ananthapur is starting.

Screenshot (52)

  • After complishing the process we can see like below image.

Screenshot (53)

  • To see the result of AWS Hadoop go to the services.
  • Select S3 under storage.

Screenshot (142)

 

  • After clicking on S3
  • select amar2017

Screenshot (149)

  • Anad then select output3 folder
  • You will see the list of files as shown below

Screenshot (146)

  • Open the part-r-00000 file
  • You will see the page as shown below

Screenshot (147)

  • Click on the Download button
  • And Open the downloaded file you will see the result as shown below.

Screenshot (57)

Creating an Amazon EC2

Creating an Amazon EC2

Creating an Amazon EC2 instance

An EC2 instance is nothing but a virtual server in Amazon Web Services terminology. It stands for Elastic Compute Cloud. It is a web service where an AWS subscriber can request and provision a compute server in AWS cloud.

  • First Create an AWS account
  • Login and access to AWS services.

Step 1) In this step,

  • Login to your AWS account and go to the AWS Services tab at the top left corner.
  • Here, you will see all of the AWS Services categorized as per their area viz. Compute, Storage, Database, etc. For creating an EC2 instance, we have to choose Computeà EC2 as in the next step.

100

  • Open all the services and click on EC2 under Compute services. This will launch the dashboard of EC2.

Here is the EC2 dashboard. Here you will get all the information in gist about the AWS EC2 resources running.

101

Step 2) On the top right corner of the EC2 dashboard, choose the AWS Region in which you want to provision the EC2 server.

Here we are selecting Asia Pacific (Singapore). AWS provides 10 Regions all over the globe.

102

Step 3) In this step

  • Once your desired Region is selected, come back to the EC2 Dashboard.
  • Click on ‘Launch Instance’ button in the section of Create Instance (as shown below).

103

Choose AMI

Step 4) In this step we will do,

  • You will be asked to choose an AMI of your choice. (An AMI is an Amazon Machine Image. It is a template basically of an Operating System platform which you can use as a base to create your instance). Once you launch an EC2 instance from your preferred AMI, the instance will automatically be booted with the desired OS. (We will see more about AMIs in the coming part of the tutorial).
  • Here we are choosing the default Amazon Linux (64 bit) AMI.

104

Choose Instance Types

Step 5) In the next step, you have to choose the type of instance you require based on your business needs.

  • We will choose t2.micro instance type, which is a 1vCPU and 1GB memory server offered by AWS.
  • Click on “Configure Instance Details” for further configurations

105

Configure Instance

Step 6)

  • No. of instances- you can provision up to 20 instances at a time. Here we are launching one instance.
  • Under Purchasing Options, keep the option of ‘Request Spot Instances’ unchecked as of now. (This is done when we wish to launch Spot instances instead of on-demand ones.

106

Step 7) Next, we have to configure some basic networking details for our EC2 server.

  • You have to decide here, in which VPC (Virtual Private Cloud) you want to launch your instance and under which subnets inside your VPC.
  • Network section will give a list of VPCs available in our platform.
  • Select an already existing VPC or you can create your own VPN by clicking onthe create new VPN link.

Here I have selected an default VPC where I want to launch my instance.

107

Step 8) In this step,

  • A VPC consists of subnets, which are IP ranges that are separated for restricting access.
  • Below,
  1. Under Subnets, you can choose the subnet where you want to place your instance.
  2. I have chosen an default existing public subnet.
  3. You can also create a new subnet in this by clicking on the Create new subnet link.

108

Step 9) In this step,

  • You can choose if you want AWS to assign it an IP automatically, or you want to do it manually later. You can enable/ disable ‘Auto assign Public IP’ feature here likewise.
  • Here we are going to assign this instance a static IP called as EIP (Elastic IP) later. So we keep this feature Enabled as of now.

 

110

Step 10) In this step,

  • In the following step, keep the option of IAM role ‘None’ as of now.
  • Shutdown Behavior – when you accidently shut down your instance, you surely don’t want it to be deleted but stopped.
  • Here we are defining my shutdown behavior as Stop.

110

Step 11) In this step,

  • In case, you have accidently terminated your instance, AWS has a layer of security mechanism. It will not delete your instance if you have enabled accidental termination protection.
  • Here we are checking the option for further protecting our instance from accidental termination.

111_LI

Step 12) In this step,

  • Under Monitoring- you can enable Detailed Monitoring if your instance is a business critical instance. Here we have kept the option unchecked. AWS will always provide Basic monitoring on your instance free of cost.
  • Under Tenancy- select the option if shared tenancy. If your application is a highly secure application, then you should go for dedicated capacity. AWS provides both options.
  • Next,Click on ‘Add Storage’ to add data volumes to your instance in next step.

111

Add Storage

Step 13) In this step we do following things,

  • In the Add Storage step, you’ll see that the instance has been automatically provisioned a General Purpose SSD root volume of 8GB. ( Maximum volume size we can give to a General Purpose volume is 16GB)
  • You can change your volume size, add new volumes, change the volume type, etc.
  • AWS provides 3 types of EBS volumes- Magnetic, General Purpose SSD, Provisioned IOPs. You can choose a volume type based on your application’s IOPs needs.
  • Here we are selected the option General Purpose SSD (GP2).

112

Tag Instance

Step 14) In this step

  • you can tag your instance with a key-value pair. This gives visibility to the AWS account administrator when there are lot number of instances.
  • The instances should be tagged based on their department, environment like Dev/SIT/Prod. Etc. this gives a clear view of the costing on the instances under one common tag.
  1. Here we have tagged the instance as a Ananthapur-jntu
  2. Go to configure Security Groups later

113

Configuring Security Groups

Step 15) In this next step of configuring Security Groups, you can restrict traffic on your instance ports. This is an added firewall mechanism provided by AWS apart from your instance’s OS firewall.

You can define open ports and IPs.

  • Since our server is a webserver=, we will do following things
  1. Creating a new Security Group
  2. Naming our SG for easier reference
  3. Defining protocols which we want enabled on my instance
  4. Assigning IPs which are allowed to access our instance on the said protocols
  5. Once, the firewall rules are set- Review and launch

114

Review Instances

Step 16) In this step, we will review all our choices and parameters and go ahead to launch our instance.

115

Step 17) In the next step you will be asked to create a key pair to login to you an instance. A key pair is a set of public-private keys.

AWS stores the private key in the instance, and you are asked to download the public key. Make sure you download the key and keep it safe and secured; if it is lost you cannot download it again.

  1. Create a new key pair
  2. Give a name to your key
  3. Download and save it in your secured folder

116

  • When you download your key, you can open and have a look at your RSA private key.

142

Step 18) Once you are done downloading and saving your key, launch your instance.

  • You can see the launch status meanwhile.

143

  • You can also see the launch log.

118

  • After that click on the view instances button it will shows the your instance.

119

  • Click on the ‘Instances’ option on the left pane where you can see the status of the instance as ‘Pending’ for a brief while.
  • Once your instance is up and running, you can see its status as ‘Running’ now.
  • Note that the instance has received a Private IP from the pool of AWS.

121

Creating a EIP and connecting to your instance

An EIP is a static public IP provided by AWS. It stands for Elastic IP. Normally when you create an instance, it will receive a public IP from the AWS’s pool automatically. If you stop/reboot your instance, this public IP will change- it’dynamic. In order for your application to have a static IP from where you can connect via public networks, you can use an EIP.

Step 19) On the left pane of EC2 Dashboard, you can go to ‘Elastic IPs’ as shown below.

144

Step 20) Allocate a new Elastic IP Address.

123

  • After allocating New Address you will see the success massage as show below.

124

 

Step 21) Now assign this IP to your instance.

  • Select the IP
  • Click on Actions -> Associate Address

126

 

Step 22) In the next page,

  • Search for your instance and
  • Associate the IP to it.

127 128

  • After that,Click on the Associate button you can see the success message as shown below.

130

Step 23) Come back to your instances screen, you’ll see that your instance has received your EIP.

131

Step 24) Now open putty from your programs list and add your same EIP in there as below.

132

Step 25) In this step,

Add your private key in putty for secure connection

  1. Go to Auth
  2. Browse your private key in .ppk (putty private key) format

Once done click on “Open” button

133

  • Once you connect, you will successfully see the Linux prompt.
  • Please note that the machine you are connecting from should be enabled on the instance Security Group for SSH (like in the steps above).

134

  • Now you can see the Running instanses status is 1 in EC2 Dashboard is as shown below.

135

Step 26)

If you want to stop/close the instances  select the  left pane of EC2 Dashboard.

  • click on Actions > Instance state >terminate/stop

136

  • You will see like below image then click on the Yes,Terminate button.

138

  • After clicking on that we can see like below images.

139 140

  • Now see the EC2 Dashboard you shuld see the running instances is ‘0’.

141