Getting Started with HDP on Google Cloud

Step 1: Creating an account on Google Cloud.

  1. Open the link https://cloud.google.com on the browser.

     


     

  2. Click on ‘Free Trial’ button.

     


     

  3. On the Next screen Select the Account type as ‘Individual’ and Enter your address, Credit card details.


    After Entering all the details click on button.

Step 2:
Creating a Project

  1. On the Next Screen Click on ‘My First Project’, Under this select ‘Manage all Projects’.

     


     

     

  2. Click on ‘Create Project’ button to create the project.

     


     

    Then Enter the ‘Project name’ and ‘Project Id’ On the Next Screen. You need to give the same name for both ‘Project name’ and ‘Project Id’, Then click on ‘Create’.

     


     

    Wait for the Project to be Created.

     


Step 3:
Checking the API & Authentication

  1. Click on ‘API & Auth’ button on the screen.

     


     

     

  2. Under ‘Google Cloud APIs’ click on ‘Compute Engine API’

     


     

  3. Click on ‘Enable API’ button.

     


     

     

     

    Wait until it is enabled.

     


     

     

    Then click on the back button to go to the previous page.

     

     


     

     

     

  4. Click on ‘Cloud Storage API’ to make sure it is enabled or not.


     


     

    Here it is enabled; go back to the Previous Screen by clicking on button.

     

  5. Click on ‘More’ under ‘Google Cloud API’.

     


     

  6. Click on ‘Cloud Storage Json API’ to make sure it is Enabled or not.

     


     


     

    Here it is enabled, click on the button.

     

Step 4:
Configuring Google Cloud SDK & Google Cloud Storage.

  1. Installing Google Cloud SDK locally.

     

    Open the Linux Terminal Using VB or using Cygwin on Windows.

    Here the terminal is opened using VB.

     


     

     

    Enter the below Command on Linux terminal.

Command : curl https://sdk.cloud.google.com | bash


Enter the Directory name to extract all the files.



Enter ‘y’


Enter ‘y’


Press Enter


  1. To use the google cloud commands, restart the terminal.

 

  1. Checking for ‘google’ directory.

 

Command: ls -l


  1. Change the directory to ‘google’

     

    Command:
    cd google

     


     

  2. Checking the list of directories under ‘google’.

     

    Command: ls –l

     


     

  3. Installing google-cloud-sdk locally.

     

    Command: ./google-cloud-sdk/install.sh


    Enter ‘y’

     


     

    Enter ‘y’.

     


     

    Press Enter.

     


     

  4. Checking the google cloud working or not in local machine.

     

    Command: gcloud

     


     


     

     

  5. Authenticating the google cloud login.

     

    Command:
    gcloud auth login

     


     

    It will connect to the google-cloud.

     


     

     

     

     

    Enter your login details to sign in.

     


     

    Click on ‘Accept’ button.

     


     

     

    Now you are authenticated with google cloud.

     

  6. Setting the default project that is created on the google cloud.

     

    Open the terminal

     


     

    Enter the below command to set the project.

     

    Command: gcloud config set project hdp-506

     

    Note: here ‘hdp-506’ is the project-id that we created on google cloud

     


     

     

     

  7. Creating a cloud storage bucket.

     

    Command: gsutil mb -p hdp-506 gs://hdp-506

     


 

Step 4: Downloading the big data utilities.

 

  1. Change the directory.

     

    Command: cd

     


     

  2. Downloading from github.

     

    Command: git clone https://github.com/GoogleCloudPlatform/bdutil

     


     

  3. Change the directory to ‘bdutil’.

     

    Command: cd bdutil

     


     

     

  4. Checking the list of directories in ‘bdutil’.

     

    Command: ls -l

     


     

  5. If ‘bdutil’ don’t have read-write-execute permissions, then change the permissions to read-write-execute using the below command.

     

    Command: chmod +x bdutil

     


     

     

  6. Checking the ‘bdutil’ command is working or not.

     

    Command: ./bdutil –help

     


     

  7. Setting ‘project’ and ‘bucket’ in ‘bdutil_env.sh’ file

     

    Open the file to edit.

     

    Command:
    vim bdutil_env.sh

     


     

    Enter the values the values for CONFIGBUCKET,PROJECT,GCE_ZONE as below.

     

    CONFIGBUCKET = “hdp-506”

    PROJECT = “hdp-506”

     

    GCE_ZONE = “us-central1-a”

     


     

    Save the file.

     

  8. Deploy the client using the below command.

     

    Command: ./bdutil -e platforms/hdp/ambari_env.sh deploy -n 2 -m n1-standard-2

     

     

     

     

     

     

     


     


     

    It will take time finish the process.

     


     

Step 5:
Creating a tunnel.

 

  1. Enter the below command.

 

Command: gcloud –project=hdp-506 compute ssh –zone=us-central1-a hadoop-m


 

  1. Checking the Hadoop commands.

     

    Command: hdfs


     

    Command: hdfs dfsadmin –report

     



Step 6: Access Ambari.

 

  1. Exit from the tunnel.

     

    Command: exit

     


     

  2. Updating our SSH Config for direct SSH access to all nodes

     

    Command: gcloud compute config-ssh

     



  1. Quick tunnel to Apache Ambari.

     

    Command: ssh –L 8080:127.0.0.1:8080 hadoop-m.us-central1-a.hdp-506

     


     

  2. Open Ambari in the browser with the below link.

     

    Link :
    http://localhost:8080/


     

    Enter the fallowing details to sign in.

     

    Username: admin

    Password : admin

     


     

    It will connect to Ambari.


     

     

     

     

 

 

 

 

 

 

 

 


Leave a Reply