Getting Started with HDP on Google Cloud

Step 1: Creating an account on Google Cloud.

  1. Open the link on the browser.



  2. Click on ‘Free Trial’ button.



  3. On the Next screen Select the Account type as ‘Individual’ and Enter your address, Credit card details.

    After Entering all the details click on button.

Step 2:
Creating a Project

  1. On the Next Screen Click on ‘My First Project’, Under this select ‘Manage all Projects’.




  2. Click on ‘Create Project’ button to create the project.



    Then Enter the ‘Project name’ and ‘Project Id’ On the Next Screen. You need to give the same name for both ‘Project name’ and ‘Project Id’, Then click on ‘Create’.



    Wait for the Project to be Created.


Step 3:
Checking the API & Authentication

  1. Click on ‘API & Auth’ button on the screen.




  2. Under ‘Google Cloud APIs’ click on ‘Compute Engine API’



  3. Click on ‘Enable API’ button.





    Wait until it is enabled.




    Then click on the back button to go to the previous page.






  4. Click on ‘Cloud Storage API’ to make sure it is enabled or not.



    Here it is enabled; go back to the Previous Screen by clicking on button.


  5. Click on ‘More’ under ‘Google Cloud API’.



  6. Click on ‘Cloud Storage Json API’ to make sure it is Enabled or not.




    Here it is enabled, click on the button.


Step 4:
Configuring Google Cloud SDK & Google Cloud Storage.

  1. Installing Google Cloud SDK locally.


    Open the Linux Terminal Using VB or using Cygwin on Windows.

    Here the terminal is opened using VB.




    Enter the below Command on Linux terminal.

Command : curl | bash

Enter the Directory name to extract all the files.

Enter ‘y’

Enter ‘y’

Press Enter

  1. To use the google cloud commands, restart the terminal.


  1. Checking for ‘google’ directory.


Command: ls -l

  1. Change the directory to ‘google’


    cd google



  2. Checking the list of directories under ‘google’.


    Command: ls –l



  3. Installing google-cloud-sdk locally.


    Command: ./google-cloud-sdk/

    Enter ‘y’



    Enter ‘y’.



    Press Enter.



  4. Checking the google cloud working or not in local machine.


    Command: gcloud





  5. Authenticating the google cloud login.


    gcloud auth login



    It will connect to the google-cloud.






    Enter your login details to sign in.



    Click on ‘Accept’ button.




    Now you are authenticated with google cloud.


  6. Setting the default project that is created on the google cloud.


    Open the terminal



    Enter the below command to set the project.


    Command: gcloud config set project hdp-506


    Note: here ‘hdp-506’ is the project-id that we created on google cloud





  7. Creating a cloud storage bucket.


    Command: gsutil mb -p hdp-506 gs://hdp-506



Step 4: Downloading the big data utilities.


  1. Change the directory.


    Command: cd



  2. Downloading from github.


    Command: git clone



  3. Change the directory to ‘bdutil’.


    Command: cd bdutil




  4. Checking the list of directories in ‘bdutil’.


    Command: ls -l



  5. If ‘bdutil’ don’t have read-write-execute permissions, then change the permissions to read-write-execute using the below command.


    Command: chmod +x bdutil




  6. Checking the ‘bdutil’ command is working or not.


    Command: ./bdutil –help



  7. Setting ‘project’ and ‘bucket’ in ‘’ file


    Open the file to edit.





    Enter the values the values for CONFIGBUCKET,PROJECT,GCE_ZONE as below.


    CONFIGBUCKET = “hdp-506”

    PROJECT = “hdp-506”


    GCE_ZONE = “us-central1-a”



    Save the file.


  8. Deploy the client using the below command.


    Command: ./bdutil -e platforms/hdp/ deploy -n 2 -m n1-standard-2










    It will take time finish the process.



Step 5:
Creating a tunnel.


  1. Enter the below command.


Command: gcloud –project=hdp-506 compute ssh –zone=us-central1-a hadoop-m


  1. Checking the Hadoop commands.


    Command: hdfs


    Command: hdfs dfsadmin –report


Step 6: Access Ambari.


  1. Exit from the tunnel.


    Command: exit



  2. Updating our SSH Config for direct SSH access to all nodes


    Command: gcloud compute config-ssh


  1. Quick tunnel to Apache Ambari.


    Command: ssh –L 8080:



  2. Open Ambari in the browser with the below link.


    Link :


    Enter the fallowing details to sign in.


    Username: admin

    Password : admin



    It will connect to Ambari.













Leave a Reply