Spring XD Installation

Spring XD: https

Spring XD is a unified, distributed, and extensible service for data ingestion, real time analytics, batch processing, and data export.

 Requirement:

To get started, we have to make sure our system has as a minimum Java JDK 7 or newer installed.

Step 1: Download the Spring XD using the below link.

Command: wget https://repo.spring.io/libs-release/org/springframework/xd/spring-xd/1.2.0.RELEASE/spring-xd-1.2.0.RELEASE-dist.zip

1

Step 2: unzip the downloaded package.

Command: unzip spring-xd-1.2.0.RELEASE-dist.zip

1

 

1

 

Step 3: Change the Directory to “spring-xd-1.2.0.RELEASE

Command: cd spring-xd-1.2.0.RELEASE

1

Step 4: Run the  Spring-XD

Spring XD can be run in two different modes. There’s a single-node runtime option for testing and development, and there’s a distributed runtime which supports distribution of processing tasks across multiple nodes.

4.1: Change the directory to xd/bin

Command: cd xd/bin

1

4.2: Run the Spring-XD.

Command: ./xd-singlenode

1

 

1

 

Step 5: Start the Spring-XD Shell.

Spring-XD shell is used to issue commands. Open a new terminal to start the Spring-XD shell.

Command: cd spring-xd-1.2.0.RELEASE

1

Command: cd shell /bin

1

Command:  ./xd-shell

1

Step 6: Creating a Stream.

In Spring XD, a basic stream defines the ingestion of event driven data from a source to a sink that passes through any number of processors. We can create a new stream by issuing a stream create command from the XD shell.

Command: stream create –name ticktock –definition “time | log” –deploy

1

Step 7: Check the result where you have started the server.

Go to the terminal where the server is running.

1

Step 8: Undeploy the stream using the below command.

Command: stream undeploy –name ticktock

1

Type “exit” in the command prompt to exit from the shell.

1

Stop the Server by pressing cntl+c

1

Step 9: Configure the Spring-XD to use Hadoop.

9.1 : Open a new terminal. Change the directory to spring-xd-1.2.0.RELEASE

Command: cd spring-xd-1.2.0.RELEASE

1

9.2 : Change the directory to xd/config.

Command: cd xd/config

1

9.3: Edit the hadoop.properties file.

Command: vi hadoop.properties

1

add the fallowing line.

fs.default.name=hdfs://<Server IP Address>:8020

1

Save and Exit from the file.

9.4 : Change the directory to “ xd”.

Command : cd ..

1

9.5: Change the directory to “bin”.

Command: cd bin

1

Running the spring-xd.

Command: ./xd-singlenode –hadoopDistro cdh5

1

 

1

 

The Hadoop distribution is used for HDFS access

9.5 : Start the Spring-XD shell.

Command: cd spring-xd-1.2.0.RELEASE

1

Command: cd shell/bin

1

Command: ./xd-shell –hadoopDistro cdh5

1

9.6 : Set the name node URL.

Command: hadoop config fs –namenode hdfs://localhost:8020

1

9.7: Test HDFS with a below command .

Command: hadoop fs ls /

1

Exit from the shell

1

Stop the server

1

Step 10: Accessing twitter data into HDFS using Spring XD.

To stream information from Twitter, we will need to set-up a Twitter Developer app so we can get the necessary keys.

Using the keys we can create a stream to access the twitter data.

10.1 : Open a terminal and change the directory to spring-xd.

Command: cd spring-xd-1.2.0.RELEASE

1

Change the directory to xd/config.

Command: cd xd/config

1

Change the directory to “modules”

Command :  cd modules

1

Check the list of files under modules.

Command: ls –l

1

10.2: Modify “ modules.yml” file.

Command: vi modules.yml

1

Add the twitter keys in the file and uncomment the lines.

1

Save and exit the file.

10.4:  Change the directory to “source”.

Command: cd source

1

10.5: Modify the “twittersearch” properties file.

Change the directory to “twittersearch”

Command: cd twittersearch

1

Open the file to edit.

Command: vi twittersearch.properties

1

 

1

Save and exit the file.

10.6 : Modify the “twitterstream” properties file.

Command:  cd ..

1

Change the directory to “twitterstream”

Command: cd twitterstream

1

Open the file to Edit.

Command: vi twitterstream.properties

1

Save and exit the file.

10.7 : Run the spring-xd server.

Command: cd

1

Chane the directory to “spring-xd-1.2.0.RELEASE”

Command: cd spring-xd-1.2.0.RELEASE

1

Change the directory to xd/bin

Command: cd xd/bin

1

Run the Server.

Command: ./xd-singlenode –hadoopDistro cdh5

1

 

1

10.8: Open a new terminal and Start the shell.

Command: cd spring-xd-1.2.0.RELEASE

1

Command: cd shell/bin

1

Command: ./xd-shell –hadoopDistro cdh5

1

Set the name node URL.

Command: hadoop config fs –namenode hdfs://localhost:8020

1

10.9 : Streaming the twitter data using the source “twittersearch” and storing it into Hadoop using the sink “hdfs”.

Command: stream create –name twittersearchcake –definition “twittersearch –query=’cake’ | hdfs” –deploy

Checking the Result in hdfs.

Command: hadoop fs ls /xd/twittersearcake

 

Leave a Reply