HDFS Command

HDFS commands

Print the Hadoop version

hadoop version

 

List the contents of the root directory in HDFS

hadoop fs -ls /

 

Run a DFS filesystem checking utility

hadoop fsck – /

 

Create a new directory named “hadoop” below the

hadoop fs -mkdir /user/training/hadoop

Add a sample text file from the local directory

hadoop fs -put data/sample.txt /user/training/hadoop

 

List the contents of this new directory in HDFS.

hadoop fs -ls /user/training/hadoop

 

Add the entire local directory called “retail” to the

hadoop fs -put data/retail /user/training/hadoop

Delete a file ‘customers’ from the “retail” directory.

hadoop fs -rm hadoop/retail/customers

List the contents that match the specified path, if path not specified it displays the contents of current user.

-ls <path>

Similar to ls command, but recursively list the contents

-lsr <path>

Moves files from <src> to <dst>. If <src> is a pattern that may return multiple files then <dst> should be a directory

- mv <src> <dst>

Similar to “mv” command, but it source won’t be removed after the file(s) copied

-cp <src> <dst>

Copies files from Local system to HDFS

-copyFromLocal <localsrc> <HDFSdst>

Same as copyFromLocal, but the source deleted once the file copied

-moveFromLocal <localsrc> <HDFSdst>

Similar to copyFromLocal

-put <localsrc> <HDFSdst>

Displays the amount of space, in bytes, for files

-du <path>

Displays the amount of space, in bytes, for a specified directory

-dus <path>

Delete all file(s) that match the <src> pattern

-rm <src>

Delete all directory(s) that match the <src> pattern

-rmr <src>

Enables you to specify the File/directory permissions on <src>

-chmod (0-7) (0-7) (0-7) <src>

Count the number of directories, files and bytes under <path>

          -count <path>

Displays statistics about file/directory at <path>

-stat <path>

Write a timestamp in yyyy-MM-dd HH:mm:ss format in an empty file at <path>

-touchz <path>

Set the replication level of a file

-setrep <path>

Show the last 1KB of the file.

-tail <file>

Outputs the file in text format only for zip file or TextRecordInputStream.

-text <src>

Displays help for a command specified

-help [cmd]

Share folder in Window(O/s) & access in Hadoop(VM)

Hi all,

As we are focusing more on practical aspect we need big files to test/learn big data technology. In order to work on that we can share big files in our local machine (window) & can be access in hadoop using below action.


1.  Create Folder & share in our Windows System (ShareWindow)

2.  Then go to Virtual Machine

     Create Folder where we want to see share folder data (ShareVirtual)

3. Run Below command in order to see the content

         sudo mount -t vboxsf sharewindow sharevirtual

          

           (/Sharevirtual) in case different path

Hadoop Configuration File

Hello Guys,

Please keep this configuration(.xml) file handy in order to work in Hadoop Ecosystem.

1. core-site.xml:    It contains the configuration settings for Hadoop Core such as I/O settings that are common to HDFS and MapReduce.

2. hdfs-site.xml: All the configuration settings for HDFS daemons(background process), the namenode, the secondary namenode and the data nodes are specified or can be specified in this file.

1. mapred-site.xml: Configuration settings related to MapReduce daemons : the job-tracker and the task-trackers can be done here.