附：HDFS shell guide文檔地址
http://hadoop.apache.org/docs/r2.5.2/hadoop-project-dist/hadoop-common/FileSystemShell.html
啟動HDFS后，輸入hadoop fs命令，即可顯示HDFS常用命令的用法
[hadoop@localhost hadoop-2.5.2]$ hadoop fs Usage: hadoop fs [generic options] [-appendToFile <localsrc> ... <dst>] [-cat [-ignoreCrc] <src> ...] [-checksum <src> ...] [-chgrp [-R] GROUP PATH...] [-chmod [-R] <MODE[,MODE]... | OCTALMODE> PATH...] [-chown [-R] [OWNER][:[GROUP]] PATH...] [-copyFromLocal [-f] [-p] <localsrc> ... <dst>] [-copyToLocal [-p] [-ignoreCrc] [-crc] <src> ... <localdst>] [-count [-q] <path> ...] [-cp [-f] [-p | -p[topax]] <src> ... <dst>] [-createSnapshot <snapshotDir> [<snapshotName>]] [-deleteSnapshot <snapshotDir> <snapshotName>] [-df [-h] [<path> ...]] [-du [-s] [-h] <path> ...] [-expunge] [-get [-p] [-ignoreCrc] [-crc] <src> ... <localdst>] [-getfacl [-R] <path>] [-getfattr [-R] {-n name | -d} [-e en] <path>] [-getmerge [-nl] <src> <localdst>] [-help [cmd ...]] [-ls [-d] [-h] [-R] [<path> ...]] [-mkdir [-p] <path> ...] [-moveFromLocal <localsrc> ... <dst>] [-moveToLocal <src> <localdst>] [-mv <src> ... <dst>] [-put [-f] [-p] <localsrc> ... <dst>] [-renameSnapshot <snapshotDir> <oldName> <newName>] [-rm [-f] [-r|-R] [-skipTrash] <src> ...] [-rmdir [--ignore-fail-on-non-empty] <dir> ...] [-setfacl [-R] [{-b|-k} {-m|-x <acl_spec>} <path>]|[--set <acl_spec> <path>]] [-setfattr {-n name [-v value] | -x name} <path>] [-setrep [-R] [-w] <rep> <path> ...] [-stat [format] <path> ...] [-tail [-f] <file>] [-test -[defsz] <path>] [-text [-ignoreCrc] <src> ...] [-touchz <path> ...] [-usage [cmd ...]] Generic options supported are -conf <configuration file> specify an application configuration file -D <property=value> use value for given property -fs <local|namenode:port> specify a namenode -jt <local|jobtracker:port> specify a job tracker -files <comma separated list of files> specify comma separated files to be copied to the map reduce cluster -libjars <comma separated list of jars> specify comma separated jar files to include in the classpath. -archives <comma separated list of archives> specify comma separated archives to be unarchived on the compute machines. The general command line syntax is bin/hadoop command [genericOptions] [commandOptions]
>幫助相關命令
- usage
查看命令的用法，例查看ls的用法
[hadoop@localhost hadoop-2.5.2]$ hadoop fs -usage ls Usage: hadoop fs [generic options] -ls [-d] [-h] [-R] [<path> ...]
- help
查看命令的詳細幫助，例查看ls命令的幫助：
[hadoop@localhost hadoop-2.5.2]$ hadoop fs -help ls -ls [-d] [-h] [-R] [<path> ...] : List the contents that match the specified file pattern. If path is not specified, the contents of /user/<currentUser> will be listed. Directory entries are of the form: permissions - userId groupId sizeOfDirectory(in bytes) modificationDate(yyyy-MM-dd HH:mm) directoryName and file entries are of the form: permissions numberOfReplicas userId groupId sizeOfFile(in bytes) modificationDate(yyyy-MM-dd HH:mm) fileName -d Directories are listed as plain files. -h Formats the sizes of files in a human-readable fashion rather than a number of bytes. -R Recursively list the contents of directories.
>查看相關命令
- ls
查看文件或目錄，下例中：hdfs://localhost:9000是fs.defaultFS配置的值，hdfs://localhost:9000/即表示HDFS文件系統中根目錄，如果使用的是HDFS文件系統， 可以簡寫為/。
[hadoop@localhost hadoop-2.5.2]$ hadoop fs -ls hdfs://localhost:9000/ Found 3 items drwxr-xr-x - hadoop supergroup 0 2015-03-27 19:19 hdfs://localhost:9000/input -rw-r--r-- 1 hadoop supergroup 14 2015-03-31 07:17 hdfs://localhost:9000/input1.txt drwxr-xr-x - hadoop supergroup 0 2015-03-27 19:16 hdfs://localhost:9000/output [hadoop@localhost hadoop-2.5.2]$ hadoop fs -ls / Found 3 items drwxr-xr-x - hadoop supergroup 0 2015-03-27 19:19 /input -rw-r--r-- 1 hadoop supergroup 14 2015-03-31 07:17 /input1.txt drwxr-xr-x - hadoop supergroup 0 2015-03-27 19:16 /output
選項-R：連同子目錄的文件一起列出，例：
[hadoop@localhost hadoop-2.5.2]$ hadoop fs -ls -R / drwxr-xr-x - hadoop supergroup 0 2015-03-27 19:19 /input -rw-r--r-- 1 hadoop supergroup 14 2015-03-27 19:19 /input/input1.txt --子目錄下的文件也被列出 -rw-r--r-- 1 hadoop supergroup 32 2015-03-27 19:19 /input/input2.txt -rw-r--r-- 1 hadoop supergroup 14 2015-03-31 07:17 /input1.txt drwxr-xr-x - hadoop supergroup 0 2015-03-27 19:16 /output
- cat
顯示文件內容
[hadoop@localhost hadoop-2.5.2]$ hadoop fs -cat /input1.txt hello hadoop! hello hadoop!
- text
將給定的文件以文本的格式輸出，允許的格式zip、TextRecordInputStream、Avro。當文件為文本文件時，等同於cat。例：
[hadoop@localhost hadoop-2.5.2]$ hadoop fs -text /input1.txt hello hadoop!
- tail
顯示文件最后1KB的內容
選項-f：當文件內容增加時顯示追加的內容
- checksum
顯示文件的校驗和信息。因為需要和存儲文件每個塊的datanode互相通信，因此對大量的文件使用此命令效率可能會低
[hadoop@localhost hadoop-2.5.2]$ hadoop fs -checksum /input.zip /input.zip MD5-of-0MD5-of-0CRC32 00000000000000000000000070bc8f4b72a86921468bf8e8441dce51
>文件及目錄相關命令
- touchz
創建一個空文件，如果存在指定名稱的非空文件，則返回錯誤
[hadoop@localhost hadoop-2.5.2]$ hadoop fs -ls / Found 3 items drwxr-xr-x - hadoop supergroup 0 2015-03-27 19:19 /input -rw-r--r-- 1 hadoop supergroup 184 2015-03-31 08:14 /input1.zip drwxr-xr-x - hadoop supergroup 0 2015-04-02 08:34 /output [hadoop@localhost hadoop-2.5.2]$ hadoop fs -touchz /input1.zip touchz: `/input1.zip': Not a zero-length file --非空時給出錯誤提示 [hadoop@localhost hadoop-2.5.2]$ hadoop fs -touchz /input.zip [hadoop@localhost hadoop-2.5.2]$ hadoop fs -ls / Found 4 items drwxr-xr-x - hadoop supergroup 0 2015-03-27 19:19 /input -rw-r--r-- 1 hadoop supergroup 0 2015-04-02 08:43 /input.zip --創建成功 -rw-r--r-- 1 hadoop supergroup 184 2015-03-31 08:14 /input1.zip drwxr-xr-x - hadoop supergroup 0 2015-04-02 08:34 /output
- appendToFile
向現有文件中追加內容，例：
[hadoop@localhost hadoop-2.5.2]$ hadoop fs -text /input1.txt hello hadoop! [hadoop@localhost hadoop-2.5.2]$ hadoop fs -appendToFile ~/Desktop/input1.txt /input1.txt [hadoop@localhost hadoop-2.5.2]$ hadoop fs -text /input1.txt
hello hadoop! hello hadoop! --查看追加后的文件內容
- put
從本地文件系統上傳文件到HDFS
[hadoop@localhost hadoop-2.5.2]$ hadoop fs -put ~/Desktop/input1.txt / [hadoop@localhost hadoop-2.5.2]$ hadoop fs -text /input1.txt --查看上傳后的文件內容 hello hadoop!
選項-f：如果文件已經存在，覆蓋已有文件
[hadoop@localhost hadoop-2.5.2]$ hadoop fs -put ~/Desktop/input1.txt / put: `/input1.txt': File exists --文件已存在時給出錯誤提示 [hadoop@localhost hadoop-2.5.2]$ hadoop fs -put -f ~/Desktop/input1.txt / [hadoop@localhost hadoop-2.5.2]$ --使用-f選項后沒有再報錯
選項-p：保留原文件的訪問、修改時間，用戶和組，權限屬性
[hadoop@localhost hadoop-2.5.2]$ ll ~/input1.txt -rw-r--r--. 1 hadoop hadoops 28 Mar 31 08:59 /home/hadoop/input1.txt --本地文件屬性 [hadoop@localhost hadoop-2.5.2]$ chmod 777 ~/input1.txt --修改權限為rwxrwxrwx [hadoop@localhost hadoop-2.5.2]$ hadoop fs -put ~/input1.txt / [hadoop@localhost hadoop-2.5.2]$ hadoop fs -ls /input1.txt -rw-r--r-- 1 hadoop supergroup 28 2015-04-02 05:19 /input1.txt --不使用-p選項，上傳后文件屬性 [hadoop@localhost hadoop-2.5.2]$ hadoop fs -put -f -p ~/input1.txt / [hadoop@localhost hadoop-2.5.2]$ hadoop fs -ls /input1.txt -rwxrwxrwx 1 hadoop hadoops 28 2015-03-31 08:59 /input1.txt --使用-p選項，上傳后文件屬性
- get
從HDFS上下載文件到本地，與put不同，沒有覆蓋本地已有文件的選項
[hadoop@localhost hadoop-2.5.2]$ hadoop fs -get /input1.txt ~ [hadoop@localhost hadoop-2.5.2]$ cat ~/input1.txt --查看本地下載的文件 hello hadoop! hellp hadoop!
- getmerge
將指定的HDFS中原目錄下的文件合並成一個文件並下載到本地，源文件保留
[hadoop@localhost hadoop-2.5.2]$ hadoop fs -text /input/input1.txt hello hadoop! --input1.txt內容 [hadoop@localhost hadoop-2.5.2]$ hadoop fs -text /input/input2.txt welcome to the world of hadoop! --input2.txt內容 [hadoop@localhost hadoop-2.5.2]$ hadoop fs -getmerge /input/ ~/merge.txt [hadoop@localhost hadoop-2.5.2]$ cat ~/merge.txt hello hadoop! --合並后本地文件的內容 welcome to the world of hadoop!
選項-nl：在每個文件的最后增加一個新行
[hadoop@localhost hadoop-2.5.2]$ hadoop fs -getmerge -nl /input/ ~/merge.txt [hadoop@localhost hadoop-2.5.2]$ cat ~/merge.txt hello hadoop! --input1.txt增加的新行 welcome to the world of hadoop! --input2.txt增加的新行 [hadoop@localhost hadoop-2.5.2]$
- copyFromLocal
從本地文件系統上傳文件到HDFS，與put命令相同
- copyToLocal
從HDFS下載文件到本地文件系統，與get命令相同
- moveFromLocal
與put命令相同，只是上傳成功后本地文件會被刪除
- moveToLocal
該命令還未實現
- mv
同linux的mv命令，移動或重命名文件
[hadoop@localhost hadoop-2.5.2]$ hadoop fs -ls / Found 5 items drwxr-xr-x - hadoop supergroup 0 2015-03-27 19:19 /input -rw-r--r-- 1 hadoop supergroup 184 2015-03-31 08:14 /input.zip -rwxrwxrwx 1 hadoop hadoops 28 2015-03-31 08:59 /input1.txt drwxr-xr-x - hadoop supergroup 0 2015-03-27 19:16 /output drwxr-xr-x - hadoop supergroup 0 2015-04-02 07:10 /text [hadoop@localhost hadoop-2.5.2]$ hadoop fs -mv /input.zip /input1.zip [hadoop@localhost hadoop-2.5.2]$ hadoop fs -ls / Found 5 items drwxr-xr-x - hadoop supergroup 0 2015-03-27 19:19 /input -rwxrwxrwx 1 hadoop hadoops 28 2015-03-31 08:59 /input1.txt -rw-r--r-- 1 hadoop supergroup 184 2015-03-31 08:14 /input1.zip --重命名 drwxr-xr-x - hadoop supergroup 0 2015-03-27 19:16 /output drwxr-xr-x - hadoop supergroup 0 2015-04-02 07:10 /text [hadoop@localhost hadoop-2.5.2]$ hadoop fs -mv /input1.zip /text/ [hadoop@localhost hadoop-2.5.2]$ hadoop fs -ls -R / drwxr-xr-x - hadoop supergroup 0 2015-03-27 19:19 /input -rw-r--r-- 1 hadoop supergroup 14 2015-03-27 19:19 /input/input1.txt -rw-r--r-- 1 hadoop supergroup 32 2015-03-27 19:19 /input/input2.txt -rwxrwxrwx 1 hadoop hadoops 28 2015-03-31 08:59 /input1.txt drwxr-xr-x - hadoop supergroup 0 2015-03-27 19:16 /output drwxr-xr-x - hadoop supergroup 0 2015-04-02 07:12 /text -rw-r--r-- 1 hadoop supergroup 184 2015-03-31 08:14 /text/input1.zip --移動文件
- cp
復制文件
[hadoop@localhost hadoop-2.5.2]$ hadoop fs -ls -R / drwxr-xr-x - hadoop supergroup 0 2015-03-27 19:19 /input -rw-r--r-- 1 hadoop supergroup 14 2015-03-27 19:19 /input/input1.txt -rw-r--r-- 1 hadoop supergroup 32 2015-03-27 19:19 /input/input2.txt -rwxrwxrwx 1 hadoop hadoops 28 2015-03-31 08:59 /input1.txt -rw-r--r-- 1 hadoop supergroup 184 2015-03-31 08:14 /input1.zip drwxr-xr-x - hadoop supergroup 0 2015-03-27 19:16 /output drwxr-xr-x - hadoop supergroup 0 2015-04-02 07:29 /text [hadoop@localhost hadoop-2.5.2]$ hadoop fs -cp /input1.txt /input.txt [hadoop@localhost hadoop-2.5.2]$ hadoop fs -ls -R / drwxr-xr-x - hadoop supergroup 0 2015-03-27 19:19 /input -rw-r--r-- 1 hadoop supergroup 14 2015-03-27 19:19 /input/input1.txt -rw-r--r-- 1 hadoop supergroup 32 2015-03-27 19:19 /input/input2.txt -rw-r--r-- 1 hadoop supergroup 28 2015-04-02 07:31 /input.txt --新復制文件 -rwxrwxrwx 1 hadoop hadoops 28 2015-03-31 08:59 /input1.txt -rw-r--r-- 1 hadoop supergroup 184 2015-03-31 08:14 /input1.zip drwxr-xr-x - hadoop supergroup 0 2015-03-27 19:16 /output drwxr-xr-x - hadoop supergroup 0 2015-04-02 07:29 /text
選項-f：如果文件已存在，覆蓋已有文件
[hadoop@localhost hadoop-2.5.2]$ hadoop fs -cp /input1.txt /input.txt cp: `/input.txt': File exists --文件已存在時給出錯誤提示 [hadoop@localhost hadoop-2.5.2]$ hadoop fs -cp -f /input1.txt /input.txt [hadoop@localhost hadoop-2.5.2]$
- mkdir
創建文件夾
[hadoop@localhost hadoop-2.5.2]$ hadoop fs -mkdir /text [hadoop@localhost hadoop-2.5.2]$ hadoop fs -ls / Found 5 items drwxr-xr-x - hadoop supergroup 0 2015-03-27 19:19 /input -rw-r--r-- 1 hadoop supergroup 184 2015-03-31 08:14 /input.zip -rw-r--r-- 1 hadoop supergroup 210 2015-03-31 07:49 /input1.txt drwxr-xr-x - hadoop supergroup 0 2015-03-27 19:16 /output drwxr-xr-x - hadoop supergroup 0 2015-03-31 08:23 /text
選項-p：如果上層目錄不存在，遞歸建立所需目錄
[hadoop@localhost hadoop-2.5.2]$ hadoop fs -mkdir /text1/text2 mkdir: `/text1/text2': No such file or directory --上層目錄不存在，給出錯誤提示 [hadoop@localhost hadoop-2.5.2]$ hadoop fs -mkdir -p /text1/text2 [hadoop@localhost hadoop-2.5.2]$ hadoop fs -ls -R / drwxr-xr-x - hadoop supergroup 0 2015-03-27 19:19 /input -rw-r--r-- 1 hadoop supergroup 14 2015-03-27 19:19 /input/input1.txt -rw-r--r-- 1 hadoop supergroup 32 2015-03-27 19:19 /input/input2.txt -rw-r--r-- 1 hadoop supergroup 184 2015-03-31 08:14 /input.zip -rw-r--r-- 1 hadoop supergroup 210 2015-03-31 07:49 /input1.txt drwxr-xr-x - hadoop supergroup 0 2015-03-27 19:16 /output drwxr-xr-x - hadoop supergroup 0 2015-03-31 08:23 /text drwxr-xr-x - hadoop supergroup 0 2015-03-31 08:26 /text1 drwxr-xr-x - hadoop supergroup 0 2015-03-31 08:26 /text1/text2 --使用-p選項，創建成功
- rm
刪除文件
[hadoop@localhost hadoop-2.5.2]$ hadoop fs -rm /input.zip 15/03/31 08:02:32 INFO fs.TrashPolicyDefault: Namenode trash configuration: Deletion interval = 0 minutes, Emptier interval = 0 minutes. Deleted /input.zip
選項-r：遞歸的刪除，可以刪除非空目錄
[hadoop@localhost hadoop-2.5.2]$ hadoop fs -rm /text rm: `/text': Is a directory --刪除文件夾時，給出錯誤提示 [hadoop@localhost hadoop-2.5.2]$ hadoop fs -rm -r /text --使用-r選項，文件夾及文件夾下文件刪除成功 15/04/02 08:28:42 INFO fs.TrashPolicyDefault: Namenode trash configuration: Deletion interval = 0 minutes, Emptier interval = 0 minutes. Deleted /text
- rmdir
刪除空目錄
[hadoop@localhost hadoop-2.5.2]$ hadoop fs -ls -R / drwxr-xr-x - hadoop supergroup 0 2015-03-27 19:19 /input -rw-r--r-- 1 hadoop supergroup 14 2015-03-27 19:19 /input/input1.txt -rw-r--r-- 1 hadoop supergroup 32 2015-03-27 19:19 /input/input2.txt -rw-r--r-- 1 hadoop supergroup 184 2015-03-31 08:14 /input1.zip drwxr-xr-x - hadoop supergroup 0 2015-04-02 08:34 /output -rwxrwxrwx 1 hadoop hadoops 28 2015-03-31 08:59 /output/input1.txt [hadoop@localhost hadoop-2.5.2]$ hadoop fs -rmdir /output rmdir: `/output': Directory is not empty --不能刪除非空目錄
選項--ignore-fail-on-non-empty：忽略非空刪除失敗時的提示
[hadoop@localhost hadoop-2.5.2]$ hadoop fs -rmdir --ignore-fail-on-non-empty /output [hadoop@localhost hadoop-2.5.2]$ hadoop fs -ls -R / drwxr-xr-x - hadoop supergroup 0 2015-03-27 19:19 /input -rw-r--r-- 1 hadoop supergroup 14 2015-03-27 19:19 /input/input1.txt -rw-r--r-- 1 hadoop supergroup 32 2015-03-27 19:19 /input/input2.txt -rw-r--r-- 1 hadoop supergroup 184 2015-03-31 08:14 /input1.zip drwxr-xr-x - hadoop supergroup 0 2015-04-02 08:34 /output --不給出錯誤提示，但文件未刪除 -rwxrwxrwx 1 hadoop hadoops 28 2015-03-31 08:59 /output/input1.txt
- setrep
改變一個文件的副本數
[hadoop@localhost hadoop-2.5.2]$ hadoop fs -stat %r /input.zip 1 --原副本數 [hadoop@localhost hadoop-2.5.2]$ hadoop fs -setrep 2 /input.zip Replication 2 set: /input.zip [hadoop@localhost hadoop-2.5.2]$ hadoop fs -stat %r /input.zip 2 --改變后副本數
選項-w：命令等待副本數調整完成
[hadoop@localhost hadoop-2.5.2]$ hadoop fs -setrep -w 1 /input.zip Replication 1 set: /input.zip Waiting for /input.zip ... done [hadoop@localhost hadoop-2.5.2]$ hadoop fs -stat %r /input.zip 1
- expunge
清空回收站
[hadoop@localhost hadoop-2.5.2]$ hadoop fs -expunge 15/04/03 01:52:46 INFO fs.TrashPolicyDefault: Namenode trash configuration: Deletion interval = 0 minutes, Emptier interval = 0 minutes.
- chgrp
修改文件用戶組
[hadoop@localhost hadoop-2.5.2]$ hadoop fs -ls -R / drwxr-xr-x - hadoop supergroup 0 2015-03-27 19:19 /input -rw-r--r-- 1 hadoop supergroup 14 2015-03-27 19:19 /input/input1.txt -rw-r--r-- 1 hadoop supergroup 32 2015-03-27 19:19 /input/input2.txt -rw-r--r-- 1 hadoop supergroup 0 2015-04-02 08:43 /input.zip -rw-r--r-- 1 hadoop supergroup 184 2015-03-31 08:14 /input1.zip drwxr-xr-x - hadoop supergroup 0 2015-04-02 08:34 /output --文件原用戶組 -rwxrwxrwx 1 hadoop hadoops 28 2015-03-31 08:59 /output/input1.txt [hadoop@localhost hadoop-2.5.2]$ hadoop fs -chgrp test /output [hadoop@localhost hadoop-2.5.2]$ hadoop fs -ls -R / drwxr-xr-x - hadoop supergroup 0 2015-03-27 19:19 /input -rw-r--r-- 1 hadoop supergroup 14 2015-03-27 19:19 /input/input1.txt -rw-r--r-- 1 hadoop supergroup 32 2015-03-27 19:19 /input/input2.txt -rw-r--r-- 1 hadoop supergroup 0 2015-04-02 08:43 /input.zip -rw-r--r-- 1 hadoop supergroup 184 2015-03-31 08:14 /input1.zip drwxr-xr-x - hadoop test 0 2015-04-02 08:34 /output --修改后的用戶組（未建立test組，仍可成功） -rwxrwxrwx 1 hadoop hadoops 28 2015-03-31 08:59 /output/input1.txt --目錄下文件的用戶組未修改
選項-R：遞歸修，如果是目錄，則遞歸的修改其下的文件及目錄
[hadoop@localhost hadoop-2.5.2]$ hadoop fs -chgrp -R testgrp /output [hadoop@localhost hadoop-2.5.2]$ hadoop fs -ls -R / drwxr-xr-x - hadoop supergroup 0 2015-03-27 19:19 /input -rw-r--r-- 1 hadoop supergroup 14 2015-03-27 19:19 /input/input1.txt -rw-r--r-- 1 hadoop supergroup 32 2015-03-27 19:19 /input/input2.txt -rw-r--r-- 1 hadoop supergroup 0 2015-04-02 08:43 /input.zip -rw-r--r-- 1 hadoop supergroup 184 2015-03-31 08:14 /input1.zip drwxr-xr-x - hadoop testgrp 0 2015-04-02 08:34 /output --目錄及其下文件都被更改 -rwxrwxrwx 1 hadoop testgrp 28 2015-03-31 08:59 /output/input1.txt
- chmod
修改文件權限，權限模式同linux shell命令中的模式
[hadoop@localhost hadoop-2.5.2]$ hadoop fs -ls -R / drwxr-xr-x - hadoop supergroup 0 2015-03-27 19:19 /input -rw-r--r-- 1 hadoop supergroup 14 2015-03-27 19:19 /input/input1.txt -rw-r--r-- 1 hadoop supergroup 32 2015-03-27 19:19 /input/input2.txt -rw-r--r-- 1 hadoop supergroup 0 2015-04-02 08:43 /input.zip -rw-r--r-- 1 hadoop supergroup 184 2015-03-31 08:14 /input1.zip drwxr-xr-x - hadoop supergroup 0 2015-04-02 08:34 /output --文件原權限 -rwxrwxrwx 1 hadoop supergroup 28 2015-03-31 08:59 /output/input1.txt [hadoop@localhost hadoop-2.5.2]$ hadoop fs -chmod 754 /output [hadoop@localhost hadoop-2.5.2]$ hadoop fs -ls -R / drwxr-xr-x - hadoop supergroup 0 2015-03-27 19:19 /input -rw-r--r-- 1 hadoop supergroup 14 2015-03-27 19:19 /input/input1.txt -rw-r--r-- 1 hadoop supergroup 32 2015-03-27 19:19 /input/input2.txt -rw-r--r-- 1 hadoop supergroup 0 2015-04-02 08:43 /input.zip -rw-r--r-- 1 hadoop supergroup 184 2015-03-31 08:14 /input1.zip drwxr-xr-- - hadoop supergroup 0 2015-04-02 08:34 /output --修改后的權限 -rwxrwxrwx 1 hadoop supergroup 28 2015-03-31 08:59 /output/input1.txt --目錄下文件的權限未修改 [hadoop@localhost hadoop-2.5.2]$ hadoop fs -chmod -R 775 /output [hadoop@localhost hadoop-2.5.2]$ hadoop fs -ls -R / drwxr-xr-x - hadoop supergroup 0 2015-03-27 19:19 /input -rw-r--r-- 1 hadoop supergroup 14 2015-03-27 19:19 /input/input1.txt -rw-r--r-- 1 hadoop supergroup 32 2015-03-27 19:19 /input/input2.txt -rw-r--r-- 1 hadoop supergroup 0 2015-04-02 08:43 /input.zip -rw-r--r-- 1 hadoop supergroup 184 2015-03-31 08:14 /input1.zip drwxrwxr-x - hadoop supergroup 0 2015-04-02 08:34 /output --目錄及其下文件都被更改 -rwxrwxr-x 1 hadoop supergroup 28 2015-03-31 08:59 /output/input1.txt
- chown
修改文件的用戶或組
[hadoop@localhost hadoop-2.5.2]$ hadoop fs -ls -R / drwxr-xr-x - hadoop supergroup 0 2015-03-27 19:19 /input -rw-r--r-- 1 hadoop supergroup 14 2015-03-27 19:19 /input/input1.txt -rw-r--r-- 1 hadoop supergroup 32 2015-03-27 19:19 /input/input2.txt -rw-r--r-- 1 hadoop supergroup 0 2015-04-02 08:43 /input.zip -rw-r--r-- 1 hadoop supergroup 184 2015-03-31 08:14 /input1.zip drwxrwxr-x - hadoop supergroup 0 2015-04-02 08:34 /output --文件原用戶和組 -rwxrwxr-x 1 hadoop supergroup 28 2015-03-31 08:59 /output/input1.txt [hadoop@localhost hadoop-2.5.2]$ hadoop fs -chown test /output [hadoop@localhost hadoop-2.5.2]$ hadoop fs -ls -R / drwxr-xr-x - hadoop supergroup 0 2015-03-27 19:19 /input -rw-r--r-- 1 hadoop supergroup 14 2015-03-27 19:19 /input/input1.txt -rw-r--r-- 1 hadoop supergroup 32 2015-03-27 19:19 /input/input2.txt -rw-r--r-- 1 hadoop supergroup 0 2015-04-02 08:43 /input.zip -rw-r--r-- 1 hadoop supergroup 184 2015-03-31 08:14 /input1.zip drwxrwxr-x - test supergroup 0 2015-04-02 08:34 /output --修改后的用戶（未建立test用戶，仍可成功） -rwxrwxr-x 1 hadoop supergroup 28 2015-03-31 08:59 /output/input1.txt --目錄下文件的用戶未修改
選項-R：遞歸修改，如果是目錄，則遞歸的修改其下的文件及目錄
[hadoop@localhost hadoop-2.5.2]$ hadoop fs -chown -R testown:testgrp /output [hadoop@localhost hadoop-2.5.2]$ hadoop fs -ls -R / drwxr-xr-x - hadoop supergroup 0 2015-03-27 19:19 /input -rw-r--r-- 1 hadoop supergroup 14 2015-03-27 19:19 /input/input1.txt -rw-r--r-- 1 hadoop supergroup 32 2015-03-27 19:19 /input/input2.txt -rw-r--r-- 1 hadoop supergroup 0 2015-04-02 08:43 /input.zip -rw-r--r-- 1 hadoop supergroup 184 2015-03-31 08:14 /input1.zip drwxrwxr-x - testown testgrp 0 2015-04-02 08:34 /output --目錄及其下文件都被更改 -rwxrwxr-x 1 testown testgrp 28 2015-03-31 08:59 /output/input1.txt
- getfacl
顯示訪問控制列表ACLs(Access Control Lists)
[hadoop@localhost bin]$ hadoop fs -getfacl /input.zip # file: /input.zip # owner: hadoop # group: supergroup user::rw- group::r-- other::r--
選項-R：遞歸顯示
[hadoop@localhost bin]$ hadoop fs -getfacl -R /input # file: /input # owner: hadoop # group: supergroup user::rwx group::r-x other::r-x # file: /input/input1.txt # owner: hadoop # group: supergroup user::rw- group::r-- other::r-- # file: /input/input2.txt # owner: hadoop # group: supergroup user::rw- group::r-- other::r--
- setfacl
設置訪問控制列表，acls默認未開啟，直接使用該命令會報錯
[hadoop@localhost bin]$ hadoop fs -setfacl -b /output/input1.txt setfacl: The ACL operation has been rejected. Support for ACLs has been disabled by setting dfs.namenode.acls.enabled to false.
開啟acls，配置hdfs-site.xml
[hadoop@localhost hadoop-2.5.2]$ vi etc/hadoop/hdfs-site.xml <property> <name>dfs.namenode.acls.enabled</name> <value>true</value> </property>
選項-m：修改acls
[hadoop@localhost hadoop-2.5.2]$ hadoop fs -getfacl /output/input1.txt # file: /output/input1.txt # owner: testown # group: testgrp user::rwx group::rwx other::r-x [hadoop@localhost hadoop-2.5.2]$ hadoop fs -setfacl -m user::rw-,user:hadoop:rw-,group::r--,other::r-- /output/input1.txt [hadoop@localhost hadoop-2.5.2]$ hadoop fs -getfacl /output/input1.txt # file: /output/input1.txt # owner: testown # group: testgrp user::rw- user:hadoop:rw- group::r-- mask::rw- other::r--
選項-x：刪除指定規則
[hadoop@localhost hadoop-2.5.2]$ hadoop fs -setfacl -m user::rw-,user:hadoop:rw-,group::r--,other::r-- /output/input1.txt [hadoop@localhost hadoop-2.5.2]$ hadoop fs -getfacl /output/input1.txt # file: /output/input1.txt # owner: testown # group: testgrp user::rw- user:hadoop:rw- group::r-- mask::rw- other::r-- [hadoop@localhost hadoop-2.5.2]$ hadoop fs -setfacl -x user:hadoop /output/input1.txt [hadoop@localhost hadoop-2.5.2]$ hadoop fs -getfacl /output/input1.txt # file: /output/input1.txt # owner: testown # group: testgrp user::rw- group::r-- mask::r-- other::r--
以下選項未做實驗
選項-b：基本的acl規則(所有者，群組，其他）被保留，其他規則全部刪除.
選項-k：刪除缺省規則
- setfattr
設置擴展屬性的名稱和值
選項-n：屬性名稱 選項-v：屬性值
[hadoop@localhost hadoop-2.5.2]$ hadoop fs -getfattr -d /input.zip # file: /input.zip [hadoop@localhost hadoop-2.5.2]$ hadoop fs -setfattr -n user.web -v www.baidu.com /input.zip [hadoop@localhost hadoop-2.5.2]$ hadoop fs -getfattr -d /input.zip # file: /input.zip user.web="www.baidu.com"
選項-x：刪除擴展屬性
[hadoop@localhost hadoop-2.5.2]$ hadoop fs -getfattr -d /input.zip # file: /input.zip user.web="www.baidu.com" [hadoop@localhost hadoop-2.5.2]$ hadoop fs -setfattr -x user.web /input.zip [hadoop@localhost hadoop-2.5.2]$ hadoop fs -getfattr -d /input.zip # file: /input.zip
- getfattr
顯示擴展屬性的名稱和值
[hadoop@localhost hadoop-2.5.2]$ hadoop fs -getfattr -d /input.zip # file: /input.zip user.web="www.baidu.com" user.web2="www.google.com"
選項-n：顯示指定名稱的屬性值
[hadoop@localhost hadoop-2.5.2]$ hadoop fs -getfattr -n user.web /input.zip# file: /input.zip user.web="www.baidu.com"
>統計相關命令
- count
顯示指定文件或目錄的：DIR_COUNT、FILE_COUNT、CONTENT_SIZE、 FILE_NAME，分別表示：子目錄個數(如果指定路徑是目錄，則包含該目錄本身)、文件個數、使用字節個數，以及文件或目錄名。
[hadoop@localhost hadoop-2.5.2]$ hadoop fs -ls -R / drwxr-xr-x - hadoop supergroup 0 2015-03-27 19:19 /input -rw-r--r-- 1 hadoop supergroup 14 2015-03-27 19:19 /input/input1.txt -rw-r--r-- 1 hadoop supergroup 32 2015-03-27 19:19 /input/input2.txt -rw-r--r-- 1 hadoop supergroup 28 2015-04-02 07:32 /input.txt -rwxrwxrwx 1 hadoop hadoops 28 2015-03-31 08:59 /input1.txt -rw-r--r-- 1 hadoop supergroup 184 2015-03-31 08:14 /input1.zip drwxr-xr-x - hadoop supergroup 0 2015-03-27 19:16 /output drwxr-xr-x - hadoop supergroup 0 2015-04-02 07:29 /text [hadoop@localhost hadoop-2.5.2]$ hadoop fs -count / 4 5 286 /
選項-q：顯示配額信息（在多人共用的情況下，可以通過限制用戶寫入目錄，並設置目錄的quota ，防止不小心就把所有的空間用完造成別人無法存取的情況）。配額信息包括：QUOTA、REMAINING_QUOTA、SPACE_QUOTA、REMAINING_SPACE_QUOTA，分別表示某個目錄下檔案及目錄的總數、剩余目錄或文檔數量、目錄下空間的大小、目錄下剩余空間。
計算公式：
QUOTA – (DIR_COUNT + FILE_COUNT) = REMAINING_QUOTA；
SPACE_QUOTA – CONTENT_SIZE = REMAINING_SPACE_QUOTA。
none和inf表示未配置。
[hadoop@localhost hadoop-2.5.2]$ hadoop fs -count -q / 9223372036854775807 9223372036854775798 none inf 4 5 286 /
- du
顯示文件大小，如果指定目錄，會顯示該目錄中每個文件的大小
[hadoop@localhost hadoop-2.5.2]$ hadoop fs -ls -R / drwxr-xr-x - hadoop supergroup 0 2015-03-27 19:19 /input -rw-r--r-- 1 hadoop supergroup 14 2015-03-27 19:19 /input/input1.txt -rw-r--r-- 1 hadoop supergroup 32 2015-03-27 19:19 /input/input2.txt -rw-r--r-- 1 hadoop supergroup 28 2015-04-02 07:32 /input.txt -rwxrwxrwx 1 hadoop hadoops 28 2015-03-31 08:59 /input1.txt -rw-r--r-- 1 hadoop supergroup 184 2015-03-31 08:14 /input1.zip drwxr-xr-x - hadoop supergroup 0 2015-03-27 19:16 /output drwxr-xr-x - hadoop supergroup 0 2015-04-02 07:29 /text [hadoop@localhost hadoop-2.5.2]$ hadoop fs -du / 46 /input 28 /input.txt 28 /input1.txt 184 /input1.zip 0 /output 0 /text
選項-s：顯示總的統計信息，而不是顯示每個文件的信息
[hadoop@localhost hadoop-2.5.2]$ hadoop fs -du -s / 286 /
- df
檢查文件系統的磁盤空間占用情況
[hadoop@localhost hadoop-2.5.2]$ hadoop fs -df / Filesystem Size Used Available Use% hdfs://localhost:9000 18713219072 73728 8864460800 0%
- stat
顯示文件統計信息。
格式： %b - 文件所占的塊數； %g - 文件所屬的用戶組 ；%n - 文件名； %o - 文件塊大小；%r - 備份數 ；%u - 文件所屬用戶；%y - 文件修改時間
[hadoop@localhost hadoop-2.5.2]$ hadoop fs -stat %b,%g,%n,%o,%r,%u,%y /input.zip 0,supergroup,input.zip,134217728,1,hadoop,2015-04-02 15:43:24
>快照命令
- createSnapshot
創建快照，
附：官方文檔 http://hadoop.apache.org/docs/r2.5.2/hadoop-project-dist/hadoop-hdfs/HdfsSnapshots.html
snapshot(快照)是一個全部文件系統、或者某個目錄在某一時刻的鏡像。創建動作僅僅是在目錄對應的Inode上加個快照的標簽，不會涉及到數據塊的拷貝操作，也不會對讀寫性能有影響，但是會占用namenode一定的額外內存來存放快照中被修改的文件和目錄的元信息
[hadoop@localhost hadoop-2.5.2]$ hadoop fs -ls -R /output -rwxrwxr-x 1 testown testgrp 28 2015-03-31 08:59 /output/input1.txt [hadoop@localhost hadoop-2.5.2]$ hadoop fs -createSnapshot /output s1 createSnapshot: Directory is not a snapshottable directory: /output --直接創建給出錯誤 [hadoop@localhost hadoop-2.5.2]$ hdfs dfsadmin -allowSnapshot /output --對開啟某一目錄的快照功能 Allowing snaphot on /output succeeded [hadoop@localhost hadoop-2.5.2]$ hadoop fs -createSnapshot /output s1 --創建快照 Created snapshot /output/.snapshot/s1 [hadoop@localhost hadoop-2.5.2]$ hadoop fs -ls -R /output -rwxrwxr-x 1 testown testgrp 28 2015-03-31 08:59 /output/input1.txt [hadoop@localhost hadoop-2.5.2]$ hadoop fs -ls /output/.snapshot/s1 Found 1 items -rwxrwxr-x 1 testown testgrp 28 2015-03-31 08:59 /output/.snapshot/s1/input1.txt --查看快照
- renameSnapshot
重命名快照
[hadoop@localhost hadoop-2.5.2]$ hadoop fs -ls /output/.snapshot/s1Found 1 items -rwxrwxr-x 1 testown testgrp 28 2015-03-31 08:59 /output/.snapshot/s1/input1.txt --原快照 [hadoop@localhost hadoop-2.5.2]$ hadoop fs -renameSnapshot /output/ s1 s2[hadoop@localhost hadoop-2.5.2]$ hadoop fs -ls /output/.snapshot/s2 Found 1 items -rwxrwxr-x 1 testown testgrp 28 2015-03-31 08:59 /output/.snapshot/s2/input1.txt --新快照
- deleteSnapshot
刪除快照
[hadoop@localhost hadoop-2.5.2]$ hadoop fs -ls /output/.snapshot/s2 Found 1 items -rwxrwxr-x 1 testown testgrp 28 2015-03-31 08:59 /output/.snapshot/s2/input1.txt [hadoop@localhost hadoop-2.5.2]$ hadoop fs -deleteSnapshot /output/ s2 [hadoop@localhost hadoop-2.5.2]$ hadoop fs -ls /output/.snapshot/s2 ls: `/output/.snapshot/s2': No such file or directory