个人工具
登录
查看“利用Cloudera实现Hadoop”的源代码 - Ubuntu中文
页面
讨论
查看源代码
历史
搜索
导航
首页
最近更改
随机页面
页面分类
帮助
编辑
编辑指南
沙盒
新闻动态
字词处理
工具
链入页面
相关更改
特殊页面
页面信息
查看“利用Cloudera实现Hadoop”的源代码
来自Ubuntu中文
←
利用Cloudera实现Hadoop
跳转至:
导航
,
搜索
因为以下原因,你没有权限编辑本页:
您所请求的操作仅限于该用户组的用户使用:
用户
您可以查看与复制此页面的源代码。
=== 测试Hadoop<br> === Hadoop架设好了,接下来就是要对其进行测试,看看它是否能正常工作,具体代码如下: <pre>hadoop@hadoop-01:~$ hadoop-0.20 fs -mkdir input hadoop@hadoop-01:~$ hadoop-0.20 fs -put /etc/hadoop-0.20/conf/*.xml input hadoop@hadoop-01:~$ hadoop-0.20 fs -ls input Found 6 items -rw-r--r-- 3 hadoop supergroup 3936 2010-03-11 08:55 /user/hadoop/input/capacity-scheduler.xml -rw-r--r-- 3 hadoop supergroup 400 2010-03-11 08:55 /user/hadoop/input/core-site.xml -rw-r--r-- 3 hadoop supergroup 3032 2010-03-11 08:55 /user/hadoop/input/fair-scheduler.xml -rw-r--r-- 3 hadoop supergroup 4190 2010-03-11 08:55 /user/hadoop/input/hadoop-policy.xml -rw-r--r-- 3 hadoop supergroup 536 2010-03-11 08:55 /user/hadoop/input/hdfs-site.xml -rw-r--r-- 3 hadoop supergroup 266 2010-03-11 08:55 /user/hadoop/input/mapred-site.xml hadoop@hadoop-01:~$ hadoop-0.20 jar /usr/lib/hadoop-0.20/hadoop-*-examples.jar grep input output 'dfs[a-z.]+' 10/03/11 14:35:57 INFO mapred.FileInputFormat: Total input paths to process : 6 10/03/11 14:35:58 INFO mapred.JobClient: Running job: job_201003111431_0001 10/03/11 14:35:59 INFO mapred.JobClient: map 0% reduce 0% 10/03/11 14:36:14 INFO mapred.JobClient: map 33% reduce 0% 10/03/11 14:36:20 INFO mapred.JobClient: map 66% reduce 0% 10/03/11 14:36:26 INFO mapred.JobClient: map 66% reduce 22% 10/03/11 14:36:36 INFO mapred.JobClient: map 100% reduce 22% 10/03/11 14:36:44 INFO mapred.JobClient: map 100% reduce 100% 10/03/11 14:36:46 INFO mapred.JobClient: Job complete: job_201003111431_0001 10/03/11 14:36:46 INFO mapred.JobClient: Counters: 19 10/03/11 14:36:46 INFO mapred.JobClient: Job Counters 10/03/11 14:36:46 INFO mapred.JobClient: Launched reduce tasks=1 10/03/11 14:36:46 INFO mapred.JobClient: Rack-local map tasks=4 10/03/11 14:36:46 INFO mapred.JobClient: Launched map tasks=6 10/03/11 14:36:46 INFO mapred.JobClient: Data-local map tasks=2 10/03/11 14:36:46 INFO mapred.JobClient: FileSystemCounters 10/03/11 14:36:46 INFO mapred.JobClient: FILE_BYTES_READ=100 10/03/11 14:36:46 INFO mapred.JobClient: HDFS_BYTES_READ=12360 10/03/11 14:36:46 INFO mapred.JobClient: FILE_BYTES_WRITTEN=422 10/03/11 14:36:46 INFO mapred.JobClient: HDFS_BYTES_WRITTEN=204 10/03/11 14:36:46 INFO mapred.JobClient: Map-Reduce Framework 10/03/11 14:36:46 INFO mapred.JobClient: Reduce input groups=4 10/03/11 14:36:46 INFO mapred.JobClient: Combine output records=4 10/03/11 14:36:46 INFO mapred.JobClient: Map input records=315 10/03/11 14:36:46 INFO mapred.JobClient: Reduce shuffle bytes=124 10/03/11 14:36:46 INFO mapred.JobClient: Reduce output records=4 10/03/11 14:36:46 INFO mapred.JobClient: Spilled Records=8 10/03/11 14:36:46 INFO mapred.JobClient: Map output bytes=86 10/03/11 14:36:46 INFO mapred.JobClient: Map input bytes=12360 10/03/11 14:36:46 INFO mapred.JobClient: Combine input records=4 10/03/11 14:36:46 INFO mapred.JobClient: Map output records=4 10/03/11 14:36:46 INFO mapred.JobClient: Reduce input records=4 10/03/11 14:36:46 WARN mapred.JobClient: Use GenericOptionsParser for parsing the arguments. Applications should implement Tool for the same. 10/03/11 14:36:46 INFO mapred.FileInputFormat: Total input paths to process : 1 10/03/11 14:36:46 INFO mapred.JobClient: Running job: job_201003111431_0002 10/03/11 14:36:47 INFO mapred.JobClient: map 0% reduce 0% 10/03/11 14:36:56 INFO mapred.JobClient: map 100% reduce 0% 10/03/11 14:37:08 INFO mapred.JobClient: map 100% reduce 100% 10/03/11 14:37:10 INFO mapred.JobClient: Job complete: job_201003111431_0002 10/03/11 14:37:11 INFO mapred.JobClient: Counters: 18 10/03/11 14:37:11 INFO mapred.JobClient: Job Counters 10/03/11 14:37:11 INFO mapred.JobClient: Launched reduce tasks=1 10/03/11 14:37:11 INFO mapred.JobClient: Launched map tasks=1 10/03/11 14:37:11 INFO mapred.JobClient: Data-local map tasks=1 10/03/11 14:37:11 INFO mapred.JobClient: FileSystemCounters 10/03/11 14:37:11 INFO mapred.JobClient: FILE_BYTES_READ=100 10/03/11 14:37:11 INFO mapred.JobClient: HDFS_BYTES_READ=204 10/03/11 14:37:11 INFO mapred.JobClient: FILE_BYTES_WRITTEN=232 10/03/11 14:37:11 INFO mapred.JobClient: HDFS_BYTES_WRITTEN=62 10/03/11 14:37:11 INFO mapred.JobClient: Map-Reduce Framework 10/03/11 14:37:11 INFO mapred.JobClient: Reduce input groups=1 10/03/11 14:37:11 INFO mapred.JobClient: Combine output records=0 10/03/11 14:37:11 INFO mapred.JobClient: Map input records=4 10/03/11 14:37:11 INFO mapred.JobClient: Reduce shuffle bytes=0 10/03/11 14:37:11 INFO mapred.JobClient: Reduce output records=4 10/03/11 14:37:11 INFO mapred.JobClient: Spilled Records=8 10/03/11 14:37:11 INFO mapred.JobClient: Map output bytes=86 10/03/11 14:37:11 INFO mapred.JobClient: Map input bytes=118 10/03/11 14:37:11 INFO mapred.JobClient: Combine input records=0 10/03/11 14:37:11 INFO mapred.JobClient: Map output records=4 10/03/11 14:37:11 INFO mapred.JobClient: Reduce input records=4 </pre> 不难看出,上述测试已经成功,这说明Hadoop部署成功,能够在上面进行Map/Reduce分布性计算了。
返回至
利用Cloudera实现Hadoop
。