rocky linux 使用 hadoop

2301_79487934

191人浏览 · 2025-09-10 18:45:46

2301_79487934 · 2025-09-10 18:45:46 发布

export JAVA_HOME=/usr/local/jdk1.7.0_71

使用id命令查看当前用户

root@ubuntu:~# id
uid=0(root) gid=0(root) groups=0(root)

查看文件包含系统中所有用户的信息

root@ubuntu:~# cat /etc/passwd  |cut -d: -f1
root
daemon
bin
sys
sync
games
man
lp
mail
news
uucp
proxy
www-data
backup
list
irc
gnats
nobody
systemd-network
systemd-resolve
systemd-timesync
messagebus
syslog
_apt
tss
uuidd
tcpdump
avahi-autoipd
usbmux
rtkit
dnsmasq
cups-pk-helper
speech-dispatcher
avahi
kernoops
saned
nm-openvpn
hplip
whoopsie
colord
geoclue
pulse
gnome-initial-setup
gdm
sssd
jacklinux
systemd-coredump
sshd
hadoop
root@ubuntu:~#

获取系统中所有用户信息

root@ubuntu:~# getent
getent: wrong number of arguments
Try `getent --help' or `getent --usage' for more information.
root@ubuntu:~# getent passwd
root:x:0:0:root:/root:/bin/bash
daemon:x:1:1:daemon:/usr/sbin:/usr/sbin/nologin
bin:x:2:2:bin:/bin:/usr/sbin/nologin
sys:x:3:3:sys:/dev:/usr/sbin/nologin
sync:x:4:65534:sync:/bin:/bin/sync
games:x:5:60:games:/usr/games:/usr/sbin/nologin
man:x:6:12:man:/var/cache/man:/usr/sbin/nologin
lp:x:7:7:lp:/var/spool/lpd:/usr/sbin/nologin
mail:x:8:8:mail:/var/mail:/usr/sbin/nologin
news:x:9:9:news:/var/spool/news:/usr/sbin/nologin
uucp:x:10:10:uucp:/var/spool/uucp:/usr/sbin/nologin
proxy:x:13:13:proxy:/bin:/usr/sbin/nologin
www-data:x:33:33:www-data:/var/www:/usr/sbin/nologin
backup:x:34:34:backup:/var/backups:/usr/sbin/nologin
list:x:38:38:Mailing List Manager:/var/list:/usr/sbin/nologin
irc:x:39:39:ircd:/var/run/ircd:/usr/sbin/nologin
gnats:x:41:41:Gnats Bug-Reporting System (admin):/var/lib/gnats:/usr/sbin/nologin
nobody:x:65534:65534:nobody:/nonexistent:/usr/sbin/nologin
systemd-network:x:100:102:systemd Network Management,,,:/run/systemd:/usr/sbin/nologin
systemd-resolve:x:101:103:systemd Resolver,,,:/run/systemd:/usr/sbin/nologin
systemd-timesync:x:102:104:systemd Time Synchronization,,,:/run/systemd:/usr/sbin/nologin
messagebus:x:103:106::/nonexistent:/usr/sbin/nologin
syslog:x:104:110::/home/syslog:/usr/sbin/nologin
_apt:x:105:65534::/nonexistent:/usr/sbin/nologin
tss:x:106:111:TPM software stack,,,:/var/lib/tpm:/bin/false
uuidd:x:107:114::/run/uuidd:/usr/sbin/nologin
tcpdump:x:108:115::/nonexistent:/usr/sbin/nologin
avahi-autoipd:x:109:116:Avahi autoip daemon,,,:/var/lib/avahi-autoipd:/usr/sbin/nologin
usbmux:x:110:46:usbmux daemon,,,:/var/lib/usbmux:/usr/sbin/nologin
rtkit:x:111:117:RealtimeKit,,,:/proc:/usr/sbin/nologin
dnsmasq:x:112:65534:dnsmasq,,,:/var/lib/misc:/usr/sbin/nologin
cups-pk-helper:x:113:120:user for cups-pk-helper service,,,:/home/cups-pk-helper:/usr/sbin/nologin
speech-dispatcher:x:114:29:Speech Dispatcher,,,:/run/speech-dispatcher:/bin/false
avahi:x:115:121:Avahi mDNS daemon,,,:/var/run/avahi-daemon:/usr/sbin/nologin
kernoops:x:116:65534:Kernel Oops Tracking Daemon,,,:/:/usr/sbin/nologin
saned:x:117:123::/var/lib/saned:/usr/sbin/nologin
nm-openvpn:x:118:124:NetworkManager OpenVPN,,,:/var/lib/openvpn/chroot:/usr/sbin/nologin
hplip:x:119:7:HPLIP system user,,,:/run/hplip:/bin/false
whoopsie:x:120:125::/nonexistent:/bin/false
colord:x:121:126:colord colour management daemon,,,:/var/lib/colord:/usr/sbin/nologin
geoclue:x:122:127::/var/lib/geoclue:/usr/sbin/nologin
pulse:x:123:128:PulseAudio daemon,,,:/var/run/pulse:/usr/sbin/nologin
gnome-initial-setup:x:124:65534::/run/gnome-initial-setup/:/bin/false
gdm:x:125:130:Gnome Display Manager:/var/lib/gdm3:/bin/false
sssd:x:126:131:SSSD system user,,,:/var/lib/sss:/usr/sbin/nologin
jacklinux:x:1000:1000:ubuntulinux,,,:/home/jacklinux:/bin/bash
systemd-coredump:x:999:999:systemd Core Dumper:/:/usr/sbin/nologin
sshd:x:127:65534::/run/sshd:/usr/sbin/nologin
hadoop:x:1001:1001::/home/hadoop:/bin/sh
root@ubuntu:~#

列出/home 命令，系统中大多数用户都位于/home之下

root@ubuntu:~# ls /home/
jacklinux
root@ubuntu:~#

使用增加用户命令，显示已经有该用户

root@ubuntu:~# sudo useradd -m hadoop -s /bin/bash
useradd: user 'hadoop' already exists
root@ubuntu:~# ls /home/
jacklinux
root@ubuntu:~#

重新设置hadoop的密码

root@ubuntu:~# sudo passwd hadoop
New password:
Retype new password:
passwd: password updated successfully
root@ubuntu:~#

root@ubuntu:~# sudo adduser hadoop sudo
Adding user `hadoop' to group `sudo' ...
Adding user hadoop to group sudo
Done.
root@ubuntu:~#
给hadoop用户增加sudo的权限

root@ubuntu:~# su hadoop
$ ls
ls: cannot open directory '.': Permission denied
$ sudo su 退出刚才的切换用户状态，回到root用户
[sudo] password for hadoop:
root@ubuntu:~# sudo passwd hadoop
New password:更改新的密码，建议设置不同root的密码，否则不知哪个用户
Retype new password:
passwd: password updated successfully
root@ubuntu:~#

再切换用户则会出现以下情况
root@ubuntu:~# su hadoop
$ cd /
$ ls
bin   cdrom  etc     home  lib    lib64   lost+found  mnt    opt   qt5       root  sbin     snap  swapfile  tmp  var
boot  dev    hadoop  jdk   lib32  libx32  media       numpy  proc  qt5where  run   sklearn  srv   sys       usr
$ passwd
Changing password for hadoop.
Current password: 不换密码只是测下
passwd: Authentication token manipulation error
passwd: password unchanged
$ ls
bin   cdrom  etc     home  lib    lib64   lost+found  mnt    opt   qt5       root  sbin     snap  swapfile  tmp  var
boot  dev    hadoop  jdk   lib32  libx32  media       numpy  proc  qt5where  run   sklearn  srv   sys       usr
$

再用sudo su 命令即可切换回root用户，用su hadoop命令来到普通用户状态

安装SSH server

$ sudo apt-get install openssh-server
[sudo] password for hadoop:
Reading package lists... Done
Building dependency tree
Reading state information... Done
openssh-server is already the newest version (1:8.2p1-4ubuntu0.13).
The following packages were automatically installed and are no longer required:
  libcbor0.6 libfido2-1 libfwupdplugin1
Use 'sudo apt autoremove' to remove them.
0 upgraded, 0 newly installed, 0 to remove and 332 not upgraded.
$在hadoop用户环境里

使用 ssh localhost 登录

用命令查看hadoop 普通用户的Java版本

$ java -version
java version "1.8.0_251"
Java(TM) SE Runtime Environment (build 1.8.0_251-b08)
Java HotSpot(TM) 64-Bit Server VM (build 25.251-b08, mixed mode)
$

用xterm 工具将hadoop3.1.3.tar.gz 传到/jdk文件夹里

先在/usr/local查下有啥东西

$ cd /usr/local
$ ls
bin  etc  games  include  lib  man  sbin  share  src

$ sudo tar -zxf hadoop-3.1.3.tar.gz  -C /usr/local
将hdoop压缩包解压到/usr/local里


$ cd hadoop
$ ls
bin  etc  include  lib  libexec  LICENSE.txt  NOTICE.txt  README.txt  sbin  share
$ ./bin/hadoop version  在hadoop3的目录之外不能启动里边的bin/hadoop的应用
./bin/hadoop: line 198: hadoop_abs: command not found
./bin/hadoop: line 207: hadoop_need_reexec: command not found
./bin/hadoop: line 215: hadoop_verify_user_perm: command not found
./bin/hadoop: line 226: hadoop_add_client_opts: command not found
./bin/hadoop: line 233: hadoop_subcommand_opts: command not found
./bin/hadoop: line 236: hadoop_generic_java_subcmd_handler: command not found
$ ls
bin  etc  include  lib  libexec  LICENSE.txt  NOTICE.txt  README.txt  sbin  share
$ cd bin 进到/hadoop3的bin目录
$ ls
container-executor  hadoop  hadoop.cmd  hdfs  hdfs.cmd  mapred  mapred.cmd  test-container-executor  yarn  yarn.cmd
$ hadoop version 直接输入此命令才可以查看hdoop版本
Hadoop 2.4.1
Subversion http://svn.apache.org/repos/asf/hadoop/common -r 1604318
Compiled by jenkins on 2014-06-21T05:43Z
Compiled with protoc 2.5.0
From source with checksum bb7ac0a3c73dc131f4844b873c74b630
This command was run using /hadoop/hadoop2/share/hadoop/common/hadoop-common-2.4.1.jar
$

查看java的安装目录

$ which java
/jdk/jdk1.8.0_251/bin/java
$
在/目录的hadoop3文件夹新建一个java路径设置的sh
export JAVA_HOME=/jdk/jdk1.8.0_251/bin/java
export JRE_HOME=${JAVA_HOME}/jre
export CLASSPATH=.:${JAVA_HOME}/lib:${JRE_HOME}/lib
export PATH=${JAVA_HOME}/bin:$PATH

加上sudo 命令才能新建保存
$ sudo vi javalujingshezhi.sh
$ vi javalujingshezhi.sh

$ cd /hadoop3
$ ls
javalujingshezhi.sh
$ source javalujingshezhi.sh 普通用户里竟然没有source 命令
-sh: 144: source: not found
$ sudo source javalujingshezhi.sh
sudo: source: command not found
$ sh javalujingshezhi.sh  改用sh命令就可以执行此路径设置程序
$

测试hadoop的试验例程

$ ls   在/usr/local/hadoop/bin 路径里 使用下一行命令测试 筛选符合正则表达式的单词
container-executor  hadoop  hadoop.cmd  hdfs  hdfs.cmd  mapred  mapred.cmd  test-container-executor  yarn  yarn.cmd  
$ hadoop jar /usr/local/hadoop/share/hadoop/mapreduce/hadoop-mapreduce-examples-3.1.3.jar grep ./input  ./output 'dfs[a-z.]+'
Java HotSpot(TM) 64-Bit Server VM warning: You have loaded library /hadoop/hadoop2/lib/native/libhadoop.so.1.0.0 which might have disabled stack guard. The VM will try to fix the stack guard now.
It's highly recommended that you fix the library with 'execstack -c <libfile>', or link it with '-z noexecstack'.
25/09/10 03:33:04 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
25/09/10 03:33:05 INFO Configuration.deprecation: session.id is deprecated. Instead, use dfs.metrics.session-id
25/09/10 03:33:05 INFO jvm.JvmMetrics: Initializing JVM Metrics with processName=JobTracker, sessionId=
25/09/10 03:33:06 WARN mapreduce.JobSubmitter: No job jar file set.  User classes may not be found. See Job or Job#setJar(String).
25/09/10 03:33:06 INFO input.FileInputFormat: Total input paths to process : 9
25/09/10 03:33:06 INFO mapreduce.JobSubmitter: number of splits:9
25/09/10 03:33:06 INFO mapreduce.JobSubmitter: Submitting tokens for job: job_local762208783_0001
25/09/10 03:33:06 WARN conf.Configuration: file:/tmp/hadoop-hadoop/mapred/staging/hadoop762208783/.staging/job_local762208783_0001/job.xml:an attempt to override final parameter: mapreduce.job.end-notification.max.retry.interval;  Ignoring.
25/09/10 03:33:06 WARN conf.Configuration: file:/tmp/hadoop-hadoop/mapred/staging/hadoop762208783/.staging/job_local762208783_0001/job.xml:an attempt to override final parameter: mapreduce.job.end-notification.max.attempts;  Ignoring.
25/09/10 03:33:07 WARN conf.Configuration: file:/tmp/hadoop-hadoop/mapred/local/localRunner/hadoop/job_local762208783_0001/job_local762208783_0001.xml:an attempt to override final parameter: mapreduce.job.end-notification.max.retry.interval;  Ignoring.
25/09/10 03:33:07 WARN conf.Configuration: file:/tmp/hadoop-hadoop/mapred/local/localRunner/hadoop/job_local762208783_0001/job_local762208783_0001.xml:an attempt to override final parameter: mapreduce.job.end-notification.max.attempts;  Ignoring.
25/09/10 03:33:07 INFO mapreduce.Job: The url to track the job: http://localhost:8080/
25/09/10 03:33:07 INFO mapreduce.Job: Running job: job_local762208783_0001
25/09/10 03:33:07 INFO mapred.LocalJobRunner: OutputCommitter set in config null
25/09/10 03:33:07 INFO mapred.LocalJobRunner: OutputCommitter is org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter
25/09/10 03:33:07 INFO mapred.LocalJobRunner: Waiting for map tasks
25/09/10 03:33:07 INFO mapred.LocalJobRunner: Starting task: attempt_local762208783_0001_m_000000_0
25/09/10 03:33:07 INFO mapred.Task:  Using ResourceCalculatorProcessTree : [ ]
25/09/10 03:33:07 INFO mapred.MapTask: Processing split: file:/usr/local/hadoop/bin/input/hadoop-policy.xml:0+11392
25/09/10 03:33:07 INFO mapred.MapTask: Map output collector class = org.apache.hadoop.mapred.MapTask$MapOutputBuffer
25/09/10 03:33:07 INFO mapred.MapTask: (EQUATOR) 0 kvi 26214396(104857584)
25/09/10 03:33:07 INFO mapred.MapTask: mapreduce.task.io.sort.mb: 100
25/09/10 03:33:07 INFO mapred.MapTask: soft limit at 83886080
25/09/10 03:33:07 INFO mapred.MapTask: bufstart = 0; bufvoid = 104857600
25/09/10 03:33:07 INFO mapred.MapTask: kvstart = 26214396; length = 6553600
25/09/10 03:33:07 INFO mapred.LocalJobRunner:
25/09/10 03:33:07 INFO mapred.MapTask: Starting flush of map output
25/09/10 03:33:07 INFO mapred.MapTask: Spilling map output
25/09/10 03:33:07 INFO mapred.MapTask: bufstart = 0; bufend = 17; bufvoid = 104857600
25/09/10 03:33:07 INFO mapred.MapTask: kvstart = 26214396(104857584); kvend = 26214396(104857584); length = 1/6553600
25/09/10 03:33:07 INFO mapred.MapTask: Finished spill 0
25/09/10 03:33:07 INFO mapred.Task: Task:attempt_local762208783_0001_m_000000_0 is done. And is in the process of committing
25/09/10 03:33:07 INFO mapred.LocalJobRunner: map
25/09/10 03:33:07 INFO mapred.Task: Task 'attempt_local762208783_0001_m_000000_0' done.
25/09/10 03:33:07 INFO mapred.LocalJobRunner: Finishing task: attempt_local762208783_0001_m_000000_0
25/09/10 03:33:07 INFO mapred.LocalJobRunner: Starting task: attempt_local762208783_0001_m_000001_0
25/09/10 03:33:07 INFO mapred.Task:  Using ResourceCalculatorProcessTree : [ ]
25/09/10 03:33:07 INFO mapred.MapTask: Processing split: file:/usr/local/hadoop/bin/input/capacity-scheduler.xml:0+8260
25/09/10 03:33:07 INFO mapred.MapTask: Map output collector class = org.apache.hadoop.mapred.MapTask$MapOutputBuffer
25/09/10 03:33:07 INFO mapred.MapTask: (EQUATOR) 0 kvi 26214396(104857584)
25/09/10 03:33:07 INFO mapred.MapTask: mapreduce.task.io.sort.mb: 100
25/09/10 03:33:07 INFO mapred.MapTask: soft limit at 83886080
25/09/10 03:33:07 INFO mapred.MapTask: bufstart = 0; bufvoid = 104857600
25/09/10 03:33:07 INFO mapred.MapTask: kvstart = 26214396; length = 6553600
25/09/10 03:33:07 INFO mapred.LocalJobRunner:
25/09/10 03:33:07 INFO mapred.MapTask: Starting flush of map output
25/09/10 03:33:07 INFO mapred.Task: Task:attempt_local762208783_0001_m_000001_0 is done. And is in the process of committing
25/09/10 03:33:07 INFO mapred.LocalJobRunner: map
25/09/10 03:33:07 INFO mapred.Task: Task 'attempt_local762208783_0001_m_000001_0' done.
25/09/10 03:33:07 INFO mapred.LocalJobRunner: Finishing task: attempt_local762208783_0001_m_000001_0
25/09/10 03:33:07 INFO mapred.LocalJobRunner: Starting task: attempt_local762208783_0001_m_000002_0
25/09/10 03:33:07 INFO mapred.Task:  Using ResourceCalculatorProcessTree : [ ]
25/09/10 03:33:07 INFO mapred.MapTask: Processing split: file:/usr/local/hadoop/bin/input/kms-acls.xml:0+3518
25/09/10 03:33:07 INFO mapred.MapTask: Map output collector class = org.apache.hadoop.mapred.MapTask$MapOutputBuffer
25/09/10 03:33:07 INFO mapred.MapTask: (EQUATOR) 0 kvi 26214396(104857584)
25/09/10 03:33:07 INFO mapred.MapTask: mapreduce.task.io.sort.mb: 100
25/09/10 03:33:07 INFO mapred.MapTask: soft limit at 83886080
25/09/10 03:33:07 INFO mapred.MapTask: bufstart = 0; bufvoid = 104857600
25/09/10 03:33:07 INFO mapred.MapTask: kvstart = 26214396; length = 6553600
25/09/10 03:33:07 INFO mapred.LocalJobRunner:
25/09/10 03:33:07 INFO mapred.MapTask: Starting flush of map output
25/09/10 03:33:07 INFO mapred.Task: Task:attempt_local762208783_0001_m_000002_0 is done. And is in the process of committing
25/09/10 03:33:07 INFO mapred.LocalJobRunner: map
25/09/10 03:33:07 INFO mapred.Task: Task 'attempt_local762208783_0001_m_000002_0' done.
25/09/10 03:33:07 INFO mapred.LocalJobRunner: Finishing task: attempt_local762208783_0001_m_000002_0
25/09/10 03:33:07 INFO mapred.LocalJobRunner: Starting task: attempt_local762208783_0001_m_000003_0
25/09/10 03:33:07 INFO mapred.Task:  Using ResourceCalculatorProcessTree : [ ]
25/09/10 03:33:07 INFO mapred.MapTask: Processing split: file:/usr/local/hadoop/bin/input/hdfs-site.xml:0+775
25/09/10 03:33:07 INFO mapred.MapTask: Map output collector class = org.apache.hadoop.mapred.MapTask$MapOutputBuffer
25/09/10 03:33:07 INFO mapred.MapTask: (EQUATOR) 0 kvi 26214396(104857584)
25/09/10 03:33:07 INFO mapred.MapTask: mapreduce.task.io.sort.mb: 100
25/09/10 03:33:07 INFO mapred.MapTask: soft limit at 83886080
25/09/10 03:33:07 INFO mapred.MapTask: bufstart = 0; bufvoid = 104857600
25/09/10 03:33:07 INFO mapred.MapTask: kvstart = 26214396; length = 6553600
25/09/10 03:33:07 INFO mapred.LocalJobRunner:
25/09/10 03:33:07 INFO mapred.MapTask: Starting flush of map output
25/09/10 03:33:07 INFO mapred.Task: Task:attempt_local762208783_0001_m_000003_0 is done. And is in the process of committing
25/09/10 03:33:08 INFO mapred.LocalJobRunner: map
25/09/10 03:33:08 INFO mapred.Task: Task 'attempt_local762208783_0001_m_000003_0' done.
25/09/10 03:33:08 INFO mapred.LocalJobRunner: Finishing task: attempt_local762208783_0001_m_000003_0
25/09/10 03:33:08 INFO mapred.LocalJobRunner: Starting task: attempt_local762208783_0001_m_000004_0
25/09/10 03:33:08 INFO mapred.Task:  Using ResourceCalculatorProcessTree : [ ]
25/09/10 03:33:08 INFO mapred.MapTask: Processing split: file:/usr/local/hadoop/bin/input/core-site.xml:0+774
25/09/10 03:33:08 INFO mapred.MapTask: Map output collector class = org.apache.hadoop.mapred.MapTask$MapOutputBuffer
25/09/10 03:33:08 INFO mapreduce.Job: Job job_local762208783_0001 running in uber mode : false
25/09/10 03:33:08 INFO mapreduce.Job:  map 100% reduce 0%
25/09/10 03:33:08 INFO mapred.MapTask: (EQUATOR) 0 kvi 26214396(104857584)
25/09/10 03:33:08 INFO mapred.MapTask: mapreduce.task.io.sort.mb: 100
25/09/10 03:33:08 INFO mapred.MapTask: soft limit at 83886080
25/09/10 03:33:08 INFO mapred.MapTask: bufstart = 0; bufvoid = 104857600
25/09/10 03:33:08 INFO mapred.MapTask: kvstart = 26214396; length = 6553600
25/09/10 03:33:08 INFO mapred.LocalJobRunner:
25/09/10 03:33:08 INFO mapred.MapTask: Starting flush of map output
25/09/10 03:33:08 INFO mapred.Task: Task:attempt_local762208783_0001_m_000004_0 is done. And is in the process of committing
25/09/10 03:33:08 INFO mapred.LocalJobRunner: map
25/09/10 03:33:08 INFO mapred.Task: Task 'attempt_local762208783_0001_m_000004_0' done.
25/09/10 03:33:08 INFO mapred.LocalJobRunner: Finishing task: attempt_local762208783_0001_m_000004_0
25/09/10 03:33:08 INFO mapred.LocalJobRunner: Starting task: attempt_local762208783_0001_m_000005_0
25/09/10 03:33:08 INFO mapred.Task:  Using ResourceCalculatorProcessTree : [ ]
25/09/10 03:33:08 INFO mapred.MapTask: Processing split: file:/usr/local/hadoop/bin/input/mapred-site.xml:0+758
25/09/10 03:33:08 INFO mapred.MapTask: Map output collector class = org.apache.hadoop.mapred.MapTask$MapOutputBuffer
25/09/10 03:33:08 INFO mapred.MapTask: (EQUATOR) 0 kvi 26214396(104857584)
25/09/10 03:33:08 INFO mapred.MapTask: mapreduce.task.io.sort.mb: 100
25/09/10 03:33:08 INFO mapred.MapTask: soft limit at 83886080
25/09/10 03:33:08 INFO mapred.MapTask: bufstart = 0; bufvoid = 104857600
25/09/10 03:33:08 INFO mapred.MapTask: kvstart = 26214396; length = 6553600
25/09/10 03:33:08 INFO mapred.LocalJobRunner:
25/09/10 03:33:08 INFO mapred.MapTask: Starting flush of map output
25/09/10 03:33:08 INFO mapred.Task: Task:attempt_local762208783_0001_m_000005_0 is done. And is in the process of committing
25/09/10 03:33:08 INFO mapred.LocalJobRunner: map
25/09/10 03:33:08 INFO mapred.Task: Task 'attempt_local762208783_0001_m_000005_0' done.
25/09/10 03:33:08 INFO mapred.LocalJobRunner: Finishing task: attempt_local762208783_0001_m_000005_0
25/09/10 03:33:08 INFO mapred.LocalJobRunner: Starting task: attempt_local762208783_0001_m_000006_0
25/09/10 03:33:08 INFO mapred.Task:  Using ResourceCalculatorProcessTree : [ ]
25/09/10 03:33:08 INFO mapred.MapTask: Processing split: file:/usr/local/hadoop/bin/input/yarn-site.xml:0+690
25/09/10 03:33:08 INFO mapred.MapTask: Map output collector class = org.apache.hadoop.mapred.MapTask$MapOutputBuffer
25/09/10 03:33:08 INFO mapred.MapTask: (EQUATOR) 0 kvi 26214396(104857584)
25/09/10 03:33:08 INFO mapred.MapTask: mapreduce.task.io.sort.mb: 100
25/09/10 03:33:08 INFO mapred.MapTask: soft limit at 83886080
25/09/10 03:33:08 INFO mapred.MapTask: bufstart = 0; bufvoid = 104857600
25/09/10 03:33:08 INFO mapred.MapTask: kvstart = 26214396; length = 6553600
25/09/10 03:33:08 INFO mapred.LocalJobRunner:
25/09/10 03:33:08 INFO mapred.MapTask: Starting flush of map output
25/09/10 03:33:08 INFO mapred.Task: Task:attempt_local762208783_0001_m_000006_0 is done. And is in the process of committing
25/09/10 03:33:08 INFO mapred.LocalJobRunner: map
25/09/10 03:33:08 INFO mapred.Task: Task 'attempt_local762208783_0001_m_000006_0' done.
25/09/10 03:33:08 INFO mapred.LocalJobRunner: Finishing task: attempt_local762208783_0001_m_000006_0
25/09/10 03:33:08 INFO mapred.LocalJobRunner: Starting task: attempt_local762208783_0001_m_000007_0
25/09/10 03:33:08 INFO mapred.Task:  Using ResourceCalculatorProcessTree : [ ]
25/09/10 03:33:08 INFO mapred.MapTask: Processing split: file:/usr/local/hadoop/bin/input/kms-site.xml:0+682
25/09/10 03:33:08 INFO mapred.MapTask: Map output collector class = org.apache.hadoop.mapred.MapTask$MapOutputBuffer
25/09/10 03:33:08 INFO mapred.MapTask: (EQUATOR) 0 kvi 26214396(104857584)
25/09/10 03:33:08 INFO mapred.MapTask: mapreduce.task.io.sort.mb: 100
25/09/10 03:33:08 INFO mapred.MapTask: soft limit at 83886080
25/09/10 03:33:08 INFO mapred.MapTask: bufstart = 0; bufvoid = 104857600
25/09/10 03:33:08 INFO mapred.MapTask: kvstart = 26214396; length = 6553600
25/09/10 03:33:08 INFO mapred.LocalJobRunner:
25/09/10 03:33:08 INFO mapred.MapTask: Starting flush of map output
25/09/10 03:33:08 INFO mapred.Task: Task:attempt_local762208783_0001_m_000007_0 is done. And is in the process of committing
25/09/10 03:33:08 INFO mapred.LocalJobRunner: map
25/09/10 03:33:08 INFO mapred.Task: Task 'attempt_local762208783_0001_m_000007_0' done.
25/09/10 03:33:08 INFO mapred.LocalJobRunner: Finishing task: attempt_local762208783_0001_m_000007_0
25/09/10 03:33:08 INFO mapred.LocalJobRunner: Starting task: attempt_local762208783_0001_m_000008_0
25/09/10 03:33:08 INFO mapred.Task:  Using ResourceCalculatorProcessTree : [ ]
25/09/10 03:33:08 INFO mapred.MapTask: Processing split: file:/usr/local/hadoop/bin/input/httpfs-site.xml:0+620
25/09/10 03:33:08 INFO mapred.MapTask: Map output collector class = org.apache.hadoop.mapred.MapTask$MapOutputBuffer
25/09/10 03:33:08 INFO mapred.MapTask: (EQUATOR) 0 kvi 26214396(104857584)
25/09/10 03:33:08 INFO mapred.MapTask: mapreduce.task.io.sort.mb: 100
25/09/10 03:33:08 INFO mapred.MapTask: soft limit at 83886080
25/09/10 03:33:08 INFO mapred.MapTask: bufstart = 0; bufvoid = 104857600
25/09/10 03:33:08 INFO mapred.MapTask: kvstart = 26214396; length = 6553600
25/09/10 03:33:08 INFO mapred.LocalJobRunner:
25/09/10 03:33:08 INFO mapred.MapTask: Starting flush of map output
25/09/10 03:33:08 INFO mapred.Task: Task:attempt_local762208783_0001_m_000008_0 is done. And is in the process of committing
25/09/10 03:33:08 INFO mapred.LocalJobRunner: map
25/09/10 03:33:08 INFO mapred.Task: Task 'attempt_local762208783_0001_m_000008_0' done.
25/09/10 03:33:08 INFO mapred.LocalJobRunner: Finishing task: attempt_local762208783_0001_m_000008_0
25/09/10 03:33:08 INFO mapred.LocalJobRunner: map task executor complete.
25/09/10 03:33:08 INFO mapred.LocalJobRunner: Waiting for reduce tasks
25/09/10 03:33:08 INFO mapred.LocalJobRunner: Starting task: attempt_local762208783_0001_r_000000_0
25/09/10 03:33:08 INFO mapred.Task:  Using ResourceCalculatorProcessTree : [ ]
25/09/10 03:33:08 INFO mapred.ReduceTask: Using ShuffleConsumerPlugin: org.apache.hadoop.mapreduce.task.reduce.Shuffle@3e60b7f8
25/09/10 03:33:08 INFO reduce.MergeManagerImpl: MergerManager: memoryLimit=334338464, maxSingleShuffleLimit=83584616, mergeThreshold=220663392, ioSortFactor=10, memToMemMergeOutputsThreshold=10
25/09/10 03:33:08 INFO reduce.EventFetcher: attempt_local762208783_0001_r_000000_0 Thread started: EventFetcher for fetching Map Completion Events
25/09/10 03:33:08 INFO reduce.LocalFetcher: localfetcher#1 about to shuffle output of map attempt_local762208783_0001_m_000000_0 decomp: 21 len: 25 to MEMORY
25/09/10 03:33:08 INFO reduce.InMemoryMapOutput: Read 21 bytes from map-output for attempt_local762208783_0001_m_000000_0
25/09/10 03:33:08 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 21, inMemoryMapOutputs.size() -> 1, commitMemory -> 0, usedMemory ->21
25/09/10 03:33:08 INFO reduce.LocalFetcher: localfetcher#1 about to shuffle output of map attempt_local762208783_0001_m_000003_0 decomp: 2 len: 6 to MEMORY
25/09/10 03:33:08 INFO reduce.InMemoryMapOutput: Read 2 bytes from map-output for attempt_local762208783_0001_m_000003_0
25/09/10 03:33:08 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 2, inMemoryMapOutputs.size() -> 2, commitMemory -> 21, usedMemory ->23
25/09/10 03:33:08 INFO reduce.LocalFetcher: localfetcher#1 about to shuffle output of map attempt_local762208783_0001_m_000006_0 decomp: 2 len: 6 to MEMORY
25/09/10 03:33:08 INFO reduce.InMemoryMapOutput: Read 2 bytes from map-output for attempt_local762208783_0001_m_000006_0
25/09/10 03:33:08 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 2, inMemoryMapOutputs.size() -> 3, commitMemory -> 23, usedMemory ->25
25/09/10 03:33:08 INFO reduce.LocalFetcher: localfetcher#1 about to shuffle output of map attempt_local762208783_0001_m_000002_0 decomp: 2 len: 6 to MEMORY
25/09/10 03:33:08 INFO reduce.InMemoryMapOutput: Read 2 bytes from map-output for attempt_local762208783_0001_m_000002_0
25/09/10 03:33:08 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 2, inMemoryMapOutputs.size() -> 4, commitMemory -> 25, usedMemory ->27
25/09/10 03:33:08 INFO reduce.LocalFetcher: localfetcher#1 about to shuffle output of map attempt_local762208783_0001_m_000005_0 decomp: 2 len: 6 to MEMORY
25/09/10 03:33:08 INFO reduce.InMemoryMapOutput: Read 2 bytes from map-output for attempt_local762208783_0001_m_000005_0
25/09/10 03:33:08 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 2, inMemoryMapOutputs.size() -> 5, commitMemory -> 27, usedMemory ->29
25/09/10 03:33:08 INFO reduce.LocalFetcher: localfetcher#1 about to shuffle output of map attempt_local762208783_0001_m_000008_0 decomp: 2 len: 6 to MEMORY
25/09/10 03:33:08 INFO reduce.InMemoryMapOutput: Read 2 bytes from map-output for attempt_local762208783_0001_m_000008_0
25/09/10 03:33:08 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 2, inMemoryMapOutputs.size() -> 6, commitMemory -> 29, usedMemory ->31
25/09/10 03:33:08 INFO reduce.LocalFetcher: localfetcher#1 about to shuffle output of map attempt_local762208783_0001_m_000001_0 decomp: 2 len: 6 to MEMORY
25/09/10 03:33:08 INFO reduce.InMemoryMapOutput: Read 2 bytes from map-output for attempt_local762208783_0001_m_000001_0
25/09/10 03:33:08 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 2, inMemoryMapOutputs.size() -> 7, commitMemory -> 31, usedMemory ->33
25/09/10 03:33:08 INFO reduce.LocalFetcher: localfetcher#1 about to shuffle output of map attempt_local762208783_0001_m_000004_0 decomp: 2 len: 6 to MEMORY
25/09/10 03:33:08 INFO reduce.InMemoryMapOutput: Read 2 bytes from map-output for attempt_local762208783_0001_m_000004_0
25/09/10 03:33:08 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 2, inMemoryMapOutputs.size() -> 8, commitMemory -> 33, usedMemory ->35
25/09/10 03:33:08 INFO reduce.LocalFetcher: localfetcher#1 about to shuffle output of map attempt_local762208783_0001_m_000007_0 decomp: 2 len: 6 to MEMORY
25/09/10 03:33:08 INFO reduce.InMemoryMapOutput: Read 2 bytes from map-output for attempt_local762208783_0001_m_000007_0
25/09/10 03:33:08 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 2, inMemoryMapOutputs.size() -> 9, commitMemory -> 35, usedMemory ->37
25/09/10 03:33:08 INFO reduce.EventFetcher: EventFetcher is interrupted.. Returning
25/09/10 03:33:08 INFO mapred.LocalJobRunner: 9 / 9 copied.
25/09/10 03:33:08 INFO reduce.MergeManagerImpl: finalMerge called with 9 in-memory map-outputs and 0 on-disk map-outputs
25/09/10 03:33:08 INFO mapred.Merger: Merging 9 sorted segments
25/09/10 03:33:08 INFO mapred.Merger: Down to the last merge-pass, with 1 segments left of total size: 10 bytes
25/09/10 03:33:08 INFO reduce.MergeManagerImpl: Merged 9 segments, 37 bytes to disk to satisfy reduce memory limit
25/09/10 03:33:08 INFO reduce.MergeManagerImpl: Merging 1 files, 25 bytes from disk
25/09/10 03:33:08 INFO reduce.MergeManagerImpl: Merging 0 segments, 0 bytes from memory into reduce
25/09/10 03:33:08 INFO mapred.Merger: Merging 1 sorted segments
25/09/10 03:33:08 INFO mapred.Merger: Down to the last merge-pass, with 1 segments left of total size: 10 bytes
25/09/10 03:33:08 INFO mapred.LocalJobRunner: 9 / 9 copied.
25/09/10 03:33:08 INFO Configuration.deprecation: mapred.skip.on is deprecated. Instead, use mapreduce.job.skiprecords
25/09/10 03:33:08 INFO mapred.Task: Task:attempt_local762208783_0001_r_000000_0 is done. And is in the process of committing
25/09/10 03:33:08 INFO mapred.LocalJobRunner: 9 / 9 copied.
25/09/10 03:33:08 INFO mapred.Task: Task attempt_local762208783_0001_r_000000_0 is allowed to commit now
25/09/10 03:33:08 INFO output.FileOutputCommitter: Saved output of task 'attempt_local762208783_0001_r_000000_0' to file:/usr/local/hadoop/bin/grep-temp-1525921240/_temporary/0/task_local762208783_0001_r_000000
25/09/10 03:33:08 INFO mapred.LocalJobRunner: reduce > reduce
25/09/10 03:33:08 INFO mapred.Task: Task 'attempt_local762208783_0001_r_000000_0' done.
25/09/10 03:33:08 INFO mapred.LocalJobRunner: Finishing task: attempt_local762208783_0001_r_000000_0
25/09/10 03:33:08 INFO mapred.LocalJobRunner: reduce task executor complete.
25/09/10 03:33:09 INFO mapreduce.Job:  map 100% reduce 100%
25/09/10 03:33:09 INFO mapreduce.Job: Job job_local762208783_0001 completed successfully
25/09/10 03:33:09 INFO mapreduce.Job: Counters: 33
        File System Counters
                FILE: Number of bytes read=287225
                FILE: Number of bytes written=2224540
                FILE: Number of read operations=0
                FILE: Number of large read operations=0
                FILE: Number of write operations=0
        Map-Reduce Framework
                Map input records=738
                Map output records=1
                Map output bytes=17
                Map output materialized bytes=73
                Input split bytes=1014
                Combine input records=1
                Combine output records=1
                Reduce input groups=1
                Reduce shuffle bytes=73
                Reduce input records=1
                Reduce output records=1
                Spilled Records=2
                Shuffled Maps =9
                Failed Shuffles=0
                Merged Map outputs=9
                GC time elapsed (ms)=138
                CPU time spent (ms)=0
                Physical memory (bytes) snapshot=0
                Virtual memory (bytes) snapshot=0
                Total committed heap usage (bytes)=3508011008
        Shuffle Errors
                BAD_ID=0
                CONNECTION=0
                IO_ERROR=0
                WRONG_LENGTH=0
                WRONG_MAP=0
                WRONG_REDUCE=0
        File Input Format Counters
                Bytes Read=27469
        File Output Format Counters
                Bytes Written=123
25/09/10 03:33:09 INFO jvm.JvmMetrics: Cannot initialize JVM Metrics with processName=JobTracker, sessionId= - already initialized
org.apache.hadoop.mapred.FileAlreadyExistsException: Output directory file:/usr/local/hadoop/bin/output already exists
        at org.apache.hadoop.mapreduce.lib.output.FileOutputFormat.checkOutputSpecs(FileOutputFormat.java:146)
        at org.apache.hadoop.mapreduce.JobSubmitter.checkSpecs(JobSubmitter.java:458)
        at org.apache.hadoop.mapreduce.JobSubmitter.submitJobInternal(JobSubmitter.java:343)
        at org.apache.hadoop.mapreduce.Job$10.run(Job.java:1285)
        at org.apache.hadoop.mapreduce.Job$10.run(Job.java:1282)
        at java.security.AccessController.doPrivileged(Native Method)
        at javax.security.auth.Subject.doAs(Subject.java:422)
        at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1556)
        at org.apache.hadoop.mapreduce.Job.submit(Job.java:1282)
        at org.apache.hadoop.mapreduce.Job.waitForCompletion(Job.java:1303)
        at org.apache.hadoop.examples.Grep.run(Grep.java:92)
        at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
        at org.apache.hadoop.examples.Grep.main(Grep.java:101)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at org.apache.hadoop.util.ProgramDriver$ProgramDescription.invoke(ProgramDriver.java:72)
        at org.apache.hadoop.util.ProgramDriver.run(ProgramDriver.java:145)
        at org.apache.hadoop.examples.ExampleDriver.main(ExampleDriver.java:74)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at org.apache.hadoop.util.RunJar.main(RunJar.java:212)
$

码道开发者社区

码道开发者社区，聚焦华为云码道 CodeArts 代码智能体，沉淀 Agent、Skill、鸿蒙开发实战内容，供开发者查阅资料、交流技术、分享工程实践

更多推荐

cover

不想先学一堆 React？用 Codex 帮你完成第一支 Remotion 动效视频

码道开发者社区

cover

突发！英伟达限制员工使用Claude；程序员一条代码删掉89TB数据获刑：被判五年十个月；曝iPhone Duo有望支持微信双开 | 极客头条

码道开发者社区

cover

每日热评｜学术牛马的救星还是幻觉放大器？47k星学术研究Agent技能包深度评测

码道开发者社区

所有评论(0)

查看更多评论

2301_79487934

已为社区贡献1条内容