java - MapReduce output file empty -
i have program prints avg of balance , counts number of customers.everything working fine until noticed part-r-0000 file empty.it strange because haven't changed in hadoop configuration.i post stacktrace of cmd below
17/04/14 14:21:31 info configuration.deprecation: session.id deprecated. instead, use dfs.metrics.session-id 17/04/14 14:21:31 info jvm.jvmmetrics: initializing jvm metrics processname=jobtracker, sessionid= 17/04/14 14:21:31 warn mapreduce.jobresourceuploader: hadoop command-line option parsing not performed. implement tool interface , execute application toolrunner remedy this. 17/04/14 14:21:31 info input.fileinputformat: total input paths process : 1 17/04/14 14:21:31 info mapreduce.jobsubmitter: number of splits:1 17/04/14 14:21:32 info mapreduce.jobsubmitter: submitting tokens job: job_local1656799721_0001 17/04/14 14:21:32 info mapreduce.job: url track job: http://localhost:8080/ 17/04/14 14:21:32 info mapreduce.job: running job: job_local1656799721_0001 17/04/14 14:21:32 info mapred.localjobrunner: outputcommitter set in config null 17/04/14 14:21:32 info output.fileoutputcommitter: file output committer algorithm version 1 17/04/14 14:21:32 info mapred.localjobrunner: outputcommitter org.apache.hadoop.mapreduce.lib.output.fileoutputcommitter 17/04/14 14:21:32 info mapred.localjobrunner: waiting map tasks 17/04/14 14:21:32 info mapred.localjobrunner: starting task: attempt_local1656799721_0001_m_000000_0 17/04/14 14:21:32 info output.fileoutputcommitter: file output committer algorithm version 1 17/04/14 14:21:32 info util.procfsbasedprocesstree: procfsbasedprocesstree supported on linux. 17/04/14 14:21:32 info mapred.task: using resourcecalculatorprocesstree : org.apache.hadoop.yarn.util.windowsbasedprocesstree@7c8cb1b6 17/04/14 14:21:32 info mapred.maptask: processing split: hdfs://localhost:19000/datagen/data/customer.tbl:0+2411114 17/04/14 14:21:32 info mapred.maptask: (equator) 0 kvi 26214396(104857584) 17/04/14 14:21:32 info mapred.maptask: mapreduce.task.io.sort.mb: 100 17/04/14 14:21:32 info mapred.maptask: soft limit @ 83886080 17/04/14 14:21:32 info mapred.maptask: bufstart = 0; bufvoid = 104857600 17/04/14 14:21:32 info mapred.maptask: kvstart = 26214396; length = 6553600 17/04/14 14:21:32 info mapred.maptask: map output collector class = org.apache.hadoop.mapred.maptask$mapoutputbuffer 17/04/14 14:21:32 info mapred.localjobrunner: 17/04/14 14:21:32 info mapred.maptask: starting flush of map output 17/04/14 14:21:32 info mapred.task: task:attempt_local1656799721_0001_m_000000_0 done. , in process of committing 17/04/14 14:21:32 info mapred.localjobrunner: map 17/04/14 14:21:32 info mapred.task: task 'attempt_local1656799721_0001_m_000000_0' done. 17/04/14 14:21:32 info mapred.localjobrunner: finishing task: attempt_local1656799721_0001_m_000000_0 17/04/14 14:21:32 info mapred.localjobrunner: map task executor complete. 17/04/14 14:21:32 info mapred.localjobrunner: waiting reduce tasks 17/04/14 14:21:32 info mapred.localjobrunner: starting task: attempt_local1656799721_0001_r_000000_0 17/04/14 14:21:32 info output.fileoutputcommitter: file output committer algorithm version 1 17/04/14 14:21:32 info util.procfsbasedprocesstree: procfsbasedprocesstree supported on linux. 17/04/14 14:21:32 info mapred.task: using resourcecalculatorprocesstree : org.apache.hadoop.yarn.util.windowsbasedprocesstree@25135c4c 17/04/14 14:21:32 info mapred.reducetask: using shuffleconsumerplugin: org.apache.hadoop.mapreduce.task.reduce.shuffle@2d7e552d 17/04/14 14:21:32 info reduce.mergemanagerimpl: mergermanager: memorylimit=334338464, maxsingleshufflelimit=83584616, mergethreshold=220663392, iosortfactor=10, memtomemmergeoutputsthreshold=10 17/04/14 14:21:32 info reduce.eventfetcher: attempt_local1656799721_0001_r_000000_0 thread started: eventfetcher fetching map completion events 17/04/14 14:21:32 info reduce.localfetcher: localfetcher#1 shuffle output of map attempt_local1656799721_0001_m_000000_0 decomp: 2 len: 6 memory 17/04/14 14:21:32 info reduce.inmemorymapoutput: read 2 bytes map-output attempt_local1656799721_0001_m_000000_0 17/04/14 14:21:32 info reduce.mergemanagerimpl: closeinmemoryfile -> map-output of size: 2, inmemorymapoutputs.size() -> 1, commitmemory -> 0, usedmemory ->2 17/04/14 14:21:32 info reduce.eventfetcher: eventfetcher interrupted.. returning 17/04/14 14:21:32 info mapred.localjobrunner: 1 / 1 copied. 17/04/14 14:21:32 info reduce.mergemanagerimpl: finalmerge called 1 in-memory map-outputs , 0 on-disk map-outputs 17/04/14 14:21:32 info mapred.merger: merging 1 sorted segments 17/04/14 14:21:32 info mapred.merger: down last merge-pass, 0 segments left of total size: 0 bytes 17/04/14 14:21:32 info reduce.mergemanagerimpl: merged 1 segments, 2 bytes disk satisfy reduce memory limit 17/04/14 14:21:32 info reduce.mergemanagerimpl: merging 1 files, 6 bytes disk 17/04/14 14:21:32 info reduce.mergemanagerimpl: merging 0 segments, 0 bytes memory reduce 17/04/14 14:21:32 info mapred.merger: merging 1 sorted segments 17/04/14 14:21:32 info mapred.merger: down last merge-pass, 0 segments left of total size: 0 bytes 17/04/14 14:21:32 info mapred.localjobrunner: 1 / 1 copied. 17/04/14 14:21:32 info configuration.deprecation: mapred.skip.on deprecated. instead, use mapreduce.job.skiprecords 17/04/14 14:21:32 info mapred.task: task:attempt_local1656799721_0001_r_000000_0 done. , in process of committing 17/04/14 14:21:32 info mapred.localjobrunner: 1 / 1 copied. 17/04/14 14:21:32 info mapred.task: task attempt_local1656799721_0001_r_000000_0 allowed commit 17/04/14 14:21:32 info output.fileoutputcommitter: saved output of task 'attempt_local1656799721_0001_r_000000_0' hdfs://localhost:19000/out19/_temporary/0/task_local1656799721_0001_r_000000 17/04/14 14:21:32 info mapred.localjobrunner: reduce > reduce 17/04/14 14:21:32 info mapred.task: task 'attempt_local1656799721_0001_r_000000_0' done. 17/04/14 14:21:32 info mapred.localjobrunner: finishing task: attempt_local1656799721_0001_r_000000_0 17/04/14 14:21:32 info mapred.localjobrunner: reduce task executor complete. 17/04/14 14:21:33 info mapreduce.job: job job_local1656799721_0001 running in uber mode : false 17/04/14 14:21:33 info mapreduce.job: map 100% reduce 100% 17/04/14 14:21:33 info mapreduce.job: job job_local1656799721_0001 completed 17/04/14 14:21:33 info mapreduce.job: counters: 35 file system counters file: number of bytes read=17482 file: number of bytes written=591792 file: number of read operations=0 file: number of large read operations=0 file: number of write operations=0 hdfs: number of bytes read=4822228 hdfs: number of bytes written=0 hdfs: number of read operations=13 hdfs: number of large read operations=0 hdfs: number of write operations=4 map-reduce framework map input records=15000 map output records=0 map output bytes=0 map output materialized bytes=6 input split bytes=113 combine input records=0 combine output records=0 reduce input groups=0 reduce shuffle bytes=6 reduce input records=0 reduce output records=0 spilled records=0 shuffled maps =1 failed shuffles=0 merged map outputs=1 gc time elapsed (ms)=0 total committed heap usage (bytes)=546308096 shuffle errors bad_id=0 connection=0 io_error=0 wrong_length=0 wrong_map=0 wrong_reduce=0 file input format counters bytes read=2411114 file output format counters bytes written=0
code
public static class tokenizermapper extends mapper<longwritable, text,text ,text>{ private text segment = new text(); //private threewritablevalues cust = new threewritablevalues(); private text word = new text(); private float balance = 0; public void map(longwritable key, text value, context context) throws ioexception, interruptedexception { string[] line = value.tostring().split("\\|"); string cust_key = line[1]; int nation = integer.parseint(line[3]); if((balance > 8000) && ( nation < 15) && (nation > 1)){ segment.set(line[6]); word.set(cust_key+","+balance); context.write(segment,word); } } } public static class avgreducer extends reducer<text,text,text,text> { public void reduce(text key, iterable<text> values,context context) throws ioexception, interruptedexception { context.write(key, values.iterator().next()); } } public static void main(string[] args) throws exception { configuration conf = new configuration(); job job = job.getinstance(conf, "word count"); job.setjarbyclass(mapreducetest.class); job.setmapperclass(tokenizermapper.class); job.setcombinerclass(avgreducer.class); job.setreducerclass(avgreducer.class); job.setoutputkeyclass(text.class); job.setoutputvalueclass(text.class); fileinputformat.addinputpath(job, new path(args[0])); fileoutputformat.setoutputpath(job, new path(args[1])); system.exit(job.waitforcompletion(true) ? 0 : 1); } }
please if know something.
there no output generated @ map phase
map output records=0 map output bytes=0
in tokenizermapper
class, value of balance
defined 0
.
private float balance = 0;
and in map
method, value of balance
still 0
checked > 8000
.
if((balance > 8000) && ( nation < 15) && (nation > 1)){ segment.set(line[6]); word.set(cust_key+","+balance); context.write(segment,word); }
the if
condition never met , no mapper output , no reducer output.
Comments
Post a Comment