java - MapReduce output file empty -


i have program prints avg of balance , counts number of customers.everything working fine until noticed part-r-0000 file empty.it strange because haven't changed in hadoop configuration.i post stacktrace of cmd below

17/04/14 14:21:31 info configuration.deprecation: session.id deprecated. instead, use dfs.metrics.session-id 17/04/14 14:21:31 info jvm.jvmmetrics: initializing jvm metrics processname=jobtracker, sessionid= 17/04/14 14:21:31 warn mapreduce.jobresourceuploader: hadoop command-line option parsing not performed. implement tool interface , execute application toolrunner remedy this. 17/04/14 14:21:31 info input.fileinputformat: total input paths process : 1 17/04/14 14:21:31 info mapreduce.jobsubmitter: number of splits:1 17/04/14 14:21:32 info mapreduce.jobsubmitter: submitting tokens job: job_local1656799721_0001 17/04/14 14:21:32 info mapreduce.job: url track job: http://localhost:8080/ 17/04/14 14:21:32 info mapreduce.job: running job: job_local1656799721_0001 17/04/14 14:21:32 info mapred.localjobrunner: outputcommitter set in config null 17/04/14 14:21:32 info output.fileoutputcommitter: file output committer algorithm version 1 17/04/14 14:21:32 info mapred.localjobrunner: outputcommitter org.apache.hadoop.mapreduce.lib.output.fileoutputcommitter 17/04/14 14:21:32 info mapred.localjobrunner: waiting map tasks 17/04/14 14:21:32 info mapred.localjobrunner: starting task: attempt_local1656799721_0001_m_000000_0 17/04/14 14:21:32 info output.fileoutputcommitter: file output committer algorithm version 1 17/04/14 14:21:32 info util.procfsbasedprocesstree: procfsbasedprocesstree supported on linux. 17/04/14 14:21:32 info mapred.task:  using resourcecalculatorprocesstree : org.apache.hadoop.yarn.util.windowsbasedprocesstree@7c8cb1b6 17/04/14 14:21:32 info mapred.maptask: processing split: hdfs://localhost:19000/datagen/data/customer.tbl:0+2411114 17/04/14 14:21:32 info mapred.maptask: (equator) 0 kvi 26214396(104857584) 17/04/14 14:21:32 info mapred.maptask: mapreduce.task.io.sort.mb: 100 17/04/14 14:21:32 info mapred.maptask: soft limit @ 83886080 17/04/14 14:21:32 info mapred.maptask: bufstart = 0; bufvoid = 104857600 17/04/14 14:21:32 info mapred.maptask: kvstart = 26214396; length = 6553600 17/04/14 14:21:32 info mapred.maptask: map output collector class = org.apache.hadoop.mapred.maptask$mapoutputbuffer 17/04/14 14:21:32 info mapred.localjobrunner: 17/04/14 14:21:32 info mapred.maptask: starting flush of map output 17/04/14 14:21:32 info mapred.task: task:attempt_local1656799721_0001_m_000000_0 done. , in process of committing 17/04/14 14:21:32 info mapred.localjobrunner: map 17/04/14 14:21:32 info mapred.task: task 'attempt_local1656799721_0001_m_000000_0' done. 17/04/14 14:21:32 info mapred.localjobrunner: finishing task: attempt_local1656799721_0001_m_000000_0 17/04/14 14:21:32 info mapred.localjobrunner: map task executor complete. 17/04/14 14:21:32 info mapred.localjobrunner: waiting reduce tasks 17/04/14 14:21:32 info mapred.localjobrunner: starting task: attempt_local1656799721_0001_r_000000_0 17/04/14 14:21:32 info output.fileoutputcommitter: file output committer algorithm version 1 17/04/14 14:21:32 info util.procfsbasedprocesstree: procfsbasedprocesstree supported on linux. 17/04/14 14:21:32 info mapred.task:  using resourcecalculatorprocesstree : org.apache.hadoop.yarn.util.windowsbasedprocesstree@25135c4c 17/04/14 14:21:32 info mapred.reducetask: using shuffleconsumerplugin: org.apache.hadoop.mapreduce.task.reduce.shuffle@2d7e552d 17/04/14 14:21:32 info reduce.mergemanagerimpl: mergermanager: memorylimit=334338464, maxsingleshufflelimit=83584616, mergethreshold=220663392, iosortfactor=10, memtomemmergeoutputsthreshold=10 17/04/14 14:21:32 info reduce.eventfetcher: attempt_local1656799721_0001_r_000000_0 thread started: eventfetcher fetching map completion events 17/04/14 14:21:32 info reduce.localfetcher: localfetcher#1 shuffle output of map attempt_local1656799721_0001_m_000000_0 decomp: 2 len: 6 memory 17/04/14 14:21:32 info reduce.inmemorymapoutput: read 2 bytes map-output attempt_local1656799721_0001_m_000000_0 17/04/14 14:21:32 info reduce.mergemanagerimpl: closeinmemoryfile -> map-output of size: 2, inmemorymapoutputs.size() -> 1, commitmemory -> 0, usedmemory ->2 17/04/14 14:21:32 info reduce.eventfetcher: eventfetcher interrupted.. returning 17/04/14 14:21:32 info mapred.localjobrunner: 1 / 1 copied. 17/04/14 14:21:32 info reduce.mergemanagerimpl: finalmerge called 1 in-memory map-outputs , 0 on-disk map-outputs 17/04/14 14:21:32 info mapred.merger: merging 1 sorted segments 17/04/14 14:21:32 info mapred.merger: down last merge-pass, 0 segments left of total size: 0 bytes 17/04/14 14:21:32 info reduce.mergemanagerimpl: merged 1 segments, 2 bytes disk satisfy reduce memory limit 17/04/14 14:21:32 info reduce.mergemanagerimpl: merging 1 files, 6 bytes disk 17/04/14 14:21:32 info reduce.mergemanagerimpl: merging 0 segments, 0 bytes memory reduce 17/04/14 14:21:32 info mapred.merger: merging 1 sorted segments 17/04/14 14:21:32 info mapred.merger: down last merge-pass, 0 segments left of total size: 0 bytes 17/04/14 14:21:32 info mapred.localjobrunner: 1 / 1 copied. 17/04/14 14:21:32 info configuration.deprecation: mapred.skip.on deprecated. instead, use mapreduce.job.skiprecords 17/04/14 14:21:32 info mapred.task: task:attempt_local1656799721_0001_r_000000_0 done. , in process of committing 17/04/14 14:21:32 info mapred.localjobrunner: 1 / 1 copied. 17/04/14 14:21:32 info mapred.task: task attempt_local1656799721_0001_r_000000_0 allowed commit 17/04/14 14:21:32 info output.fileoutputcommitter: saved output of task 'attempt_local1656799721_0001_r_000000_0' hdfs://localhost:19000/out19/_temporary/0/task_local1656799721_0001_r_000000 17/04/14 14:21:32 info mapred.localjobrunner: reduce > reduce 17/04/14 14:21:32 info mapred.task: task 'attempt_local1656799721_0001_r_000000_0' done. 17/04/14 14:21:32 info mapred.localjobrunner: finishing task: attempt_local1656799721_0001_r_000000_0 17/04/14 14:21:32 info mapred.localjobrunner: reduce task executor complete. 17/04/14 14:21:33 info mapreduce.job: job job_local1656799721_0001 running in uber mode : false 17/04/14 14:21:33 info mapreduce.job:  map 100% reduce 100% 17/04/14 14:21:33 info mapreduce.job: job job_local1656799721_0001 completed 17/04/14 14:21:33 info mapreduce.job: counters: 35         file system counters                 file: number of bytes read=17482                 file: number of bytes written=591792                 file: number of read operations=0                 file: number of large read operations=0                 file: number of write operations=0                 hdfs: number of bytes read=4822228                 hdfs: number of bytes written=0                 hdfs: number of read operations=13                 hdfs: number of large read operations=0                 hdfs: number of write operations=4         map-reduce framework                 map input records=15000                 map output records=0                 map output bytes=0                 map output materialized bytes=6                 input split bytes=113                 combine input records=0                 combine output records=0                 reduce input groups=0                 reduce shuffle bytes=6                 reduce input records=0                 reduce output records=0                 spilled records=0                 shuffled maps =1                 failed shuffles=0                 merged map outputs=1                 gc time elapsed (ms)=0                 total committed heap usage (bytes)=546308096         shuffle errors                 bad_id=0                 connection=0                 io_error=0                 wrong_length=0                 wrong_map=0                 wrong_reduce=0         file input format counters                 bytes read=2411114         file output format counters                 bytes written=0 

code

public static class tokenizermapper extends mapper<longwritable, text,text ,text>{           private text segment = new text();           //private threewritablevalues cust = new threewritablevalues();           private text word = new text();           private float balance = 0;           public void map(longwritable key, text value, context context) throws ioexception, interruptedexception {            string[] line = value.tostring().split("\\|");             string cust_key = line[1];             int nation = integer.parseint(line[3]);             if((balance > 8000) && ( nation < 15) && (nation > 1)){                segment.set(line[6]);               word.set(cust_key+","+balance);               context.write(segment,word);            }          }         }      public static class avgreducer extends reducer<text,text,text,text> {       public void reduce(text key, iterable<text> values,context context) throws ioexception, interruptedexception {             context.write(key, values.iterator().next());       }     }         public static void main(string[] args) throws exception {             configuration conf = new configuration();             job job = job.getinstance(conf, "word count");             job.setjarbyclass(mapreducetest.class);             job.setmapperclass(tokenizermapper.class);             job.setcombinerclass(avgreducer.class);             job.setreducerclass(avgreducer.class);             job.setoutputkeyclass(text.class);             job.setoutputvalueclass(text.class);             fileinputformat.addinputpath(job, new path(args[0]));             fileoutputformat.setoutputpath(job, new path(args[1]));             system.exit(job.waitforcompletion(true) ? 0 : 1);           } } 

please if know something.

there no output generated @ map phase

map output records=0 map output bytes=0 

in tokenizermapper class, value of balance defined 0.

private float balance = 0; 

and in map method, value of balance still 0 checked > 8000.

if((balance > 8000) && ( nation < 15) && (nation > 1)){                   segment.set(line[6]);              word.set(cust_key+","+balance);                  context.write(segment,word);            } 

the if condition never met , no mapper output , no reducer output.


Comments

Popular posts from this blog

php - Permission denied. Laravel linux server -

google bigquery - Delta between query execution time and Java query call to finish -

python - Pandas two dataframes multiplication? -