Sunday, January 15, 2012

How to overcome on Java Heap Space Error in Hadoop MapReduce Framework

mapred.map.child.java.opts // heap size for map tasks
mapred.reduce.child.java.opts // heap size for reduce tasks

Configuration conf = new Configuration();
conf.set("mapred.map.child.java.opts", "-Xmx512m");
conf.set("mapred.reduce.child.java.opts", "-Xmx512m");
It will override any existing values.

Or

open your conf/mapred-site.xml and set these value

    mapred.map.child.java.opts
    -Xmx1024m
    heap size for map tasks 
  


  
    mapred.reduce.child.java.opts
    -Xmx1024m
    heap size for reduce tasks 
  

Make sure ((num_of_maps * map_heap_size) + (num_of_reducers * reduce_heap_size)) is not larger than memory available in the system. Max number of mappers & reducers can also be tuned looking at available system resources.

1 comment: