How to overcome on Java Heap Space Error in Hadoop MapReduce Framework

?

1 2	mapred.map.child.java.opts // heap size for map tasks mapred.reduce.child.java.opts // heap size for reduce tasks

?

1 2 3

Configuration conf = new Configuration(); conf.set("mapred.map.child.java.opts", "-Xmx512m"); conf.set("mapred.reduce.child.java.opts", "-Xmx512m");

It will override any existing values.

Or

open your conf/mapred-site.xml and set these value

?

1 2 3 4 5 6 7 8 9 10 11 12

<property>     <name>mapred.map.child.java.opts</name>     <value>-Xmx1024m</value>     <description>heap size for map tasks </description>   </property>       <property>     <name>mapred.reduce.child.java.opts</name>     <value>-Xmx1024m</value>     <description>heap size for reduce tasks </description>   </property>

Make sure ((num_of_maps * map_heap_size) + (num_of_reducers * reduce_heap_size)) is not larger than memory available in the system. Max number of mappers & reducers can also be tuned looking at available system resources.