mapred.map.child.java.opts // heap size for map tasks mapred.reduce.child.java.opts // heap size for reduce tasks
Configuration conf = new Configuration(); conf.set("mapred.map.child.java.opts", "-Xmx512m"); conf.set("mapred.reduce.child.java.opts", "-Xmx512m");It will override any existing values.
Or
open your conf/mapred-site.xml and set these value
mapred.map.child.java.opts -Xmx1024m heap size for map tasks mapred.reduce.child.java.opts -Xmx1024m heap size for reduce tasks
Make sure ((num_of_maps * map_heap_size) + (num_of_reducers * reduce_heap_size)) is not larger than memory available in the system. Max number of mappers & reducers can also be tuned looking at available system resources.
thanks for your tip!!
ReplyDelete