That’s an interesting problem. I’m not particularly certain of the answer. Looking through the configuration, no settings jump out at me in terms of controlling the combiner process. The mapred.inmem.merge.threshold parameter defaults to 1000. So I think that after 1000 intermediate files get created, they should be merged before presentation to the combiner. I’m not sure why so many millions of files are being created.

When you say that the mapper runs “a billion iterations,” does that mean that for every input record to the mapper, you generate a billion (k, v) pairs as output? How many input records were you processing? I’m not sure Hadoop was really designed for a 1,000,000,000:1 fanout ratio. Most MapReduce jobs characteristically have less data output than input.

You might have better luck getting a solution from the Hadoop Core mailing list; sign up at http://hadoop.apache.org/core/mailing_lists.html. Not only are there more of us there who can help, the list format is much more well-suited to the back-and-forth required to diagnose these sorts of issues.

Regards,
- Aaron Kimball

The framework made > 3 million files at each node in a same directory and all the inode tables were fragmented. In fact, counting the number of files itself took 20 mins.

I am wondering if there is a parameter that can be tweaked so that the intermediate map outputs are spilled infrequently and appended to existing tmp files instead of creating new ones. Note that I did not run over the default (1024) open fd limit at any point of time.

Or does the hadoop framework need to be changed in order to do this?