Out of curiosity (and with the danger of hijacking this thread as a stackoverflow discussion), I changed the awk code so that it could be used with GNU parallel - the "reduce"-step is essentially the same program as before:
$ cat map.awk
{ a[$2] += $3 }
END {
for (i in a) {
print "ignore", i, a[i]
}
}
$ cat reduce.awk
{ a[$2] += $3 }
END {
for (i in a) {
if (a[i] > max) {
maxk = i
max=a[i]
}
}
print "max_key:", maxk, "sum:", max
}
[ed: However, there are faster awks than gawk:
$ time mawk -f sum.awk ngrams.tsv
max_key: 2006 sum: 22569013
real 0m2.826s
user 0m2.391s
sys 0m0.422s
mawk is (a little) faster than pypy on my machine.
Thanks for the tip, always nice to see the author of tools commenting on hn :-)
The (old) version of parallel packaged with Ubuntu 16.04 (linux subsystem for windows) - doesn't have --pipe-part -- but running from upstream, the speed is more reasonable:
$ time (./parallel-20170522/src/parallel -a ngrams.tsv \
--pipe-part --block -1 -j4 mawk -f map.awk \
| mawk -f reduce.awk )
max_key: 2006 sum: 22569013
real 0m2.265s
user 0m4.672s
sys 0m1.672s
(Tried a few variants with/without -jN -- and this seems typical for the fast end of the spectrum).
$ time (cat ngrams.tsv \
| mawk -f map.awk \
| mawk -f reduce.awk )
max_key: 2006 sum: 22569013
real 0m3.472s
user 0m2.891s
sys 0m2.406s
[ed: btw, did a double-take when I saw your Gnu Privacy Guard id: 0x88888888 :-) ]
]