I got a couple of replies from one of the Linux kernel maintainers. This does seem to be an issue with 32-bit kernels and low memory pressure. Here are some comments (after I provided some more logs and tracepoints):
all those are from the kswapd (background memory reclaim). Which means that it doesn't catch any allocation which can stall for too long. Anyway the above tracepoint show that we are able to make some progress during the reclaim (nr_reclaimed > 0). So I suspect that this is indeed a large lowmem pressure and I do not see what we can do about that.
as well as:
and this one is hitting the min watermark while there is not really much to reclaim. Only the page cache which might be pinned and not reclaimable from this context because this is GFP_NOFS request. It is not all that surprising the reclaim context fights to get some memory. There is a huge amount of the reclaimable slab which probably just makes a slow progress.
That is not something completely surprising on 32b system I am afraid.