Allocation of more than 2GB in unreachable heap causes out of memory errror

Created by: SweetVishnya

How to reproduce

This reproduces on both Linux x86-32 and x86-64.

Just allocate more than 2GB from dr_client_main with malloc:

    int *tmp = 0;
    int cnt = 0;
    for (int i = 0; i != 4 * 96086; ++i)
    {
        tmp = (int *) malloc(30000);
        ++cnt;
        std::cout << cnt << std::endl;
        for (int j = 0; j != 30000 / sizeof(int); ++j)
        {
            tmp[j] = i * 43;
        }
    }
    std::cout << "Do not optimize out malloc " << tmp[20] << std::endl;

Run debug:

dynamorio/bin64/drrun -debug -c client.so -- /bin/ls

Get an error:

<Out of vmheap reservation - reserving 256KB. Falling back onto OS allocation>
<CURIOSITY : (0) && "Out of vmheap reservation" in file /home/vishnya/dynamorio/core/heap.c line 1614
<Out of memory - cannot reserve 256KB. Trying to recover.>
<Out of memory.  Program aborted.>
<OOM Status: R 0x0000000000000001>

What goes wrong

When unreachable memory size reaches first 2GB it tries to ask more memory from system. But this code fails and leads to out of memory.