[libfuzzer] Out-of-memory limit calculations don't work well with realloc

[mactul]

Hi, I've noticed that libfuzzer is counting the memory allocated by realloc incorrectly.
For example:

void* ptr = malloc(100);
ptr = realloc(ptr, 200);

Here, libfuzzer counts an allocation of 300 bytes (100+200) instead of just 200.

Here's the proof:

#include <stdint.h>
#include <stdlib.h>

extern int LLVMFuzzerTestOneInput(const uint8_t *Data __attribute__((unused)), size_t Size __attribute__((unused)))
{
    void* ptr = NULL;
    for(int i = 10000; i < 100000; i++)
    {
        ptr = realloc(ptr, i);
    }

    free(ptr);

    return 0;
}

By definition, this program only consumes 99999 bytes, which is well below libfuzzer's default limit of 256MB (2048Mb).

However, if you compile the program and run it:

clang -o clang_bug clang_bug.c -fsanitize=address,fuzzer -Wall -Wextra
./clang_bug

You get an out-of-memory

INFO: Running with entropic power schedule (0xFF, 100).
INFO: Seed: 1295557038
INFO: Loaded 1 modules   (3 inline 8-bit counters): 3 [0x62836eaa9b88, 0x62836eaa9b8b), 
INFO: Loaded 1 PC tables (3 PCs): 3 [0x62836eaa9b90,0x62836eaa9bc0), 
INFO: -max_len is not provided; libFuzzer will not generate inputs larger than 4096 bytes
INFO: A corpus is not provided, starting from an empty corpus
==2493== ERROR: libFuzzer: out-of-memory (used: 2855Mb; limit: 2048Mb)
   To change the out-of-memory limit use -rss_limit_mb=<N>

Live Heap Allocations: 24233028 bytes in 20 chunks; quarantined: 249670930 bytes in 9176 chunks; 51767 other chunks; total chunks: 60963; showing top 95% (at most 8 unique contexts)
24121072 byte(s) (99%) in 8 allocation(s)

My hypothesis is that the memory calculated is in fact the sum of i from 10000 to 100000.
This makes 99999*100000/2 - 9999*10000/2 or 4949955000 bytes, so about 4.6GB, well beyond libfuzzer's default limit.

This hypothesis seems even more credible to me if you consider that this program poses no problems:

#include <stdint.h>
#include <stdlib.h>

extern int LLVMFuzzerTestOneInput(const uint8_t *Data __attribute__((unused)), size_t Size __attribute__((unused)))
{
    void* ptr = NULL;
    for(int i = 10000; i < 100000; i++)
    {
        free(ptr);
        ptr = malloc(i);
    }

    free(ptr);

    return 0;
}

I use clang version 16.0.6 on Arch Linux.

[EMedhat]

==13429== ERROR: libFuzzer: out-of-memory (used: 2063Mb; limit: 2048Mb)
To change the out-of-memory limit use -rss_limit_mb=
I have the same issue when i increased the -rss_limit_mb nothing happen and same error appeared