[fuzzer] RunIndividualFiles may have more runs than the number specified in the argument

[yingcong-wu]

I observe that test fuzzer-finalstats.test will fail flakily. After investigation, I found that RunIndividualFiles may runs more than the number given in the argument.

In the

llvm-project/compiler-rt/lib/fuzzer/FuzzerDriver.cpp

Line 853 in b171849

if (RunIndividualFiles) {

      for (int Iter = 0; Iter < Runs; Iter++)
        RunOneTest(F, Path.c_str(), Options.MaxLen);

We can see that the Runs is predetermined, but inside RunOneTest()

int RunOneTest(Fuzzer *F, const char *InputFilePath, size_t MaxLen) {
  Unit U = FileToVector(InputFilePath);
  if (MaxLen && MaxLen < U.size())
    U.resize(MaxLen);
  F->ExecuteCallback(U.data(), U.size());
  if (Flags.print_full_coverage) {
    // Leak detection is not needed when collecting full coverage data.
    F->TPCUpdateObservedPCs();
  } else {
    F->TryDetectingAMemoryLeak(U.data(), U.size(), true);
  }
  return 0;
}

TryDetectingAMemoryLeak() will also have the chance to run ExecuteCallback(), and also because we call TryDetectingAMemoryLeak() with DuringInitialCorpusExecution=True, so the check for TotalNumberOfRuns inside it will not work.

  if (!DuringInitialCorpusExecution &&
      TotalNumberOfRuns >= Options.MaxNumberOfRuns)
    return;

First, I come up with this solution:

  while (F->getTotalNumberOfRuns() < (size_t)Runs) 
	RunOneTest(F, Path.c_str(), Options.MaxLen);

but this is not complete, because in the last round of RunOneTest() it is still possible to have 2 runs(one for normal run and one for the leak detection.

Then I think maybe we can disable the leak detection for the last round, but the downside is if we only have 1 run, then we will have no leak detection.

What do you think the proper solution is here?

[yingcong-wu]

Kindly ping @vitalybuka @MaskRay.