[fuzzer] A thread-safety problem with RSS thread revealed by flaky failure of symbolize-deadlock.test

[yingcong-wu]

We got flaky failure of symbolize-deadlock.test, investigation revealed there is a thread-safety problem with RSS thread.

So fuzzer has a dedicated thread to monitor the RSS usage, and when OOM, it will call RssLimitCallback().

static void RssThread(Fuzzer *F, size_t RssLimitMb) {
  while (true) {
    SleepSeconds(1);
    size_t Peak = GetPeakRSSMb();
    if (Peak > RssLimitMb)
      F->RssLimitCallback();
  }
}

And RssLimitCallback() will call DumpCurrentUnit(), which calls PrintMutationSequence()

void MutationDispatcher::PrintMutationSequence(bool Verbose) {
  Printf("MS: %zd ", CurrentMutatorSequence.size());
  size_t EntriesToPrint =
      Verbose ? CurrentMutatorSequence.size()
              : std::min(kMaxMutationsToPrint, CurrentMutatorSequence.size());
  for (size_t i = 0; i < EntriesToPrint; i++)
    Printf("%s-", CurrentMutatorSequence[i].Name);
  if (!CurrentDictionaryEntrySequence.empty()) {
    Printf(" DE: ");
    EntriesToPrint = Verbose ? CurrentDictionaryEntrySequence.size()
                             : std::min(kMaxMutationsToPrint,
                                        CurrentDictionaryEntrySequence.size());
    for (size_t i = 0; i < EntriesToPrint; i++) {
      Printf("\"");
      PrintASCII(CurrentDictionaryEntrySequence[i]->GetW(), "\"-");
    }
  }
}

But when RSS thread is printing out CurrentMutatorSequence, the main fuzzer thread is still able to change the CurrentMutatorSequence, which can lead to buffer overflow read of CurrentMutatorSequence. In our case, with -D_GLIBCXX_ASSERTIONS, an assertion error within the vector header.

stl_vector.h:797: reference std::vector<fuzzer::MutationDispatcher::Mutator>::operator[](size_type) [_Tp = fuzzer::MutationDispatcher::Mutator, _Alloc = std::allocator<fuzzer::MutationDispatcher::Mutator>]: Assertion '__builtin_expect(__n < this->size(), true)' failed.
error: Aborted (core dumped)

[yingcong-wu]

Kindly ping @vitalybuka @MaskRay.