Portable CM script failed (name = benchmark-program, return code = 35072

[PapperYZ]

I am running resnet50 benchmark using cm with a docker container, and it failed after all download and compiling were done, and just started benchmarking,

Below are my logs, what does the return code 35072 really mean?

Checking compiler version ...

Ubuntu clang version 18.1.3 (1ubuntu1)
Target: aarch64-unknown-linux-gnu
Thread model: posix
InstalledDir: /usr/bin

Compiling source files ...

main.cpp

/usr/bin/clang++ -c -std=c++14 -DCM_MODEL_RESNET50 -DCM_MLPERF_BACKEND_ONNXRUNTIME -DCM_MLPERF_DEVICE_CPU -O2 -O4 -I/root/CM/repos/local/cache/fd62bf2638f24cd3/install/include -I/root/CM/repos/local/cache/490298ae3cb5442a/install/onnxruntime-linux-aarch64-1.16.3/include -I/root/CM/repos/mlcommons@cm4mlops/script/app-mlperf-inference-mlcommons-cpp/inc main.cpp -o main.o
clang++: warning: -O4 is equivalent to -O3 [-Wdeprecated]

Linking ...

/usr/bin/clang++ -std=c++14 -DCM_MODEL_RESNET50 -DCM_MLPERF_BACKEND_ONNXRUNTIME -DCM_MLPERF_DEVICE_CPU -O2 -O4 main.o -o /root/CM/repos/local/cache/97b15df295c847d1/run.out -L/root/CM/repos/local/cache/fd62bf2638f24cd3/install/lib -L/root/CM/repos/local/cache/490298ae3cb5442a/install/onnxruntime-linux-aarch64-1.16.3/lib -lmlperf_loadgen -lpthread -lonnxruntime -O2 -O4
INFO:root:         ! call "postprocess" from /root/CM/repos/mlcommons@cm4mlops/script/compile-program/customize.py
INFO:root:  * cm run script "benchmark-mlperf"
INFO:root:         ! call "postprocess" from /root/CM/repos/mlcommons@cm4mlops/script/benchmark-program-mlperf/customize.py
INFO:root:  * cm run script "benchmark-program program"
INFO:root:    * cm run script "detect cpu"
INFO:root:      * cm run script "detect os"
INFO:root:             ! cd /root/CM/repos/local/cache/97b15df295c847d1
INFO:root:             ! call /root/CM/repos/mlcommons@cm4mlops/script/detect-os/run.sh from tmp-run.sh
INFO:root:             ! call "postprocess" from /root/CM/repos/mlcommons@cm4mlops/script/detect-os/customize.py
INFO:root:           ! cd /root/CM/repos/local/cache/97b15df295c847d1
INFO:root:           ! call /root/CM/repos/mlcommons@cm4mlops/script/detect-cpu/run.sh from tmp-run.sh
INFO:root:           ! call "postprocess" from /root/CM/repos/mlcommons@cm4mlops/script/detect-cpu/customize.py
INFO:root:         ! cd /root/CM/repos/local/cache/97b15df295c847d1
INFO:root:         ! call /root/CM/repos/mlcommons@cm4mlops/script/benchmark-program/run-ubuntu.sh from tmp-run.sh

 /root/CM/repos/local/cache/97b15df295c847d1/run.out  2>&1 | tee '/root/CM/repos/local/cache/b97ae9541a004ef7/valid_results/d5ef47345c85-cpp-cpu-onnxruntime-vdefault-default_config/resnet50/singlestream/performance/run_1/console.out'; echo ${PIPESTATUS[0]} > exitstatus
MLPerf Conf path: /root/CM/repos/local/cache/9633a0068b684365/inference/mlperf.conf
User Conf path: /root/CM/repos/mlcommons@cm4mlops/script/generate-mlperf-inference-user-conf/tmp/7c318ebe49344c7a8e292a59b7fc8756.conf
Dataset Preprocessed path: /root/CM/repos/local/cache/b43f1c701cc2419c
Dataset List filepath:
Scenario: SingleStream
Mode: PerformanceOnly
Batch size: 1
Query count override: 0
Performance sample count override in application: 0
loaded imagenet with 500 samples
starting benchmark
loading samples to ram with total sample size: 500

Config file missing for given hw_name: 'd5ef47345c85', implementation: 'cpp', device: 'cpu,  backend: 'onnxruntime', copying from default
Using MLCommons Inference source from '/root/CM/repos/local/cache/9633a0068b684365/inference'
Original configuration value 0.1 target_latency
Adjusted configuration value 0.04000000000000001 target_latency
Output Dir: '/root/CM/repos/local/cache/b97ae9541a004ef7/valid_results/d5ef47345c85-cpp-cpu-onnxruntime-vdefault-default_config/resnet50/singlestream/performance/run_1'
resnet50.SingleStream.target_latency = 0.04000000000000001
resnet50.SingleStream.max_duration = 660000

***************************************************************************
CM script::benchmark-program/run.sh

Run Directory: /root/CM/repos/local/cache/97b15df295c847d1

CMD:  /root/CM/repos/local/cache/97b15df295c847d1/run.out  2>&1 | tee '/root/CM/repos/local/cache/b97ae9541a004ef7/valid_results/d5ef47345c85-cpp-cpu-onnxruntime-vdefault-default_config/resnet50/singlestream/performance/run_1/console.out'; echo \${PIPESTATUS[0]} > exitstatus

{
    "return": 2,
    "error": "Portable CM script failed (name = benchmark-program, return code = 35072)\n\n\n^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\nNote that it is often a portability issue of a third-party tool or a native script\nwrapped and unified by this CM script (automation recipe). Please re-run\nthis script with --repro flag and report this issue with the original\ncommand line, cm-repro directory and full log here:\n\nhttps://github.com/mlcommons/cm4mlops/issues\n\nThe CM concept is to collaboratively fix such issues inside portable CM scripts\nto make existing tools and native scripts more portable, interoperable\nand deterministic. Thank you",
    "system_info": {
        "platform": "Linux-6.8.9-yocto-tiny-aarch64-with-glibc2.39",
        "architecture": "aarch64",
        "hostname": "d5ef47345c85",
        "CPU": "aarch64",
        "CPU_FREQ": null,
        "memory": "4 GB"
    }
}
Portable CM script failed (name = benchmark-program, return code = 35072)


^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Note that it is often a portability issue of a third-party tool or a native script
wrapped and unified by this CM script (automation recipe). Please re-run
this script with --repro flag and report this issue with the original
command line, cm-repro directory and full log here:

https://github.com/mlcommons/cm4mlops/issues

The CM concept is to collaboratively fix such issues inside portable CM scripts
to make existing tools and native scripts more portable, interoperable
and deterministic. Thank you

[arjunsuresh]

Actually now we are using MLC instead of CM for maintaining the MLPerf scripts. But for your specific query, can you try passing --target_latency=10 and see if it works? The current error looks like a OS error which can happen if memory is not enough.

[PapperYZ]

Thank you @arjunsuresh , would you mind providing more guidance on how to pass --target_latency=10, which command needs this switch?

[PapperYZ]

@arjunsuresh I believe I found the reason for it, see below... will the target_latency address this limitation?

[PapperYZ]

Also, should I update the mlperf.conf and change below to a slow number so accommodate small DRAM?

[arjunsuresh]

@PapperYZ Glad that you have figured out the issue. If you are using CM/MLC command to launch the run, you can simply add --target_latency=10 to the run command.

The reason why this is needed is not very clear to me either. It has nothing to do with the performance sample count as 1024 * 2242243 *(4 bytes) = ~600 MB only. But if we are using a smaller target latency like 0.2 ms, the amount of RAM needed is coming to be around 12GB - looks like a bug in the loadgen. So, giving a larger latency is just a work around for this.

[PapperYZ]

Hi @arjunsuresh , I follow your guidance to run this manually with the latest MLC environments, below are the commands I plan to run:

sudo apt install git python3-pip python3-venv
python3 -m venv mlc
source mlc/bin/activate
pip install mlc-scripts
mlcr install,python-venv --name=mlperf
export MLC_SCRIPT_EXTRA_CMD="--adr.python.name=mlperf"
mlcr run-mlperf,inference,_find-performance,_full,_r5.0-dev \
   --model=resnet50 \
   --implementation=reference \
   --framework=onnxruntime \
   --category=edge \
   --scenario=Offline \
   --execution_mode=test \
   --device=cpu  \
   --docker --quiet \
   --test_query_count=1000 \
   --target_latency=10

Please help me check if they are correct.

[Mizokuiam]

Hi there @PapperYZ! Your MLC command in comment 6 looks correct for running the benchmark with the increased target_latency. The --target_latency=10 flag is correctly placed within the mlcr run-mlperf command.

The suspicion of a loadgen bug related to target_latency seems valid, given the discrepancy between the expected dataset size (~600MB) and the apparent memory requirement exceeding available DRAM (4GB). The --target_latency=10 workaround, while not ideal for accurate performance measurement, should allow the benchmark to complete.

After running the command, if the issue persists, follow the original error message's advice: re-run with the --repro flag, gather the cm-repro directory and full logs, and create a new issue on the mlcommons/cm4mlops GitHub repository. This will help the MLC developers investigate the potential loadgen bug and provide a proper fix. Also, specify the platform and MLPerf version for clarity.

Finally, although you've located the mlperf.conf, modifying values directly there is generally discouraged within the MLC framework. Stick to command-line arguments where possible for better reproducibility and tracking of experiment parameters.

See if this works for you.

[PapperYZ]

Thanks @Mizokuiam, are you an AI agent? I can't believe you replied to me within a minute

[Mizokuiam]

Thanks @Mizokuiam, are you an AI agent? I can't believe you replied to me within a minute

i have checked your issue yesterday actually and i replied once you updated.
@arjunsuresh hope i was helpful to your matter!

[arjunsuresh]

@PapperYZ We recommend to take the commands directly from the inference docs: https://docs.mlcommons.org/inference/benchmarks/image_classification/resnet50/ as they are a bit too complex to review manually for every scenario/device/implementation combinations. Your commands look fine but if you are running Offline scenario instead of --target_latency it must be --target_qps specifying the expected performance of the system in QPS. But for --execution_mode=test this option can be omitted (Offline scenario shouldn't have the crash issue for the default QPS value) and then for further runs, MLC will automatically fetch the recorded QPS value of the system.

[PapperYZ]

Hi @arjunsuresh , using mlc venv solves the problem, I guess this can be a workaround for the cm issue with RAM size. Let me know if you need anything to fix the bug