Ask For Help! I encounter the error ,when I run the dpgen . Thanks you very much !

[sorprano]

When I run with dpgen to the 12th round, I suddenly get an error, but there are similar files in the folder, how can I solve it, thank you
here is the error.

INFO:dpgen:-------------------------iter.000012 task 08--------------------------
INFO:dpgen:failed frame: 0 in 100 0.00 %
INFO:dpgen:failed tasks: 0 in 100 0.00 %
INFO:dpgen:=============================iter.000013==============================
INFO:dpgen:-------------------------iter.000013 task 00--------------------------
INFO:dpgen:-------------------------iter.000013 task 01--------------------------
Traceback (most recent call last):
File "/home/ps/miniconda3/envs/deepmd2/bin/dpgen", line 8, in
sys.exit(main())
File "/home/ps/miniconda3/envs/deepmd2/lib/python3.10/site-packages/dpgen/main.py", line 255, in main
args.func(args)
File "/home/ps/miniconda3/envs/deepmd2/lib/python3.10/site-packages/dpgen/generator/run.py", line 5394, in gen_run
run_iter(args.PARAM, args.MACHINE)
File "/home/ps/miniconda3/envs/deepmd2/lib/python3.10/site-packages/dpgen/generator/run.py", line 4725, in run_iter
run_train(ii, jdata, mdata)
File "/home/ps/miniconda3/envs/deepmd2/lib/python3.10/site-packages/dpgen/generator/run.py", line 868, in run_train
submission.run_submission()
File "/home/ps/miniconda3/envs/deepmd2/lib/python3.10/site-packages/dpdispatcher/submission.py", line 265, in run_submission
self.try_download_result()
File "/home/ps/miniconda3/envs/deepmd2/lib/python3.10/site-packages/dpdispatcher/submission.py", line 281, in try_download_result
raise e
File "/home/ps/miniconda3/envs/deepmd2/lib/python3.10/site-packages/dpdispatcher/submission.py", line 277, in try_download_result
self.download_jobs()
File "/home/ps/miniconda3/envs/deepmd2/lib/python3.10/site-packages/dpdispatcher/submission.py", line 498, in download_jobs
self.machine.context.download(self)
File "/home/ps/miniconda3/envs/deepmd2/lib/python3.10/site-packages/dpdispatcher/contexts/local_context.py", line 163, in download
raise FileNotFoundError(
FileNotFoundError: cannot find download file /data/dpgen/02.dpgen/work/d9c1af2ab1760e3b12470d34af3da205844fb2a4/000/model.ckpt.meta

/data/dpgen/02.dpgen/work/d9c1af2ab1760e3b12470d34af32a205844fb2a4/000/,in this file i have 7dd0baba3ae9abd4b1afaa21c4a3ba21633f5cf_task_tag_finished lcurve.out model.cikpt-197000.index model.cikpt-199000.data-00000-of-00001 model.cikpt-200000.meta train.log
checkpoint model.cikpt-196000.data-00000-of-00001 model.cikpt-197000.meta model.cikpt-199000.index model.cikpt.data-00000-of-00001
frozen_model.pb model.cikpt-196000.index model.cikpt-198000.data-00000-of-00001 model.cikpt-199000.meta model.cikpt.index
input.json model.cikpt-196000.meta model.cikpt-198000.index model.cikpt-200000.data-00000-of-00001 model.cikpt.meta
input_v2_compat.json model.cikpt-197000.data-00000-of-00001 model.cikpt-198000.meta model.cikpt-200000.index out.json

[njzjz]

I suggest search model.cikpt in your files using

grep -R model.cikpt

The default filename is model.ckpt. You might insert i by mistake.

[sorprano]

Sorry for the delay in replying to you. I was ill accidentally and did not read the reply. These files were automatically generated by the system and do exist in the system

this is my file
7dd0baba3ae9a4b1afaa21c4a3ba214633f5cf_task_tag_finished lcurve.out model.cikpt-197000.index model.cikpt-199000.data-00000-of-00001 model.cikpt-200000.meta train.log
checkpoint model.cikpt-196000.data-00000-of-00001 model.cikpt-197000.meta model.cikpt-199000.index model.cikpt.data-00000-of-00001
frozen_model.pb model.cikpt-196000.index model.cikpt-198000.data-00000-of-00001 model.cikpt-199000.meta model.cikpt.index
input.json model.cikpt-196000.meta model.cikpt-198000.index model.cikpt-200000.data-00000-of-00001 model.cikpt.meta
input_v2_compat.json model.cikpt-197000.data-00000-of-00001 model.cikpt-198000.meta model.cikpt-200000.index out.json

Can I learn the names of these files directly like the following to avoid errors, because the program will work properly after I try this

folders = ['000', '001', '002', '003']

file_patterns = [
'model.cikpt.data-00000-of-00001',
'model.cikpt.index',
'model.cikpt.meta'
]

for folder in folders:
if not os.path.exists(folder):
continue

files_exist = True
for pattern in file_patterns:
    if not os.path.exists(os.path.join(folder, pattern)):
        files_exist = False
        break


if files_exist:
    for pattern in file_patterns:
        old_path = os.path.join(folder, pattern)
        new_path = os.path.join(folder, pattern.replace('cikpt', 'ckpt'))
        os.rename(old_path, new_path