Nextflow - SLURM job interface disconnect

[nickhsmith]

Bug report

I am noticing a problem with the Nextflow-SLURM interface when processing large number of jobs. For example, I am running a workflow and have spawned ~10,000 tasks (50 samples * 150 intervals/sample) + downstream tasks) using the nf-core sarek workflow. However, I notice that my tasks are hanging, and as a result, nothing is being submitted to SLURM despite the jobs being completed.

Expected behavior and actual behavior

Nextflow generates a task. Nextflow submits the task to SLURM. SLURM runs the job. SLURM finishes the job (creating .exitcode with value 0). Nextflow recognizes this and submits the next task.

This last step of recognizing a finished job seems to be failing.

Steps to reproduce the problem

More tangibly, I have used queueSize to limit my workflow to 100 jobs at 1 time. Some of those jobs will finish (.exitcode exists with the value 0 in the local work dir (work/76/69b9b4...)) BUT these jobs are still considered RUNNING when I look at the .nextflow.log file (see last photo)

Looking through the log file. I see that the SLURM job ID 636541 is submitted at 15:17 on March 2
It finishes and creates the .exitcode file (with a successful 0 exit) at 18:10. SLURM thinks the job is completed at this point.

but as of yesterday March 3 there is no completed status of this job, as a result SLURM won't start any of the remaining ~7000 jobs in the task scheduler)

Mar-03 05:26:48.603 [Task submitter] DEBUG n.processor.TaskPollingMonitor - %% executor slurm > tasks in the submission queue: 6714 -- tasks to be submitted are shown below

Any help would be appreciated, but is there a reason that the completed jobs aren't being recognized as such, or what I can do to force a re-recognition? Thanks in advance

Environment

nextflow info
Version: 21.10.6 build 5660
Created: 21-12-2021 16:55 UTC (17:55 CEST)
System: Linux 4.18.0-348.12.2.el8_5.x86_64
Runtime: Groovy 3.0.9 on OpenJDK 64-Bit Server VM 10.0.2+13
Encoding: UTF-8 (UTF-8)

shell: bash

[nickhsmith]

I am still dealing with this problem with the nf-core/rnaseq pipeline this time, and the lack of throughput is quite frustrating. @csawye01 and I are both having the same issue. Is there anywhere I can try to look or dive into why this isn't working? Attached is my .nextflow.log file

SLURM thinks the job is done

NEXTFLOW thinks the job is running

NEXTFLOW log (edited)

nextflow_log.txt

EDIT

I tried running this pipeline with the same configuration without SLURM and it worked without a problem. But I really need it to work with SLURM in my shared compute environment.

[nickhsmith]

@pditommaso I was wondering if there has been an exploration of this problem? Based on other issues it seems like there may be some problems with the platform/slurm tag, but I haven't seen a lot of discussion of these issues.

Is there anything I can do to help push for this fix? Otherwise, the lack of throughput is severely limiting.

Please let me know, I know it would also help other users (such as @csawye01) as well.

[pditommaso]

Hard to say anything w/o more detailed debug info. Please executing it again adding the -trace nextflow.executor option as shown below, and upload the .nextflow.log file when done:

nextflow -trace nextflow.executor run [your pipeline params]

[nickhsmith]

Thank you for the comment! I will upload the log file when I can. Thanks again!

[nickhsmith]

Attached is the zipped log file. I have edited the PATH names, but otherwise, this is the result after 2 days (~1 day of hanging behavior)

nextflow_trace.log.gz

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[ligrob]

Hi all,
I also did run into the same problem. In case you would consider it helpful, I would be happy to provide further information to help the debugging process.

[nickhsmith]

@ligrob I would love to know what you went through, and how you got around it! Thanks for responding!

[sanderthierens]

@ligrob I got the exact same problem as @nickhsmith, I did notice that I could force nextflow to resume by making a new ssh connection to the machine. If you could tell me the work around this would help me a lot. Thanks!

[hmehlan]

I have encountered the same problem with a nf-core/rnaseq pipeline and slurm.
A thread dump shows where Nextflow hangs:

"Task monitor" 38 prio=5 os_prio=0 cpu=25638.16ms elapsed=76825.33s allocated=1170M defined_classes=248 tid=0x00007f05fd2e7740 nid=0x2af5ff runnable  [0x00007f05d20c9000]
   java.lang.Thread.State: RUNNABLE
    at sun.nio.ch.FileDispatcherImpl.read0(java.base@17.0.6/Native Method)
    at sun.nio.ch.FileDispatcherImpl.read(java.base@17.0.6/FileDispatcherImpl.java:48)
...
    at nextflow.executor.GridTaskHandler.readExitStatus(GridTaskHandler.groovy:371)
    at nextflow.executor.GridTaskHandler.checkIfCompleted(GridTaskHandler.groovy:452)
    at nextflow.processor.TaskPollingMonitor.checkTaskStatus(TaskPollingMonitor.groovy:611)
    at nextflow.processor.TaskPollingMonitor.checkAllTasks(TaskPollingMonitor.groovy:534)

It seems that nextflow no longer recognizes completed Slurm jobs when it tries to read the file .exitcode.
The Java call to read this file gets stuck and does not return.
Nextflow resumes in my case when I call ls, lsof or stat for this file.
As a workaround, I have placed this snippet in my shell script before calling nextflow run :

WORK_DIR="./work"
nohup lsof +D ${WORK_DIR} -r 600 &> /dev/null &
LSOF_PID=$!
trap "kill $LSOF_PID" EXIT

nextflow run ...
  -work-dir ${WORK_DIR}
  ...

Environment

  nextflow version: 23.04.1 build 5866
  Ubuntu 22.04
  Ceph storage platform
  openjdk version "17.0.7" 2023-04-18

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.