DeepSomatic does not finish for some samples in Slurm runs

[lipikakalson]

Hi all,

I am running DeepSomatic on an HPC cluster using a slurm-based parallel workflow. The pipeline works successfully for the majority of samples (n>50), but for a small subset of samples, the job never finishes.

There is no error message, no crash, and the job remains running indefinitely until manually terminated. All samples were preprocessed using the exact same pipeline and tools.

Command used:

singularity exec \
  -B "${TMPDIR_HOST}":"${TMPDIR_HOST}" \
  -B /usr/lib/locale:/usr/lib/locale \
  -B "${REF_DIR}:${REF_DIR}" \
  -B "${TUMOR_DIR}:${TUMOR_DIR}" \
  -B "${OUT_DIR}:${OUT_DIR}" \
  $IMAGE run_deepsomatic \
  --model_type=FFPE_WGS_TUMOR_ONLY \
  --ref=${REF} \
  --reads_tumor=${BAM} \
  --output_vcf=${OUT_DIR}/${SAMPLE}.deepsomatic.vcf.gz \
  --sample_name_tumor=${SAMPLE} \
  --num_shards=8 \
  --logging_dir=${OUT_DIR}/logs/${SAMPLE} \
  --use_default_pon_filtering=true \
  --intermediate_results_dir=${OUT_DIR}/logs/${SAMPLE}

Please find some screenshot of logs attached:

Any guidance on how to diagnose or trace where execution stalls would be greatly appreciated.

Thank you very much for your help.

Best regards,
Lipika

[pichuan]

Hi @lipikakalson ,
If I understand, you said "for a small subset of samples, the job never finishes", and from the logs you shared, it seems like postprocess_variants is the step that never finished (but also no logs). Is that correct?

[pichuan]

Hi @lipikakalson ,

Assuming my understanding of your problem is correct, I just check with my coworker @lucasbrambrink . We both find it surprising that your postprocess_variants.log is completely empty.

A few things you can try: You can run postprocess_variants with --cpu=0 or setting --num_partitions=32. The second flag will help with memory issue (in case there was a OOM). Given that you're running with run_deepsomatic, you can add a flag like: --postprocess_variants_extra_args="num_partitions=32", which will allow you to pass that flag and value to the postprocess_variants step.

Let us know if that helps.

[lipikakalson]

Hi @pichuan , thank you for the suggestions.

'postprocess_variants' does start and produces output in the Slurm stdout log. The postprocess_variants.log file in the run folder remains empty.

I will try rerunning these samples with your suggestions and will update you soon.

Best regards,
Lipika

[lipikakalson]

Hi, sorry for the late update.

I tried running DeepSomatic using --postprocess_variants_extra_args="num_partitions=32" as you suggested, but unfortunately I am still encountering the same problem as before.

Please let me know if there are any other parameters or steps I should try.

[kishwarshafin]

@lipikakalson can you try the other extreme please and set cpus=1,num_partitions=1 and see what happens? I think multiprocessing is causing issues for you and this will disable it.

[kishwarshafin]

Closing this for now. Feel free to reopen if you have more questions.