TESTS: testing with small bladder set

[m-huertasp]

This PR is intended to make sure that the nf-tests work to test the pipeline and to ensure the test data used is prepared to be shared with users.

Added FILTERDEPTHS module to allow depth-based filtering on input tables (matching our standard computation logic).
Key changes:

1. Test Infrastructure and Documentation Improvements

• Updated tests/README.md to document cluster-based testing, test configuration, output directories, and how to adapt the suite for other environments. Added instructions for new test cases, cleaning outputs, and debugging. [1] [2] [3] [4]
• Updated nf-test.config to use a scratch work directory, ignore nf-core modules/subworkflows, and load the nft-utils plugin.

2. Expanded and Refactored Test Suite

• Refactored and expanded tests/deepcsa.nf.test to include:
  • A standard functionality test with improved assertions and updated output paths.
  • An omega analysis test with checks for output structure, sample completeness, and directory layout.

3. Resource and Parameter Updates

• Increased resource limits for test processes in conf/test.config to better match cluster capabilities (4 CPUs, 20 GB RAM, 15 min).

4. New Depth Filtering Module and Workflow Integration

• Added a new process FILTERDEPTHS in modules/local/filterdepths/main.nf that filters depth tables based on a configurable minimum mean depth threshold.
• Integrated the FILTERDEPTHS module into the DEPTH_ANALYSIS workflow in subworkflows/local/depthanalysis/main.nf, so that when use_custom_minimum_depth is set, depth tables are filtered accordingly. [1] [2]
• Registered the FILTERDEPTHS process in the Nextflow configuration (conf/modules.config) with parameter wiring for minimum_depth.

These changes collectively add a new filtering capability, improve pipeline robustness, and ensure the test suite is comprehensive and portable across different SLURM cluster environments.

closes #435

[Copilot]

Pull request overview

This PR updates the nf-test harness and documentation for running deepCSA tests on an IRB SLURM cluster, and introduces a new FILTERDEPTHS module to apply the same minimum-mean-depth filtering logic to precomputed depth tables.

Changes:

• Expanded/refactored nf-test suite (new assertions, new failure-mode test) and updated snapshot metadata.
• Added FILTERDEPTHS module and wired it into DEPTH_ANALYSIS when using precomputed depths with a minimum depth threshold.
• Updated test configs/docs to reflect cluster execution, scratch dirs, and plugin usage.

Reviewed changes

Copilot reviewed 10 out of 11 changed files in this pull request and generated 10 comments.

Show a summary per file

File	Description
tests/test_data/input.csv	Replaced prior cluster-specific samplesheet with a small placeholder samplesheet.
tests/nextflow.config	Configured SLURM + Singularity for tests and set test params (incl. MAF + precomputed depths).
tests/deepcsa.nf.test	Refactored/expanded nf-test cases and assertions.
tests/deepcsa.nf.test.snap	Updated snapshot hashes and recorded tool versions/timestamps.
tests/README.md	Expanded documentation for running/adapting tests on clusters.
nf-test.config	Updated nf-test root config (workDir, ignored paths, plugin loading).
modules/local/filterdepths/main.nf	Added new module to filter depth tables by minimum mean depth.
subworkflows/local/depthanalysis/main.nf	Integrated FILTERDEPTHS for the precomputed-depths path.
conf/modules.config	Registered FILTERDEPTHS configuration and wired ext.minimum_depth.
conf/test.config	Increased test resource limits.
bin/utils_filter.py	Fixed filter-column detection logic when selecting FILTER.* columns.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[m-huertasp]

UPDATE

This is a comment so we know what was done here when coming back from holidays:

• Removed all hardcoded cluster paths from the test setup so the tests are portable across environments without manual file editing. Missing: Should we do the same with the container? Currently the path to IRB cluster is hardcoded.
• Test data: MAF and depths files are now fetched at runtime directly from bbglab/DeepClone_protocol via raw GitHub URLs (set in the when { params } block of deepcsa.nf.test). validation.ignoreParams added to nextflow.config so nf-schema skips file-existence checks for those two params.
• Work directory: workDir in nf-test.config now reads the NFT_TEST_WORKDIR environment variable (falling back to .nf-test in the repo root). Added info in the README about this and the environment variant to set up.
• Output directory: Per-test outdir now uses nf-test's built-in $outputDir variable (auto-derived from workDir).
• README: Updated to reflect all of the above, including setup instructions for a new cluster environment.

As the data is in DeepClone, the tests will not work until the repo is public.

[FerriolCalvet]

Looks good Marta thanks for the cleaning and improvement in documentation of all the testing machinery.
I see that all the parameters that are the same as the default ones stay only in the main nextflow.config file, I guess this means that as long as we do not change the defaults, the tests should be fine, and that whenever we update the defaults we should update the snapshots right?
Maybe we should indicate this somewhere.

I think this is ready to merge and it would be great to make the DeepClone repo public ASAP so that we can test it directly with the paths referencing the repo.

I am adding @migrau in case he wants to take a look at it, but in any case we will try to merge it this week, and then if needed apply patches later.

Also as discussed offline in the past with both of you, probably some steps would benefit more from having unit tests rather than full pipeline tests since it is to wide and intertwined that it is complicated to know what is being tested in each execution unless you know exactly how everything works.

[FerriolCalvet]

did you check if the changes to the files were big? or just minor changes on decimal points and so on?
more out of curiosity than nothing else

[m-huertasp]

Now that you've commented I've realized that the snapshot should be updated... As I changed the input to only 3 bladder samples I didn't check but it could be interesting..

[FerriolCalvet]

I know this would not be for general use, but it would be great to have an example of which are the exact steps that you follow, I guess you may be indicating -outputDir or other flags in the command-line instructions so this way we have it explicitly defined here and there is no doubt on how to handle the tests efficiently.
(maybe a small section at the end would be the cleanest and simplest to not be misleading)

Maybe I am completely off and this should not be here eh!

[m-huertasp]

Missing:

• Update snapshots once DeepClone is public
• Check if there are some instructions missing on how to run the tests

[m-huertasp]

Already checked that tests PASS and snapshot updated.