Description:
clip_duration does not need to be assigned for single-mouse-v6-features, but controls resource scaling.
Details:
Currently, all our resources scale on 2 pipeline parameters: clip_duration and num_mice (multimouse tasks only).
This works rather well for the main workflow (single-mouse), since it controls the size of downstream data.
Current Workaround:
If users change clip_duration, they much change it every time they re-process data.
Potential Solutions:
(1)
Have resource calculations look at other related parameters to infer the size of data.
Since feature_bins also can indicate the size of time data, we could simply calculate the larger of the two: max(max(feature_bins)*5*60*30, clip_duration).
(2)
Have nextflow calculate the size of input data to scale it. If we can just inspect the shape of the pose data, that would not require the user to set a parameter.
I'm not sure if this is possible, but it sounds like something that nextflow would be able to do since it's a fairly common in genomics (where nextflow flourishes).
Description:
clip_durationdoes not need to be assigned forsingle-mouse-v6-features, but controls resource scaling.Details:
Currently, all our resources scale on 2 pipeline parameters:
clip_durationandnum_mice(multimouse tasks only).This works rather well for the main workflow (
single-mouse), since it controls the size of downstream data.Current Workaround:
If users change
clip_duration, they much change it every time they re-process data.Potential Solutions:
(1)
Have resource calculations look at other related parameters to infer the size of data.
Since
feature_binsalso can indicate the size of time data, we could simply calculate the larger of the two:max(max(feature_bins)*5*60*30, clip_duration).(2)
Have nextflow calculate the size of input data to scale it. If we can just inspect the shape of the pose data, that would not require the user to set a parameter.
I'm not sure if this is possible, but it sounds like something that nextflow would be able to do since it's a fairly common in genomics (where nextflow flourishes).