Explicitly specifying memory units breaks the profile in older versions of LSF · Issue #18 · Snakemake-Profiles/lsf (original) (raw)

I use LSF in two different clusters. In one, this is the LSF version:

$ lsid
IBM Spectrum LSF Standard 10.1.0.6, May 25 2018

Explicitly specifying memory units in this LSF version works just fine, e.g. this command works fine:

bsub -M 1000MB -n 1 -R 'select[mem>1000MB] rusage[mem=1000MB] span[hosts=1]' -o ls.out -e ls.err -J ls_job ls

In the other cluster, the version is a bit older:

$ lsid
IBM Spectrum LSF Standard 10.1.0.0, Jul 08 2016

Explicitly specifying memory units in this cluster fails:

$ bsub -M 1000MB -n 1 -R 'select[mem>1000MB] rusage[mem=1000MB] span[hosts=1]' -o ls.out -e ls.err -J ls_job ls
1000MB: MEMLIMT value should be a positive integer. Job not submitted.

It works if MB is removed:

$ bsub -M 1000 -n 1 -R 'select[mem>1000] rusage[mem=1000] span[hosts=1]' -o ls.out -e ls.err -J ls_job ls
Job <9491039> is submitted to default queue <standard>.

I did not track which update of LSF between these two versions enabled memory units to be specified in -M.

I wonder if:

1. Should we support older versions of LSF? Or should we require users to have an updated version of LSF? Supporting only the most updated version has the advantage of using new features, like this one, but it also means it will work only for a subset of the users. Supporting older versions (e.g. 10.1.0.0) means that a way larger fraction of users can use this profile, but we have to code around the lacking features.
2. We should choose which version to support, and setup a docker image with the chosen version, and make some real end-to-end tests with trivial pipelines to ensure the profile works. The testing framework we have now is fine, but we are mocking the behaviour of LSF. I am hoping LSF is backwards compatible, so we don't actually need to test in every version after the chosen one.