Workflow params definition (first preview)

[bentsherman]

Related to #4669

This PR is the first step towards defining workflow params entirely in the pipeline script. It allows you to define a params block like so:

params {
  input
  save_intermeds = false
}

workflow {
  // ...
}

Instead of assigning individual params. The advantage is that Nextflow can validate params natively once the params block is defined, because it guarantees that all params are declared in one place.

There is still some work required to make the validation work, but the high-level flow is:

1. User specifies params on the command line / params file
2. Config files can override script params or define "config params" which are only used by the config
3. When the params block is defined in the script, config params are ignored and only overrides from the command line / config are applied. If a script param was not specified and has no default value, an error is reported. If a CLI param was not already defined in the config or script, an error is reported

TODO:

• Separate config params from CLI params to identify invalid params (i.e. params that weren't declared in the script or config)

[netlify]

✅ Deploy Preview for nextflow-docs-staging canceled.

Name	Link
🔨 Latest commit	a4f6f70
🔍 Latest deploy log	https://app.netlify.com/sites/nextflow-docs-staging/deploys/6802505b705a58000826931d

[bentsherman]

Regarding the schema generation, there are two approaches we could take:

1. Declare only the param name / type / default value in the script, put everything else in an auxiliary JSON/YAML file
2. Put everything in the script that is used for the schema -- description, icon, form validation rules, section headers, etc

I was initially leaning towards (2), because it would simplify the schema generation and we could validate the available schema properties. But this would also make the params definition really long and verbose in the script, whereas the pipeline code only cares about the name / type / default value.

So now I'm starting to lean towards (1). In that case we could have a really concise definition:

params {
  input: Path
  save_intermeds: boolean = false
}

workflow {
  println "input = ${params.input}"
  println "save_intermeds = ${params.save_intermeds}"
}

Or even directly in the entry workflow:

workflow {
  params:
  input: Path
  save_intermeds: boolean = false

  main:
  println "input = ${input}"
  println "save_intermeds = ${save_intermeds}"
}

This concise syntax will work only if we're certain we don't need anything else in the script. I thought maybe the help text would be useful for CLI help, but that could be provided through a Javadoc comment

In this case, the schema generation would look something like this:

1. Run nextflow schema to initialize a bare-bones JSON schema from the params definition
2. Populate the schema with extra information (help text, icons)
3. Run nextflow schema periodically to update the JSON schema from the params definition, overwriting fields like name / type / schema (perhaps with an appropriate warning)

[ewels]

This concise syntax will work only if we're certain we don't need anything else in the script.

I'm not entirely sure that this is the case, the schema is used for validation of more than just type. I know that some of these things can be handled with Records (eg. enum choices), but what about things like pattern, min/max and uniqueItems etc?

I thought maybe the help text would be useful for CLI help, but that could be provided through a Javadoc comment

I'd be curious to see how this might look - ideally for both description and helptext in one.

[bentsherman]

I know that some of these things can be handled with Records (eg. enum choices), but what about things like pattern, min/max and uniqueItems etc?

Any kind of validation should be possible through custom types and constructor functions (functions that create the custom type and just implements the validation logic). But not all of those cases can be automatically translated to the schema.

For example, I can automatically generate a pattern from a type definition, but not things like min and max. Unless we did something crazy like Min<0, Max<Integer, 10>> 😅

I could go either way at this point. I like the concise syntax of declaring params in the entry workflow, but CLI libraries like argparse are also pretty standard, so maybe the concise syntax is just too restrictive

I'd be curious to see how this might look - ideally for both description and helptext in one.

Copying your example from our slack convo:

workflow {
  params:
  /**
   * Path to comma-separated file containing information about the samples in the experiment.
   *
   * You will need to create a design file with information about the samples in your experiment
   * before running the pipeline. Use this parameter to specify its location.
   * It has to be a comma-separated file with 4 columns, and a header row.
   * See [usage docs](https://nf-co.re/rnaseq/usage#samplesheet-input).
   */
  input: Path

  /**
   * If generated by the pipeline save the STAR index in the results directory.
   *
   * If an alignment index is generated by the pipeline use this parameter
   * to save it to your results folder.
   * These can then be used for future pipeline runs, reducing processing times.
   */
  save_reference: boolean = false

  main:
  // ..
}

[bentsherman]

Using the Javadoc comment is "better" in the sense that you only need to parse the script to produce the CLI help, you don't have to execute it, which i think would be excessive

[bentsherman]

See also Pydantic: https://docs.pydantic.dev/latest/concepts/fields/#validate-default-values

Simple param:

myparam: String = "default-value"

Full param:

myparam: String = Field(default: "default-value", pattern: "/some.*regex/")

[bentsherman]

Declaring params in the entry workflow means that you don't need the params. prefix anymore:

workflow {
  params:
  input: Path
  save_intermeds: boolean = false

  main:
  println "input = ${input}"
  println "save_intermeds = ${save_intermeds}"
}

On the one hand I like that it makes params more like workflow takes. On the other hand, you still need the params. prefix to use params in the config, so I fear that the end result would just be more confusing?

That would suggest that the params block is needed just for consistency with the config. Maybe we could allow the short and long forms like Pydantic:

params {
  input: Path {
    description '...'
    pattern '*.csv'
  }
  save_intermeds: boolean = false
}

workflow {
  println "input = ${params.input}"
  println "save_intermeds = ${params.save_intermeds}"
}

Though I always hesitate to add shortcuts if it makes the code less consistent

[ewels]

+1 for the separate params block, I feel like for consistency that is easier to read and understand, also avoids confusion with take, which is doing quite a similar thing.

Not sure about the squiggly bracket syntax. I like the thinking, but it means that we now have three different types of syntax for them. Nothing for config, : for types and = for variables / others. I can see that being really annoying.

That said, the Field() syntax can be confusing in its own ways, see the Pydantic docs:

Using the f: <type> = Field(...) form can be confusing and might trick users into thinking f has a default value, while in reality it is still required.

[bentsherman]

Yeah, confusing the Field() with a default value is a serious drawback

The block syntax appeals to me because it is consistent with workflow outputs:

// fetchngs...
outputs {
  samples: Channel<Sample> {
    path '...'
    // ...
  }
}

// rnaseq...
params {
  input: List<Sample> {
    // ...
  }
}

Since we want to be able to match outputs to inputs for pipeline chaining, it makes sense to me that the syntax for inputs and outputs mirror each other.

The config is another issue. Let me think through that and write a separate comment...

[bentsherman]

We could add the same params block syntax to config files if consistency is an issue. But I fear this might feel too "weird" in a configuration context.

The nice thing about config params is that they basically have to be simple values (numbers, strings, booleans, etc). So being able to declare the type isn't so important because it can be inferred from the default value.

Meanwhile, if we take the hybrid approach of generating a skeleton schema that the user can annotate manually as needed, we don't need to add new syntax to the config file to support things like validation and help text, because those can just be defined in the JSON schema

[kenibrewer]

I really like this syntax proposal a lot:

params {
  input: Path {
    description '...'
    pattern '*.csv'
  }
  save_intermeds: boolean = false
}

This is likely in part because it mirrors Python/Pydantic but I think that's a good thing for us to emulate. Imitating the syntax of the most popular typed python extension will make it easier for folks to learn and feel like they can read and understand.

[ewels]

If we're leaning more towards that syntax, we could arguably have all JSON schema parameters covered. I think we need to make sure we're crystal clear on what we want to support.

For example, maybe no description as that's "decorative" and too verbose, so the example above becomes simply:

params {
  input: Path { pattern '*.csv' }
  save_intermeds: boolean = false
}

[ewels]

@bentsherman - any ideas how the schema builder might be able to reach into Nextflow code to update these definitions based on changes made in the GUI / JSON?

[bentsherman]

I don't think that will be possible. I think the flow will have to be:

1. Write params definition in main script
2. Generate skeleton JSON schema
3. Extend the schema by hand or via CLI wizard or upload to schema builder
4. Don't edit things that are sourced from the main script

Or if you use the schema builder from scratch, you have to update the params definition by hand (or not use it at all)

[mashehu]

I am just thinking of the following user story: I set the type of a parameter to boolean in the parameter defintion. While writing the param definition in the schema builder I see that the tool actually accepts 0,1, and 2 as values. In the current version of the builder I switch the parameter type in the GUI to integer and set a min and a max value. In the currently proposed setup, I would need to go back to the nextflow code, export a new schema and open the new schema in the builder. not optimal imo.

[bentsherman]

The kind of discovery work you describe (i.e. figuring out the appropriate type of a param) is exactly the kind of thing that needs to happen in the pipeline code, so that the Nextflow compiler can verify param names and types as they are used in the entry workflow. You can't get that kind of validation in the schema builder.

Instead, the schema builder should be used to annotate a fixed set of params with things like help text, icons, form validation rules, etc. It should be primarily concerned with how params are accessed by external systems.

[ewels]

I agree that the logic / definition should be in the pipeline code: at least, that should be the source of truth. I was mostly wondering if we could have some way to update the Nextflow code from the JSON schema builder, to have the best of both worlds. The schema builder has some nice beginner-friendly functionality in it, for example a GUI with a built-in regex tester for writing patterns, and a bunch of built-in help text.

Maybe this is something that we could do with @RefTrace ? eg. From the Python CLI that launches the schema builder GUI, then go back and access the Nextflow code to edit it in place. Not sure if that's possible.

Or if we launch the schema GUI editor from Nextflow itself (with a local server etc) could there be a callback which is able to edit the Nextflow code? 🤔 We would know the param name and attributes..

[bentsherman]

The furthest I would go is to generate a code block in the schema builder that the user can copy into their Nextflow pipeline if they want.. Automatically updating code from an external source is an anti-pattern in my view

[ewels]

Suggestion from call: Would be nice to support single-line comments (//) in addition to Javadoc multi-line comments. This drops the number of lines per-param from 4 to 2, which makes quite a bit of difference if there are a lot of parameters.