add support for data installations

[smoors]

motivation

• leverage EB to install data in a standardized way with proper versioning and checksumming
• support adding datasets as dependency for software
• easily swap dataset versions with ml swap

changes

• add cmd line option --installpath-data similar to --installpath-software
• add cmd line option --subdir-data (default = data) similar to --subdir-software
• add cmd line option --sourcepath-data similar to --sourcepath
• add Easyconfig parameter data_sources similar to sources

design

• the main reason for a separate subdir_data is reusability: in contrast to software it does not have to be rebuilt/reinstalled when for example upgrading the OS or building for a new architecture
• the reason for a separate sourcepath_data is that datasets can be very large, so you may want to store them in a different file system or location.

[Micket]

Tests needs adjusting
test_prefix_option:

easybuild-framework/test/framework/options.py

Line 5032 in 92d2968

def test_prefix_option(self):

test_show_config:

easybuild-framework/test/framework/options.py

Line 4886 in 92d2968

def test_show_config(self):

[branfosj]

Is there a need to separate data_sources from sources?

[smoors]

data sources can be very big so i think it's good to at least have an option to separate them.

we can set data_sources equal to sources by default, would you prefer that?

[smoors]

done in 2e57ad1 8dbf7d3

[smoors]

hm, i see now that i misunderstood you.
the separate parameter data_sources was suggested by @boegel to make it clear they are different from software sources.
there is no real need for it, it's just cosmetic. i'm not sure if it's a good idea, happy to revert if you prefer.

[smoors]

as discussed during the EUM, we should keep data_sources to allow installing software and datasets in a single easyconfig (e.g. using components) if we want to add support for this later on. this PR does not support this, as it requires substantive changes and i'm unsure how useful it is. i've modified this PR to make it easier to implement support for it if/when desired.

[boegel]

@smoors Let's re-target this to 5.0.x / EasyBuild 5.0, not because it involves breaking changes, but because it can serve as a "flagship" feature of EasyBuild 5.0 (seems a bit too much to introduce in a 4.9.x bug fix release)?

[boegel]

@smoors I changed to target branch in this PR from 5.0.x to develop, you should synchronize your PR branch with current develop branch (which has received a massive update after the release of EasyBuild v5.0.0, see #4825)

requested changes done