Description
I'm collapsing a few older issues into a single one:
- Improve usage of NVD_API_KEY and Codecov #1961
- Refactoring NVD update tests to avoid network connections. #1977
- CI: Known issue: Windows timeouts #2677
Our tests currently use a cache of NVD that's stored in github, and we've done some work in separating out places where we need network connections, but it's been some time since we looked through it all. We aren't currently using our own mirror for much because it was developed after the tests were last revamped, but if we turn off the NVD_API_KEY on some jobs it will default to that.
I'd like to think about how we should refactor the tests to
- reduce codecov jitter (Improve usage of NVD_API_KEY and Codecov #1961 )
- reduce the number of tests that explicitly need an NVD_API_KEY set (We may want them all in the network-mayfail job, or we may need a windows equivalent of this job)
- Divide out the jobs that can't be run on the cache (Refactoring NVD update tests to avoid network connections. #1977) -- these should be all in the network-mayfail job already but I don't think they are.
- figure out if we should be using the mirror directly in any tests. Maybe the network ones as much as possible?
- improve the stability of the windows test job if possible, which currently has more network fails than the linux jobs (seems to be some sort of issue with the github actions framework for windows). This could involve splitting it into some pieces so the network fails wouldn't affect the rest of hte job, or the long parts of the job (mostly the installs from conda) could be moved into something that could run in parallel.
This is an issue best tackled by someone who's gotten some experience using the test suite, so probably not suitable for a beginner but anyone who's got a few PRs under their belt likely has an opinion about the test suite and could take a stab at it.