Open
Description
Currently known flaky tests (could be a test issue or a nerdctl issue)
Currently being investigated / worked on:
- P0: something is wrong with logs: [CI]: failure nerdctl - TestCreate/logs appears to be flaky #3717
- P1: [CI]: events filter sometimes does not work #4132 - looks like a legit issue, much more likely to happen on EL8/EL9
Hard hitting:
- P0: advanced: CNIEnv concurrency issues #3556
Third-party:
- P0: advanced: [CI]: Windows failure on pull:
UtilityVM: Incorrect function.: unknown#3524 - P0: [CI]: buildkit
Internal: not found#4142 - P1: gomodjail is experimental, and may have pending issues: [CI]: gomodjail target catch-all issues #4182
Possibly EL8 / slow environments specific:
- P1: [LEGACY] [CI]: stty output not always there (EL8 only?) #4119
- P1: [LEGACY] [CI]: ipfs /root/.ipfs/api not available #4118
- P1: [LEGACY] [CI]: TestRestartWithSignal does not seem to (always?) work correctly (EL8 only?) #4068
- P1: [LEGACY] [CI]: NoneNetworkManager is presumably flaky wrt hostname #3971
Kube specific:
- P1 (minor, but annoying): [CI]: Kube failure on commit + save #4106
- to be confirmed this is still happening with containerd v2
Test currently disabled (link to documented issue):
- [URGENT] Regression? TestRunWithTtyAndDetached on windows appears broken on main #3437
- Is IPFS+estargz working? #3475
- see TestNetworkInspect
- test should be re-enabled:
images --filter=non-existentinconsistent (docker vs. nerdctl) #3511 - test should be re-enabled: volume (probably others)
filterbehavior is inconsistent with Docker #3452
"Retry-them" tests that are clearly broken:
- [CI]: flaky with forceful retry: TestSyslogNetwork #4141
- [CI]: OCI runtime start failed: cannot start a container that has stopped: unknown #4143
- [LEGACY] [CI]: TestRunBindMountBind is not working well #4144
- [CI]: TestComposeUpWithExternalNetwork network failure #4146
- [CI]: TestLogsWithoutNewlineOrEOF flaky #4147
Others:
To confirm:
CI structural issues
- P1: [CI]: github cache is too aggressive on limiting requests (429) #3733
- P2: Tests that require
not-rootlessare getting skipped withtarget=docker(where the cli is not run as root) #3453
Tests to add
- Test login, push (and authenticated pull) against docker hub #3494
- Expert: Add testing for OCI hooks #3156
-v SRC:DST:rro,rprivateshould be tested in the CI #1989- CI: test the CRI log viewer #1957
- CI: add test for compose + restart #178
Enhancements: tigron
- [CI/tigron]: investigate file descriptor leak detection as a testing feature #4029
- [CI/tigron]: investigate go routine leak detection as a testing feature #4028
- [Tigron]: roadmap and tracking epic for work on test tooling #3448
Enhancements: generic
- dockerfile specific: [CI]: git clone revamp, part II #4031
- dockerfile specific: [CI] Epic: possible enhancements to current Dockerfile #4021
- [CI]: Discussion: should we drop testing inside a container? #3858
- [CI]: ghcr.io/containerd/stargz-snapshotter:0.15.1-kind needs an arm64 variant #3845
- We should have a publish scripts for the testing images we need, and we should not depend on third party images #3480
- [CI]: Disable test retries #3303
- Add harbor to our integration testing suite? #3257
- fix unit tests so that we can
racethem
Performance tracking
- Latest major disruption: [CI]: State of the CI #3924
- [CI]: Tracking status #3577
Ongoing work
Evergreen playground:
CI work in progress: