Open
Description
Currently, compatibility testing is error-prone--writing compatibility tests requires a priori knowledge of which VM behavior is prone to breakage, leading to gaps in test coverage. This means that changes to the VM could potentially break compatibility of real-world projects without causing tests to fail.
There should be a way to deterministically record a series of inputs into the VM and play them back to get the same results every time. Once that's in place, "snapshots" of the project state could be taken and compared against future VM versions to ensure that they match. This would allow automated testing to be done on a wide variety of projects simply by playing through them while recording your inputs.
Tasks
- Make the VM deterministic enough to reliably test
- Centralize usage of timer utilities
- Provide a way to swap out all timers with mock timers
- Add a VM mode which runs all projects as fast as possible, faking all timers
- Add recording and playback functionality to the VM
- Wrap IO classes with some sort of recorder?
- Capture which inputs happen in which order in between VM steps
- Replay those inputs in the proper order and during the proper steps in "fast mode"
- Implement a way to compare project states
- Traverse the VM and save things like active costumes, sprite positions, etc.
- Either through existing SB3 serialization or through a custom state-serialization step
- Compare these serialized states with each other, and error out if they differ