Add Pull Request Environments

[waveywaves]

New Feature / Enhancement Checklist

• I am not disclosing a vulnerability.
• I am not just asking a question.
• I have searched through existing issues.

Current Limitation

There are no previews for Pull Requests on the parse repository.

Feature / Enhancement Description

On every PR to parse-server, create an Uffizzi Preview of the parse server which runs against an instance of the parse-dashboard and an instance of mongodb. This will require the repo to be configured with Uffizzi.

Example Use Case

Alternatives / Workarounds

3rd Party References

I would like to make life easier for Parse maintainers by implementing Uffizzi previews.
Disclaimer: I work on Uffizzi
Uffizzi is a Open Source full stack previews engine and is completely free for Parse (and all open source projects). This will provide maintainers with previews of their PRs in the cloud, allowing them iterate faster and reduce time to merge.

TODO:

• Intial POC

[parse-github-assistant]

Thanks for opening this issue!

• 🎉 We are excited about your ideas for improvement!

[mtrezza]

Interesting! Could you fill out the complete template, to provide a better basis for discussion?

[waveywaves]

@mtrezza I have just filled out the template. Let me know what you think.

[mtrezza]

Thanks! Could you give some examples for how this would be used? Is this helpful for the creator of the PR, or for someone who only reported an issue but did not create the PR?

I would imagine for example:

1. user A reports an issue
2. user B fixed it in a PR
3. user B asks user A to verify that the issue has been fixed
4. user A can verify by playing around with the deployed API that contains the fix

Is that the main use case?

[waveywaves]

Hey @mtrezza

Once a Pull Request is created, a preview of the pull request would be created alongside it and then shown on the PR through a comment. This PR will be public and accessible to anyone, as long as they have the link and the PR on the repo is open.

This would be helpful for the maintainer mostly and to also the creator of the PR. The maintainer will get a instance of the preview built from the PR and check directly if the code changes work. The contributor would also be able to check and iterate on the PR by seeing if their changes reflect well in the preview.

user A who has reported the issue should be able to comment on the change they see in the preview and let user B know if they have met the criteria of their request. user A can definitely pay around with the deployed API that contains the fix.

[mtrezza]

Interesting; Parse Server has options that are provided on start-up; some of the features depend on the options set. How would that work? I assume the Parse Server that is spun-up would only have the default options set?

[mtrezza]

Also, how does this compare to Gitpod.io? Is that a similar concept?

[waveywaves]

@mtrezza we can figure out a solution where you can spin up the parse server and test it with whatever options you would like to. We have another ongoing PoC where we are testing this out.

It’s important to not that Gitpod is for development environments in the cloud, whereas Uffizzi sits helps create previews of the Pull Requests specifically is designed for the maintainer primarily.

[mtrezza]

The maintainer will get a instance of the preview built from the PR and check directly if the code changes work.

I can see how this would be beneficial for a low-config and primarily visual product like Parse Dashboard. I'm still unsure how this would practically work for a high-config and primarily non-visual product like Parse Server, where behavior is much depending on a complex combination of config start-up options and very specific test data in the DB, and it's not something the user can necessarily test in a UI.

Let's take a random open PR like #8337 for example. How would this assist in developing that PR?

[ShrutiC-git]

@mtrezza, really appreciate the logical questioning.

Going by the example of the PR you mentioned (#8337):

How would testing and approving the change look without Uffizzi?

• The contributor would test it locally on their system with the needed configuration. Since this is a non-UI change, this would mean testing some operations within mongo with a particular dataset, let's say.
• If the contributor feels the changes are good to go, they'll open a PR
• To test and verify these changes, the maintainer will have to
  • first look at the changes in code. But, this does not always give a good idea of the effects
  • the maintainer might want to test the changes locally and on dev/test environments too. They might want to test their queries with different configs.
  • to test locally, the maintainer would pull the branch locally and test the changes with different config allowDiskUse: true / allowDiskUse: false on a particular test dataset.
  • once tested locally, the maintainer might want to test on a main-like dev/test environment.
  • testing on dev/test envs, might first need to resolve conflicts
  • testing on dev/test envs also does not guarantee that the behavior you are seeing corresponds with the changes you are testing, primarily because dev/test environments are centralized, and often polluted.
  • the PR might require changes. Once the changes are made, to reconfirm if the change is working as needed, the whole process must be repeated.

---

How would testing this PR look like with Uffizzi Previews (or any PRs where the changes are not necessarily easy to test through UI)

• The contributor would test it locally on their system with the needed configuration.

• If the contributor feels the changes are good to go (in this case, the allowDiskUse works as expected), they would open a PR

  • the contributor could seed the DB with test data which would allow the maintainer to easily run queries against this changed operation.
  • To test and verify these changes, the maintainer would simply have to run their test queries against this production-like, clean environment.
  • If any changes are needed, the preview-environment will simply be updated and the maintainer can visit this clean environment and test parse-server how they would when working on it locally/in-production.

• The only thing that might be needed to test this PR would be a seeded mongo DB. We can perhaps add support for the DBs parse-support, so on the side of the contributor, the only thing they would have to edit is mounting a test init-SQL script.

• The code for parse-server is built from source, and that will make sure changes in the server are reflected in how the server behaves.

• In case specific start-up operations are needed, too, to set up the Parse-server environment in a specific way, those changes can go in the docker-compose.uffizzi.yml. So for this example, that would mean adding a MongoDB instance to the docker-compose.uffizzi.yml, seeding it with the init-SQL script, and then starting the PR → this would build Parse-server with a MongoDB server (in addition to whatever services are defined in the docker-compose.uffizzi.yml.

• The main benefit you get from this is Uffizzi is simplifying your testing and verifying process; it is enhancing developer productivity, because the constant back-and-forth between testing and verifying is eliminated, and increasing release velocity because changes can be tested faster.

• Testing will look the same for both testing it locally and testing it on Uffizzi — for example, running a query with allowDiskUse. What is changed, however, is that testing this change will not require pre- and post- setups. Plus, the merging will be faster, because the changes have been approved in essentially a vacuumed production instance.

[mtrezza]

Please correct me if I don't grasp this right, but our contribution and review process works very different and more automated than how you describe it.

the maintainer might want to test the changes locally and on dev/test environments too. They might want to test their queries with different configs.

the contributor could seed the DB with test data which would allow the maintainer to easily run queries against this changed operation.

A maintainer (or reviewer) should never do any testing on their own. Neither with a local clone of the PR, nor against an environment that is spun up by Uffizzi. Such custom tests would be outside of the CI, non-repeatable and undocumented. Our policy demands that tests are an integral part of the PR. In other words, we wouldn't want a reviewer to do their own testing, neither would it be feasible for an organization of our size. We have a clear separation of roles. A contributor's role is to write a code change, manually test it and write repeatable CI tests as part of their PR. A reviewer's role is to ensure that the tests cover the change sufficiently, not to do manual testing.

The only thing that might be needed to test this PR would be a seeded mongo DB. We can perhaps add support for the DBs parse-support, so on the side of the contributor, the only thing they would have to edit is mounting a test init-SQL script.

testing on dev/test envs also does not guarantee that the behavior you are seeing corresponds with the changes you are testing, primarily because dev/test environments are centralized, and often polluted.

We do not have a seeded database for testing. A test (group) seeds the database as part of the test definition. This is important, so a test can start with an empty DB and for the tests not to influence / depend on one another. No pollution there.

The main benefit you get from this is Uffizzi is simplifying your testing and verifying process

I don't see this corroborated so far. Quite the contrary. The contribution and review process you described seems to involve more manual steps and blur roles and separation of tasks between contributor and reviewer.

Again, the only benefit I see so far is a for a visual product like Parse Dashboard, where we don't have UI tests (yet). A reviewer could look at the more subtile UI changes, which is something a UI test can hardly evaluate. So even if we have UI tests (for which we have an open issue there), just taking a quick look at UI-related changes could be beneficial for process and save the reviewer some time.

[jpthurman]

Hi @mtrezza
I'm one of the founders at Uffizzi and discussed this integration with @ShrutiC-git. Thanks for helping us understand the nuances of your contribution and testing workflow.

Based on the discussion, here's what I would propose. We can close this PR to parse-server and open a PR to parse dashboard - which sounds like would provide the most value to your project now.

If this turns out to be beneficial, we can look at evolving the capability to facilitate some additional testing layers like interface to database and/or front end automation testing across both repositories - as it makes sense with your project goals.

How does this sound?

cc @ShrutiC-git