Skip to content

saveAll should use Mongo's insertMany method for bulk saving new documents #9143

Open
@pocketcolin

Description

@pocketcolin

New Feature / Enhancement Checklist

Current Limitation

Unless I'm misreading the code, it appears that under the hood the Parse.Object.saveAll method is making many synchronous requests instead of batching them together into a single atomic Bulk request. I see in the MongoStorageAdapter that Parse uses updateMany and even deleteMany, but I don't see insertMany anywhere and the only create function appears to be createObject (singular). The batch endpoint handler also appears to be running the requests separately (here). Please let me know if I'm missing something that we are actually inserting documents in bulk!

Feature / Enhancement Description

By leveraging Mongo's native insert bulk function, we'd reduce number of connections and speed up large requests.

Example Use Case

I need to insert 1000 new documents but want the inserts to happen as close to simultaneously as possible.

Alternatives / Workarounds

Using the Mongo driver directly.

Activity

parse-github-assistant

parse-github-assistant commented on Jun 3, 2024

@parse-github-assistant

Thanks for opening this issue!

  • 🎉 We are excited about your ideas for improvement!
mtrezza

mtrezza commented on Jun 4, 2024

@mtrezza
Member

Haven't looked into this now, but one reason may be what the MongoDB driver returns in its response. Depending on that this may need to be implemented as a new separate Parse method, or can be implemented as an improvement of the existing method.

added
type:featureNew feature or improvement of existing feature
on Jun 4, 2024
pocketcolin

pocketcolin commented on Jun 4, 2024

@pocketcolin
ContributorAuthor

Thanks for the insight, @mtrezza . That definitely makes sense. I started digging into the functionality because we noticed that when running a saveAll on a large array of new documents the new documents would appear in Mongo a few at a time as opposed to all at once. Mongo does some batching under the hood and indexes can take a second to update when large amounts of data is inserted at once, but the tests seem to indicate that at least some of the delay is on Parse's end. That said, when I really dug into the Mongo driver in Parse, I definitely got lost trying to follow the batch paths so it's totally possible I missed something and Parse is batching those requests. If you have time to verify one way or another, I might be able to help out with this improvement (maybe a new optional config param on saveAll?)!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    type:featureNew feature or improvement of existing feature

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

      Development

      No branches or pull requests

        Participants

        @mtrezza@pocketcolin

        Issue actions

          saveAll should use Mongo's insertMany method for bulk saving new documents · Issue #9143 · parse-community/parse-server