insertMany and updateMany support

[SebC99]

Is there a plan to support insertMany and updateMany in Parse?
It would increase a lot the performance of batch operations, either to create new objects or to update some existing objects.
2 uses cases would be:

• to clone a certain amount of data, by creating 1000 objects at once. For now, saving those 1000 new objects is very very long, even with direct access, as the server starts one database insert op per object.
• to increment a property in 1000 objects at once (or to update a specific field with the same value). For now, the server updates each object one by one, even if the update command is strictly the same.

I have no idea how to implement that so it would be hard to try to propose a PR, but I can see the updateMany is already in the mongo adapter as it's used by the push controller to update the pushStatus of the sent pushes.

[omairvaiyani]

I believe the limitation is the beforeSave hook; Parse allows its developers to validate the data before inserting to the DB. We would have to craft a scenario where the beforeSave is either skipped, or runs once, separately for a batch request, for example a beforeSaveAll. An RFC with problems and alternatives would be needed before something like this is added.

[mtrezza]

IIRC updateMany does not return which objects have been modified. I cannot think of a way around this that would be scalable and maintain the performance gain of updateMany vs. updating individual objects.

So I think it can only be collective hooks like beforeSaveAll as @omairvaiyani suggested. Or no hook at all (initially), which would also be justified because it would still be the cheapest way to update multiple objects.

[SebC99]

Just to give more details, directly using the existing updateMany method of the mongo Adapter results in 0.5s save for 5000 objects instead of 6seconds with the usual parse saveall method...
Of course I skipped the beforeSave, various controls, etc. but still, the gain could be huge.
One way would also be to activate it only if no declared beforeSave?

[mtrezza]

I think updateMany has to be an additional method, it is unlikely that it can increase performance of an existing method.

That’s because updateMany only applies to a specific use case: apply exactly the same modification to objects that are selected by a query, without returning those objects.

Parse SDKs currently have no comparable concept of mass modifying objects that are selected based on a query, in one single step. For example, currently a query fetches objects, you loop through and modify each object, and call saveAll. We would have to determine if the same manipulation has been done to each object of the query to use updateMany, and whether no objects have been added or removed from the list of object we want to update. That doesn't look feasible to me.

So if we decide to implement updateMany, there would be no performance improvement for existing deployments. You’d have to identify queries that are fetching objects, modify them in exactly the same way without any conditional logic, and save them. Only these queries can be replaced with the new updateMany method to improve performance.

A updateMany method will have to bypass any before/after hooks, because the objects are not returned and even if we fetch the objects in an extra step after modifying them, we’d have to apply a collection lock to ensure they have not been modified between the updateMany and find operation. That’s not feasible performance wise.

I don’t see a reasonable use case for a generic beforeSaveAll hook. It defies the purpose of a hook as all we know is that an updateMany command will be executed on the collection.

The PR should be straightforward: add a new method for ParseQuery that takes a modification definition object, which you pass directly to the updateMany command.

let query = new ParseQuery(“_User”);
query.equalTo(“blocked”, true);
let update = { “$set”: { “blocked”: false }};
query.updateAll(update, { useMasterKey: true });

_{Sent with GitHawk}

[SebC99]

I would add a saveAll case which would be an insertMany and not an updateMany'. And a deleteMany` too, as it would provide a way to remove and replace multiple objects at the same time allowing for much faster saving.

[mtrezza]

@SebC99 Sounds good, do you have a suggestion for the syntax? Since we already have a saveAll.

[parse-github-assistant]

The label type:feature cannot be used in combination with type:improvement.