Remove redundant rows from subscription updates

[joshua-spacetime]

Description of Changes

Avoids sending trivially empty subscription updates to clients. That is, if a row is inserted n times and deleted n times, we remove it from the result set to avoid network and client side deserialization costs.

API and ABI breaking changes

None

Expected complexity level and risk

1

Testing

• Current regression tests pass
• Bot test

[jsdt]

If this is happening a lot for joins, I assume that is because there are a lot of RHS rows that are modified without falling out of the query range. IIUC, in that case, we will still end up doing the work of joining the new row with the LHS table and joining the old row with the LHS table, only to dedupe the resulting rows from the LHS.

How difficult would it be to push this deduplication into the RHS of the query plan (before the join)? For example, given the query in the test (select u.* from u join v on u.i = v.i where v.x = 5), if we started by computing the delta of select v.i from v where v.x = 5, then joined that with u, we wouldn't need to do any index lookups from u. We also would be deduplicating with the field used for the join, which should be cheaper to hash than the whole result set rows.

[joshua-spacetime]

@jsdt

How difficult would it be to push this deduplication into the RHS of the query plan

This would just require:

1. evaluating inserts and deletes in a single plan (currently they're evaluated separately), and
2. introducing some new operators and optimizations to the query planner

It's not complicated, but certainly not as trivial as this patch.

We also would be deduplicating with the field used for the join, which should be cheaper to hash than the whole result set rows.

For integers yes, hashing the field would be cheaper, however keep in mind that these are row ids, so it's not the same as hashing a product value for instance.

[mamcx]

LGTM