Open
Description
What is the problem you are trying to solve?
Presently, blocks that are too large to compact cause compactors to fail with an error, mentioning that blocks are exceeding max size of 64GiB.
Which solution do you envision (roughly)?
In #6588, we added the ability to automatically mark blocks as no-compact when they are irreparably corrupt. We should do something similar for blocks that are too large to compact.
Have you considered any alternatives?
While a sophisticated solution may be to automatically increase shard sizes in this case and re-process these blocks, automatically marking them as no-compact is better than the current status quo.
Any additional context to share?
No response
How long do you think this would take to be developed?
Not sure
What are the documentation dependencies?
No response
Proposer?
No response