Excessively large memory buffers created when sample size is small.

[benjeffery]

At core.py:158 we set the size of the BufferedArray's first dimension to the minimum of the size and chunksize, however, for other dimensions, we just use the chunk size. This came up in #366.

[jeromekelleher]

It's not obvious to me that we're doing that actually - on this line we set the buffer to the shape of the array, not the chunks. So that should be fine?

[benjeffery]

But isn't array.shape here the full array? So in the case of many samples it isn't reducing the buffer to the size of the samples chunk?

[jeromekelleher]

That's the design - we're buffering the full variant chunk.

[jeromekelleher]

OK, thanks to @moshejasper we can see that there definitely is an issue. I'm not convinced that this is the location it's happening though.

[jeromekelleher]

Hmm, looks like it's Zarr that's doing this actually - it'll allocate some arrays that are chunk-sized internally when doing writes.

I can see two solutions:

1. Always clamp the chunk size to min(chunk_size, num_samples)
2. Set the default chunk size to min(num_samples, 10_000)

I guess we shouldn't overrule the user if they really do want to set the chunk size to some value, so I'd vote for option (2)?

[Note: edited this as I got it the wrong way around first time!]

[benjeffery]

Sounds good, makes sense that zarr has to allocate chunk-sized buffers to compress and write them.

[jeromekelleher]

I don't think there's any harm to setting the default chunk size to min(num_samples, 10_000), so I think we go with that in the first instance.

[jeromekelleher]

I'll pick this one up