r/programming • u/IiIIIlllllLliLl • 3d ago

ClickHouse outcompresses Parquet: Surprises from our migration

https://tomalard.github.io/posts/clickhouse-outcompresses-parquet-surprises-from-our-migration/

I pinky promise that I'm not a ClickHouse employee

113 Upvotes

95% Upvoted

9 comments sorted by

17

u/tecedu 2d ago

Yeah we previously have come to same conclusion, parquet as a format is kinda outdated especially compared to all of the new things out there, but its just portable and everyone uses it and its good enough. Also I think haivng forced encoding might also be hurting as well as engine choice, last I remember with pyarrow vs arrow-rs, pyarrow compressed 14% better for encoding but we never found why?

We have been playing around with good results. https://blosc.org/python-blosc2/reference/ctable.html

Before, we were using blosc on netcdf file before with huuuuge success.

6

u/alexcommacommacomma 2d ago

Great writeup! As sad as it is I don’t see good write ups anymore due to slop

4

u/QueasyEntrance6269 2d ago

Not sure this really matters in practice; you can use MergeTree on S3 but you might as well use Parquet so you're not locked in.

1

u/l_maier-628 11h ago

That depends on whether S3 is the interchange layer or just ClickHouse’s storage: keeping MergeTree for queries and exporting Parquet for downstream consumers can preserve portability without giving up the native layout.

3

u/theAndrewWiggins 2d ago

Would be very cool to see this analysis with lance/vortex/nimble/etc.

1

u/zzulus 2d ago

Using zstd lvl 11 on columnar formats usually doesn't make much sense, because you pay for the X factor increase in the witer cpu time with a very marginal increase in compression rate. You should usually stick with 3 or 7, in fact in some rare cases 1 would outperform 3 & 7 depending on the data shape (eg strings of certain length with repeating prefixes).

Source - I used to support columnar dwh file format libraries.

-4

u/coolcosmos 3d ago

Ctrl+f "brotli": zero match

you can't say Parquet is bigger without trying it's most space efficient compression algorithm....

24

u/IiIIIlllllLliLl 3d ago

We did try brotli at some point, it turned out to be way slower for a marginally better result if I remember correctly. In any case, Parquet already has a compression codec advantage (ZSTD 3 vs 11). I don't think I'm being unfair to Parquet here.

-3

u/TonTinTon 3d ago

Parquet is pretty bad doing zstd always...