Fields where Native Americans farmed a thousand years ago discovered in Michigan (smithsonianmag.com)

Regular zstd compressed files consist of a single frame, meaning you have to start decompression at the beginning. The seekable format splits compressed data into a series of independent frames, each compressed individually, so that decompression of a section in the middle of an archive only requires zstd to decompress at most a frame's worth of extra data, instead of the entire archive.

I started working with the seekable format because I wanted to resume downloads of big zstd compressed files that are decompressed and written to disk on the fly. At first I created and used bindings to the C functions that are available upstream[1], however, I stumbled over the first segfault rather quickly (it's now fixed) and found out that the functions only allow basic things. After looking closer at the upstream implementation, I noticed that is uses functions of the core API that are now deprecated and it doesn't allow access to low-level (de)compression contexts. To me it looks like a PoC/demo implementation that isn't maintained the same way as the zstd core API, probably that's also the reason it's in the contrib directory.

My use-case seemed to require a complete rewrite of the seekable format, so I decided to implement it from scratch in Rust using bindings to the advanced zstd compression API, available from zstd 1.4.0.

The result is a single dependency library crate[2], and a CLI crate[3] for the seekable format that feels similar to the regular zstd tool.

Any feedback is highly appreciated!

[1]: https://github.com/facebook/zstd/tree/dev/contrib/seekable_f... [2]: https://crates.io/crates/zeekstd [3]: https://github.com/rorosen/zeekstd/tree/main/cli

Comments (11)

rwmj · 48m ago

Seekable formats also allow random reads which lets you do trickery like booting qemu VMs from remotely hosted, compressed files (over HTTPS). We do this already for xz: https://libguestfs.org/nbdkit-xz-filter.1.html https://rwmj.wordpress.com/2018/11/23/nbdkit-xz-curl/

Has zstd actually standardized the seekable version? Last I checked (which was quite a while ago) it had not been declared a standard, so I was reluctant to write a filter for nbdkit, even though it's very much a requested feature.

ncruces · 13m ago

How's tool support these days to create compress a file with seekable zstd?

Given existing libraries, it should be really simple to create an SQLite VFS for my Go driver that reads (not writes) compressed databases transparently, but tool support was kinda lacking.

Will the zstd CLI ever support it? https://github.com/facebook/zstd/issues/2121

simeonmiteff · 1h ago

This is very cool. Nice work! At my day job, I have been using a Go library[1] to build tools that require seekable zstd, but felt a bit uncomfortable with the lack of broader support for the format.

Why zeek, BTW? Is it a play on "zstd" and "seek"? My employer is also the custodian of the zeek project (https://zeek.org), so I was confused for a second.

[1] https://github.com/SaveTheRbtz/zstd-seekable-format-go

rorosen · 29m ago

Thanks! I was also surprised that there are very few tools to work with the seekable format. I could imagine that at least some people have a use-case for it.

Yes, the name is a combination of zstd and seek. Funnily enough, I wanted to name it just zeek first before I knew that it already exists, so I switched to zeekstd. You're not the first person asking me if there is any relation to zeek and I understand how that is misleading. In hindsight the name is a little unfortunate.

etyp · 1m ago

Zeek is well known in "security" spaces, but not as much in "developer" spaces. It did get me a bit excited to see Zeek here until I realized it was unrelated, though :)

stu2010 · 1h ago

This is cool, I'd say that the most common tool in this space is bgzip[1]. Have you thought about training a dictionary on the first few chunks of each file and embedding the dictionary in a skippable frame at the start? Likely makes less difference if your chunk size is 2MB, but at smaller chunk sizes that could have significant benefit.

[1] https://www.htslib.org/doc/bgzip.html

jeroenhd · 1h ago

Looking at the spec (https://github.com/facebook/zstd/blob/dev/contrib/seekable_f...), I don't see any mention of custom dictionaries like you describe.

The spec does mention:

> While only Checksum_Flag currently exists, there are 7 other bits in this field that can be used for future changes to the format, for example the addition of inline dictionaries.

so I don't think seekable zstd supports these dictionaries just yet.

With multiple inline dictionaries, one could detect when new chunks compress badly with the previous dictionary and train new ones on the fly. Could be useful for compressing formats with headers and mixed data (i.e. game files, which can contain a mix of text + audio + video, or just regular old .tar files I suppose).

throebrifnr · 19m ago

Gz has --rsyncable option that does something similar.

Explanation here https://beeznest.wordpress.com/2005/02/03/rsyncable-gzip/

tyilo · 47m ago

I already use zstd_seekable (https://docs.rs/zstd-seekable/) in a project. Could you compare the API's of this crate and yours?

tyilo · 42m ago

Correct me if I'm wrong, but it doesn't seem like you provide the equivalent of Seekable::decompress in zstd_seekable which decompresses at a specific offset, without having to calculate which frame(s) to decompress.

This is basically the only function I use from zstd_seekable, so it would be nice to have that in zeekstd as well.

77pt77 · 17m ago

BTW, something similar can be done with zlib/gzip.

Working on databases from prison: How I got here pt. 2 (turso.tech)

Show HN: Zeekstd – Rust Implementation of the ZSTD Seekable Format (github.com)

Start your own Internet Resiliency Club (bowshock.nl)

Show HN: Socket-call – Call socket.io events like normal JavaScript functions (github.com)

Nanonets-OCR-s – OCR model that transforms documents into structured markdown (huggingface.co)

LLM Chat via SSH (github.com)

Solving LinkedIn Queens with APL (pitr.ca)

Infracost (YC W21) is hiring software engineers (GMT+2 to GMT-6) (infracost.io)

Maya Blue: Unlocking the Mysteries of an Ancient Pigment (mexicolore.co.uk)

Jokes and Humour in the Public Android API (voxelmanip.se)

Why SSL was renamed to TLS in late 90s (2014) (tim.dierks.org)

DARPA program sets distance record for power beaming (darpa.mil)

Modifying an HDMI dummy plug's EDID using a Raspberry Pi (downtowndougbrown.com)

Childhood leukemia: how a deadly cancer became treatable (ourworldindata.org)

Real-time CO2 monitoring without batteries or external power (news.kaist.ac.kr)

Twin – A Textmode WINdow Environment (github.com)

Datalog in Rust (github.com)

Chemical knowledge and reasoning of large language models vs. chemist expertise (nature.com)

First 2D, non-silicon computer developed (psu.edu)

Hyperspectral scans of historical pigments and painting reconstructions (github.com)

How to modify Starlink Mini to run without the built-in WiFi router (olegkutkov.me)

The Hewlett-Packard Archive (hparchive.com)

Fields where Native Americans farmed a thousand years ago discovered in Michigan (smithsonianmag.com)

Telephone Exchanges in the UK (telephone-exchanges.org.uk)

Datalog in miniKanren (deosjr.github.io)

The long afterlife of a literary classic (thecritic.co.uk)

Simplest C++ Callback, from SumatraPDF (blog.kowalczyk.info)

Foundations of Computer Vision (2024) (visionbook.mit.edu)

Is Gravity Just Entropy Rising? Long-Shot Idea Gets Another Look (quantamagazine.org)

Reinventing circuit breakers with supercritical CO2 (spectrum.ieee.org)

Lisp-stat: Lisp environment for statistical computing (lisp-stat.dev)

The Art of Lisp and Writing (2003) (dreamsongs.com)

Cyborg Embryos Offer New Insights into Brain Growth (spectrum.ieee.org)

Accumulation of cognitive debt when using an AI assistant for essay writing task (arxiv.org)

Text-to-LoRA: Hypernetwork that generates task-specific LLM adapters (LoRAs) (github.com)

Unprecedented optical clock network lays groundwork for redefining the second (phys.org)

Canyon.mid (canyonmid.com)

Ruby on Rails Audit Complete (ostif.org)

SQLite Date and Time Functions (2007) (www2.sqlite.org)

The experience continues until you stop experiencing it (strangemachine.tv)

Cure Dolly's Japanese Grammar Lessons (kellenok.github.io)

A skyscraper that could have toppled over in the wind (1995) (newyorker.com)

Believing you only have one option is dangerous (clearerthinking.org)

The Illusion of the Illusion of Thinking – A Comment on Shojaee et al. (2025) (arxiv.org)

How fast can the RPython GC allocate? (pypy.org)

Tiny-diffusion: A minimal implementation of probabilistic diffusion models (github.com)

An origin trial for a new HTML <permission> element (2024) (developer.chrome.com)

Meta's Llama 3.1 can recall 42 percent of the first Harry Potter book (understandingai.org)

An Introduction to the Hieroglyphic Language of Early 1900s Train-Hoppers (openculture.com)

It’s nearly impossible to buy an original Bob Ross painting (2021) (thehustle.co)

Show HN: Zeekstd – Rust Implementation of the ZSTD Seekable Format

Comments (11)