> Snapshots are proudly hosted on Cloudflare R2, a storage service optimized for fast, global scalable access to large datasets.
Little curious to hear how this goes for everyone. R2's promise of infinite egress feels huge. But there's some grumbles in HN threads here & there that folks tend to get forced into more expensive contracts after they actually start using R2. Folks downloading ~90GB data sets daily sounds… taxing.
It sounds like OSM themselves offer diffs? I assume there's something about the format of the diffs, something to do with indexing, that makes full daily refreshes compelling? I don't quite understand the excitement for this release; I'm not well versed here & it feels like I'm missing some key technical reasons why this would be popular.
rustc · 6h ago
It would be interesting to see how much you can push this. At $0.36/million requests, this would be $0.36 for 90PB of data. Serving 90PB from AWS S3 would cost over $5M if my calculations are correct. Has anyone tried serving petabytes of data from R2?
jt_b · 3h ago
Not sure about petabytes, but pretty sure Brandon Liu of Protomaps is hosting all the daily protomaps extracts via R2. Not sure what the egress usage is.
Interesting. Even OSM's own server only releases a weekly full planet PBF [0]. However I believe it releases diffs more regularly, which I guess is how OPD is creating a daily planet PBF.
What a great initiative! I need to look more into it to determine how it relates to Geofrabrik's data caches.
I initially thought it was Planet serving up a research grade data cache, which would really help unlock innovation in the earth observation space.
durkie · 6h ago
Wow this is my first time hearing about the Geodesk format. It seems super useful...I do a ton of OSM data processing, and I can't really do anything with it until it's loaded in to PostGIS, so this would save a ton of time/storage space over that.
morgenkaffee · 9h ago
Well done. Every conpany with maps otherwise needs their own daily diff server.
The challenge tho is running it over years reliably especially if something in upstream OSM breaks.
Aachen · 3h ago
How else were you going to do it than ingesting the new data constantly, if you're a company who needs regular updates? You need a server, either to pull the whole planet file over and over or to pull the trickle of minute by minute diffs. Might as well opt for diffs
aeuropean12 · 10h ago
Well done! Great initiative, thumbs up.
ajsnigrutin · 8h ago
Is there some offline viewer for this? Something that would make offline maps (maybe even with search and navigation) possible on computers?
Little curious to hear how this goes for everyone. R2's promise of infinite egress feels huge. But there's some grumbles in HN threads here & there that folks tend to get forced into more expensive contracts after they actually start using R2. Folks downloading ~90GB data sets daily sounds… taxing.
It sounds like OSM themselves offer diffs? I assume there's something about the format of the diffs, something to do with indexing, that makes full daily refreshes compelling? I don't quite understand the excitement for this release; I'm not well versed here & it feels like I'm missing some key technical reasons why this would be popular.
[0] https://planet.openstreetmap.org/pbf/
[0] https://github.com/openplanetdata/osm/blob/924d680ff8df6263f...
I initially thought it was Planet serving up a research grade data cache, which would really help unlock innovation in the earth observation space.
The challenge tho is running it over years reliably especially if something in upstream OSM breaks.