Show HN: I built an AI that uses a metacognitive loop 2 solve invention problems (robw1se.substack.com)

> So: the matrix.org database secondary lost its FS due to a RAID failure earlier today (11:17 UTC). Then, we lost the primary at 17:26. We're trying to restore the primary DB FS (which could be fastish), while also doing a point-in-time backup restore from last night (which takes >10h). We believe the incremental DB traffic since last night is intact however. Apologies for the downtime; folks on their own homeserver are of course not impacted.

The stuff of absolute nightmares...

https://mastodon.matrix.org/@matrix/115136245785561439

Bender · 14h ago

10 hours seems like a long time for a db restore of a chat server. Matrix is still just a chat server, right? I have so many questions that maybe I should keep my nose out of.

[Edit] From another comment, 55TB?!? Holy wat-man...

creatonez · 12h ago

There's more than you might think...

- Probably thousands of large chatrooms, and hundreds of millions of small chatrooms

- Probably hundreds of millions of messages that include a media upload like an image or video, including countless re-posts of random memes

- Overhead from ratchet algorithm cryptography, as well as additional message metadata that is likely in JSON format

- Huge excesses of messages from bridge bots, spam bots, and malfunctioning utility bots. To give a sense of scale... the entirety of Libera.chat (formerly Freenode IRC) used to be bridged to matrix.org, meaning almost every single message from Libera would be copied to matrix.org automatically.

- Everything from other homeservers that federate with matrix.org and have been joined by at least one matrix.org user, including homeservers that no longer exist

Arathorn · 1h ago

There's around 50M rooms, and the dataset is back to 2014.

However, much of the room is taken up by the Synapse DB schema being particularly denormalised (prioritising performance over disk footprint) - especially caching snapshots of historical key/value state for rooms, which currently takes up ~65x more space than then actual underlying dataset. Ironically, we're looking into that currently, but not fast enough to speed up this DB rebuild.

Bender · 50m ago

Thanks for that explanation. It makes a lot more sense now. Back in the day I managed a Jabber + bandersnatch corporate server and it had many years of Jabber + AIM + MSN + ICQ + Yahoo chat but the database was tiny which is why I was very surprised to see a chat server take up so much space.

Good luck on getting the schema overhead out of the way. I'm sure nowadays you are probably also using faster underlying storage SSD's behind the raid controllers. Dell/HP keep them overpriced of course but I found them to be very much worth it for databases as did the DBA's.

I hope your on-call teams get to take a week off after that incident.

BrenBarn · 9h ago

Ouch, sounds like bad luck. Best wishes for the recovery.

q3k · 18h ago

> We are in the process of restoring the matrix.org database from a backup. The matrix.org homeserver will be offline until this has been completed

whoops

yabones · 19h ago

Best of luck to the team at Matrix/Element for restoring from a nasty outage.

mostlyk · 18h ago

Hope this gets done sooner, horrible outage

q3k · 18h ago

Now would be a good time to migrate to a different homeserver :).

BrenBarn · 9h ago

Not really possible in a comprehensive way.

jMyles · 15h ago

Love y'all and love matrix. Thanks for the free matrix.org server. But maybe now is the time to research setting up a home server for mission critical stuff?

I don't immediately see an official doc on this; is it right under my nose?

Is this doc good? https://www.redpill-linpro.com/techblog/2025/04/08/matrix-ba...

Arathorn · 14h ago

Matrix.org itself doesn't publish an 'official' way to run a server, given there are multiple implementations and distros out there.

If you're happy using kubernetes, https://element.io/server-suite/community should be a good bet (or https://element.io/server-suite/pro if you are actually doing mission-critical stuff and want a version professionally supported by Element)

If you're happy using docker-compose, then https://github.com/element-hq/element-docker-demo is a very simple template for getting going.

Alternatively, https://github.com/spantaleev/matrix-docker-ansible-deploy is quite popular as a 3rd-party distro using ansible-managed docker containers.

Sorry all for the downtime on matrix.org - we're having to do a full 55TB db restore from backup which will take ~17 hours to run. :|

wind12 · 1h ago

This is the perfect chance to stress that people should choose different homeservers.

But it is hard to trust a random server, if all you know is the name and mean uptime. Mastodon shows the community posts and an introduction by the local admin, before you make an account. Matrix should do the same.

bakugo · 14h ago

Looks like it's going to take a while to come back up.

> Sorry, but it's bad news: we haven't been able to restore the DB primary filesystem to a state we're confident in running as a primary (especially given our experiences with slow-burning postgres db corruption). So we're having to do a full 55TB DB snapshot restore from last night, which will take >10h to recover the data, and then >4h to actually restore, and then >3h to catch up on missing traffic.

https://mastodon.matrix.org/@matrix/115136866878237078

Show HN: Text2SQL with a Graph Semantic Layer (github.com)

Show HN: LightCycle, a FOSS game in Rust based on Tron (github.com)

Show HN: Amber – better Beeper, a modern all-in-one messenger (useamber.app)

Show HN: Moribito – A TUI for LDAP Viewing/Queries (github.com)

Show HN: We built an open-source alternative to expensive pair programming apps (github.com)

Show HN: A hacky app for location sharing without suirvellance (fyrspot.app)

Show HN: My first Go project, a useless animated bunny sign for your terminal (github.com)

Show HN: Multi-Agent-Coder Is #12 on Stanford's TBench. Beats Claude Code (github.com)

Show HN: I built an AI that uses a metacognitive loop 2 solve invention problems (robw1se.substack.com)

Show HN: Davia – A community platform to build, share, and edit applications (docs.davia.ai)

Show HN: Fst – Lightweight C utility for detailed directory statistics LGPL 3.0

Show HN: Tail Lens – Visually edit tailwind css dev tool

Show HN: I built a deep research tool for local file system (github.com)

Show HN: Simple modenized .NET NuGet server reached RC (github.com)

Show HN: Spotilyrics – See synchronized Spotify lyrics inside VS Code (github.com)

Show HN: slack-explorer-mcp – Let AI find historical context in Slack (github.com)

Show HN: An ncurses CUDA-based fluid simulation (github.com)

Show HN: Woomarks, transfer your Pocket links to this app or self-host it (woomarks.com)

Show HN: Provably secure vibe coding is now a thing (secureaf.lovable.app)

Show HN: Lightweight server-driven template language for JavaScript (github.com)

Show HN: Unity WebGL Playground (onejs.com)

Show HN: Hacker News em dash user leaderboard pre-ChatGPT (gally.net)

Show HN: Fine-tuned Llama 3.2 3B to match 70B models for local transcripts (bilawal.net)

Show HN: Open-source AI writing your javadoc (deviantabstraction.com)

Show HN: Anonymous Age Verification (gist.github.com)

Show HN: Sosumi.ai – Convert Apple Developer docs to AI-readable Markdown (sosumi.ai)

Show HN: Whodunit – Solve AI written mysteries (whodunit.rip)

Show HN: Promptproof – GitHub Action to test LLM prompts, catch bad JSON schemas (github.com)

Show HN: MCP Secrets Vault – Local MCP proxy to keep API keys out of LLM context (github.com)

Show HN: PasteVault – An open-source, E2EE pastebin with a VS Code-like editor (pastevault.dev)

Show HN: Blueprint: Fast, Nunjucks-like templating engine for Java 8 and beyond

Show HN: StoryMotion, hand-drawn motion graphics editor based on Excalidraw (storymotion.video)

Show HN: Forward Error Correction for Pion WebRTC (pion.ly)

Show HN: Ruby-TI mruby type analyser (github.com)

Show HN: Zyg – Stop Writing Status Updates (zyg.sh)

Show HN: Neuron – Cognitive Multi-Agent Architecture for Reasoning

Show HN: A founder community with true anonymity(HMAC identities,no socialgraph) (foundermood.zorentia.com)

Show HN: Find Hidden Gems on HN (pj4533.com)

Show HN: I made an Animal Crossing style letter editor (acmail.idreesinc.com)

Show HN: Meetup.com and eventribe alternative to small groups (github.com)

Show HN: I integrated my from-scratch TCP/IP stack into the xv6-riscv OS (github.com)

Show HN: Turn Markdown into React/Svelte/Vue UI at runtime, zero build step (markdown-ui.com)

Show HN: A zoomable, searchable archive of BYTE magazine (byte.tsundoku.io)

Show HN: Base, an SQLite database editor for macOS (menial.co.uk)

Show HN: SwiftAI – open-source library to easily build LLM features on iOS/macOS (github.com)

Show HN: Auto-Match – How We Built Receipt-to-Transaction Matching (Open Source) (midday.ai)

Show HN: Dream Prompter: Bringing Nano Banana to GIMP (thoughts.greyh.at)

Show HN: Async – Claude code and Linear and GitHub PRs in one opinionated tool (github.com)

Show HN: A usercript to help you filter "Who's Hiring". (github.com)

Show HN: FilterQL – A tiny query language for filtering structured data (github.com)

Matrix.org – Database Incident

Comments (15)