Ask HN: Is anyone doing intelligent tiering for logs?

We adopted a tiered approach that's working well for us so far:

1. Hot Tier (Last 7 Days): Elasticsearch. For our real-time debugging and immediate operational needs, nothing beats the query speed of Elasticsearch. We keep a rolling 7-day window of all logs here. It's expensive, but essential.

2. Warm Tier (7-90 Days): AWS S3 Standard. After 7 days, our log shipper (Fluentd) automatically archives the logs to S3. If we need to investigate an older issue, we can still query these logs directly using AWS Athena. It's much slower than Elasticsearch, but for occasional, deep-dive investigations, the cost savings are massive.

3. Cold Tier (After 90 Days): S3 Glacier Deep Archive. After 90 days, the logs are transitioned to Glacier Deep Archive via S3's lifecycle policies. This is purely for long-term compliance and "break glass in case of emergency" scenarios. It's incredibly cheap to store, but we know that retrieving it would be a slow and deliberate process.

The key lesson for us was to be realistic about our actual query patterns. We found that over 95% of our queries were for logs less than 3 days old. This data-driven approach allowed us to be aggressive with our tiering strategy without sacrificing critical visibility.