Remember when heroku outages would reliably make the HN frontpage? Apparently not anymore?
My app is still up, but heroku dashboard is down for me now -- and (re)starting/stopping dynos may not be working. (hirefire is complaining on and off that it can't scale workers).
In the past, with this kind of outage, apps have often stayed up as long as they don’t try to redeploy, which sometimes takes them irreparably down. So... recommend you don't try to deploy!
carloscheddar · 8h ago
I went in fully expecting it to be the top result like how it's been in the past but couldn't find it; sorted by new, couldn't find it. It seems like Heroku is not what it once was.
thejosh · 7h ago
"May your SaaS never be purchased by Salesforce."
jeremyjh · 4h ago
That happened 15 years ago.
nomilk · 5h ago
The baffling lack of candor from Heroku is alarming! At first their status page showed absolutely no problems. Some time later it received updates to acknowledge an incident. Why not go out of your way to be up front?
I feared this may be hackers of something much worse than heroku/salesforce is letting on, due to the complete lack of transparency.
But this line in the salesforce status page makes me think it isn’t something of that nature:
> Our efforts remain focused on internal testing and validation as we continue to see incremental improvements
“Testing and validation” isn’t something you’d do (or say you’re doing) during a hack.
Why don’t companies learn to communicate outages in order to help their customers? Even the heroku status twitter account has posted twice in twelve hours. There’s no excuse for such opaqueness.
ojame · 18h ago
We host our app on Heroku - the core service and a few microservices. The heroku dashboard is down, and we can't use the cli due to the in-browser auth callback that it needs (which is also down).
The status page itself is either saying nothing is wrong, or points to an error page[0]. The incident itself[1] hasn't been updated, which is pretty frustrating.
We can't submit a support ticket because, well, it requires the authentication procedure as well.
We use worker queues, and the queues are getting blown out because heroku can't action anything. We're having our microservices yo-yo now, which suggests things are getting worse, now better.
I've always been a huge Heroku advocate, but the last 5 years have been death by a thousand cuts.
This feels like the last paper cut for me. It's easy enough to keep using Heroku because it's "easy for the team," even though it is overpriced. But when they can't even acknowledge an incident and there is no way to log in and file a ticket, it feels like they have failed.
Which is odd, heroku I'd think would be pretty good at keeping it's status page infrastructure separate enough to stay up. Must mean something pretty fundamental in their architecture is malfunctioning. :(
but when I am able to see the error page, it did say "Heroku continues to investigate and remediate an issue with intermittent outages" -- I would say it is acknolwedged. Yes, that message is 3 hours old. The fact that it's taking them over 3 hours to fix is disturbing, but getting contant progress communication isn't really urgent for me -- I know they know about it, I know they are working to fix it, I'd like them to fix it _quicker_ but I don't need a play-by-play, "can't even acknowledge an incident" is NOT a problem being exhibited, it's acknowledged.
We'll wait and see what it was. A good retrospective write-up goes a long way to increasing many people's confidence, including mine.
senfiaj · 9h ago
I wonder whether heroku is some kind of monolith, if something crashes, the whole thing crashes.
jrochkind1 · 10h ago
In general, I was getting death-by-a-thousand-cuts pessmisim about heroku, but I'm really optimistic about the new Fir stack, which will fix some of my biggest complaints.
Stability definitely still matters though, of course.
senfiaj · 9h ago
The cli doesn't seem to work anyways, I can't access or restart dynos.
"Lobsters is hosted on three VPSs at DigitalOcean: a s-4vcpu-8gb for the web server, a s-4vcpu-8gb for the mariadb server, and a s-1vcpu-1gb for the IRC bot. DNS is hosted at DNSimple and we use restic for backups to b2. (Setup details are available in our ansible repo.) Lobsters is cheap to run, so we don't take donations."
fside · 19h ago
It’s amazing. We cannot even submit a support ticket, because they require you to login and the incident is on the authentication path. You think such a company wouldn’t introduce a circular dependency at such a critical path…
Nextgrid · 18h ago
Counterpoint: what's the point of submitting a ticket in this case? With an incident of such magnitude, do you think they don't know?
ojame · 17h ago
It's also a good paper-trail if you have an SLA. Pointing to a ticket saying "this is when we knew" makes time calculation for compensation very trivial and transparent. Of course if your salesforce account manager is like most, they don't need such "evidence" but in the past it has helped me.
tough · 10h ago
wouldn't any email suffice in such a case as proof?
fside · 17h ago
Well, we have realized the issue like 1 hr before they shared anything in their pages. That 1 hour feels forever. Essentially, I want my problem to be acknowledged from a service provider which they failed to do so in their status page.
sleepyhead · 11h ago
Their EU region was down for 8 hours last November and it took 2 hours before they were aware of it so submitting a ticket is definitively worth while. I'm suspecting their monitoring is not good enough.
jrochkind1 · 10h ago
They are definitely aware of it, although the status page is having trouble loading, you should get it eventually.
If their triage system is good than overwhelming them with duplicate non-specific "things are wonky" error reports might not hurt, but it def doesn't help.
Heroku continues to investigate and remediate an issue with intermittent outages.
Posted 3 hours ago, Jun 10, 2025 14:20 UTC
Issue
Beginning at 06:03 UTC, Heroku is having intermittent outages which are currently being investigated.
Posted 4 hours ago, Jun 10, 2025 13:07 UTC
Investigating
Engineers are continuing to investigate an issue accessing Heroku services.
Posted 4 hours ago, Jun 10, 2025 12:58 UTC
Investigating
Engineers are continuing to investigate an issue accessing Heroku services.
Posted 8 hours ago, Jun 10, 2025 09:19 UTC
Investigating
Engineers are investigating an issue with the Heroku platform.
Posted 9 hours ago, Jun 10, 2025 08:04 UTC
joeldrapper · 17h ago
With no updates to their status page or social media accounts, it’s possible they don’t know.
graypegg · 2h ago
Had a sudden urge to change something on a side project I touch pretty irregularly, and realized I can't even deploy to it. Joy. Just getting "release finished" with no log output. This is the only thing I have on Heroku thankfully.
emilsedgh · 10h ago
Our Enterprise Account manager was denying there's an issue. Claiming it's only us, I should open up a ticket, etc.
jrochkind1 · 7h ago
WTF, it's definitely on the status page -- if you can get to the status page, which, the status page having an outage is an alarm sign?
It has been there for over 9 hours now.
I would definitely shame them -- do they not even bother looking at status pages before giving you information? How has Heroku not distributed this information to any customer-contacting staff already?
It's crap like this that disturbs me even more than the outage in fact. I know bad outages can happen even to the best of us, but your account manager not knowing about it and not even bothering to try to find out before giving you bad information is the mark of mediocrity at best.
nomilk · 6h ago
Status page definitely showed ‘all green’ (no issues) when I checked after noticing database backups (via the heroku CLI) were failing.
amackera · 10h ago
Very worrying. This outage is serious and we're getting almost nothing from the official channels.
jrochkind1 · 7h ago
After 9 hours, I'd have thought a customer email would be in order. Unless they are so down they can't even do that?
ageitgey · 18h ago
I have several services on Heroku, and they are up and down randomly. So it's definitely not just the Heroku admin layer that is down.
Access to check logs or perform other Heroku CLI functions is very intermittent. Some of our services work sometimes. Others don't. When I can occasionally access the logs, it seems like there are issues pinging other services. The apps aren't able to push their logs to DataDog.
It's a mess, and there's not much we can do right now, which is frustrating. And the fact that the Heroku status page itself is broken is embarrassing.
ikeshare · 6h ago
Is there any update on what exactly is down? Their status page is useless, and we've seen everything from the CLI being down, to the dashboard, to parts of our apps seeming to work "sometimes". Any consensus? Also not sure if there is a correlation with Salesforce itself
jeremyjh · 4h ago
For us it was only API down most of the day. We didn't have to deploy anything or fix anything, so we almost got away without downtime.
But then, about 7 hours in (~1:40 PM EDT) for about 30 minutes all web requests were failing with a 503, logged as H99 "Platform error". No requests made it to our dyno during that period.
We can't failover to contingency without making API calls - the read replica has to be forked and promoted. I've always comforted myself by saying "well, the data plane and control plane have always been separate and aren't impacted by the same events". No longer.
My guess is this may have been some kind of trouble-shooting that caused this issue, but it doesn't matter. The point is I can no longer ignore this global single point of failure and really the only answer is to migrate away. Which I am reluctant to do, and I don't know that it will increase reliability.
We have never had application downtime as a result of a Heroku issue in the last 8 years this has been in production. 30 minutes in 8 years isn't bad. But if it were down for 3 days I'd have the same options: fuck and all.
mccorkle · 6h ago
Our application which depends on heroku connect has started responding again, so at least some services are coming back. No messaging from Heroku or salesforce yet.
fiatpandas · 6h ago
Slight correlation to Cisco Live conference happening right now. Heroku network admins preoccupied?
We have apps in EU in private spaces and most of them are down.
drrotmos · 18h ago
We have multiple apps across both Common Runtime (EU) and Private Spaces (eu-west-1). It's a mixed bag. Some are up, some are down.
cranberryturkey · 19h ago
Heroku is just a slick frontend for AWS.
cornfieldlabs · 19h ago
I understand that but I think only that "front-end" is down
Our servers are up till now
cranberryturkey · 18h ago
people are reporting otherwise. depends on your geolocation. i think its an aws outage for us-east-1 or something i saw further up.
jmuguy · 11h ago
Still having issues, had a deploy (luckily just to staging) that seemed like it went through but old code still present in app. Can't restart dynos, cli commands sometimes work, sometimes don't.
senfiaj · 6h ago
Looks like there is a limited access from CLI, although CLI doesn't work reliably. I ran heroku ps:restart multiple times and my site came back to life, although there are errors in the console.
Adn201 · 6h ago
It has been long time now since heroku is down, I do not think they take seriously , the status update is too mediocre, o interest in keep informed to the users and provide an ETA, the whole support team seems is in India, who always gave me bad answers on my tickets. Heroku down for more than 14 hours is something to consider heroku as not serious platform or some platform for a production env. stay away of heroku.
Our app is not working and we can't access the console to see the logs. External logs say that it can't access some external systems it depends on.
And the official Heroku status page says it's all fine.
cornfieldlabs · 19h ago
I was pushing a small update to my tiny niche social network with
~10 users so no real damage was done.
I feel bad for companies with a lot of users
carloscheddar · 9h ago
Can confirm that my apps are up and working normally but the Heroku dashboard and services fail to connect. At this point if I manage to get something like review apps somewhere else I'm jumping ship.
sebslomski · 14h ago
Very poor communication on the major issue by Salesforce. Keep in mind, it affects Salesforce' entire cloud infra.
My best guess: The authentication issue affects them more than it does us.
jesuscanwalk · 14h ago
Mmm. We've had an issue with our private-space since this morning with the following issues:
1. Our platform can't access any external API services from that dyno specifically.
2. Can't get into dashboard.heroku.com to login to our services/infrastructure.
3. heroku-cli doesnt work so cant access any logs from our dyno.
4. We're an enterprise client and had next to no update or no response from our account manager albeit we're right at the bottom of the pecking order but no reason.
that is a plain weird message. single server of what?
jmuguy · 6h ago
Salesforce status page is down now too. That’s an entire day lost for us. I’ve been pushing our team to finally ditch Heroku, maybe this will be what we need to be done with them.
fiatpandas · 5h ago
What would you move to?
johtso · 11h ago
My long running worker still seems to be working, but my scheduled task has stopped running.
We’ve identified the likely trigger of the issue and are making strong progress toward a resolution. A recent automated update disrupted network connectivity on affected instances.
ChrisArchitect · 5h ago
> 21:48:25 UTC
The Heroku dashboard is now back online. Customers can now attempt to recycle their dynos on the dashboard.
Hackbyrd · 12h ago
Jesus christ guys, its still down and even the status page is down!
Not anymore! Now the Salesforce status page just says:
> The Salesforce Trust site is currently experiencing a service disruption. During this time, users will be unable to access the Trust site or receive notifications about service-impacting incidents or maintenances. We will continue to provide updates here until the issue is resolved.
And the heroku status page actually has recent upates, although they are just repeating this every hour:
> Heroku continues to investigate and remediate an issue with intermittent outages.
My app is still up, but heroku dashboard is down for me now -- and (re)starting/stopping dynos may not be working. (hirefire is complaining on and off that it can't scale workers).
In the past, with this kind of outage, apps have often stayed up as long as they don’t try to redeploy, which sometimes takes them irreparably down. So... recommend you don't try to deploy!
I feared this may be hackers of something much worse than heroku/salesforce is letting on, due to the complete lack of transparency.
But this line in the salesforce status page makes me think it isn’t something of that nature:
> Our efforts remain focused on internal testing and validation as we continue to see incremental improvements
“Testing and validation” isn’t something you’d do (or say you’re doing) during a hack.
Why don’t companies learn to communicate outages in order to help their customers? Even the heroku status twitter account has posted twice in twelve hours. There’s no excuse for such opaqueness.
The status page itself is either saying nothing is wrong, or points to an error page[0]. The incident itself[1] hasn't been updated, which is pretty frustrating.
We can't submit a support ticket because, well, it requires the authentication procedure as well.
We use worker queues, and the queues are getting blown out because heroku can't action anything. We're having our microservices yo-yo now, which suggests things are getting worse, now better.
I've always been a huge Heroku advocate, but the last 5 years have been death by a thousand cuts.
0: https://status.heroku.com/error 1: https://status.heroku.com/incidents/2822
Which is odd, heroku I'd think would be pretty good at keeping it's status page infrastructure separate enough to stay up. Must mean something pretty fundamental in their architecture is malfunctioning. :(
but when I am able to see the error page, it did say "Heroku continues to investigate and remediate an issue with intermittent outages" -- I would say it is acknolwedged. Yes, that message is 3 hours old. The fact that it's taking them over 3 hours to fix is disturbing, but getting contant progress communication isn't really urgent for me -- I know they know about it, I know they are working to fix it, I'd like them to fix it _quicker_ but I don't need a play-by-play, "can't even acknowledge an incident" is NOT a problem being exhibited, it's acknowledged.
We'll wait and see what it was. A good retrospective write-up goes a long way to increasing many people's confidence, including mine.
Stability definitely still matters though, of course.
* Heroku and most of Salesforce
* Pipedrive: https://news.ycombinator.com/item?id=44234098
* OpenAI: https://status.openai.com/incidents/01JXCAW3K3JAE0EP56AEZ7CB...
* Lobsters: https://news.ycombinator.com/item?id=44234075
What's the common denominator?
From the last archived version of lobste.rs/about: https://web.archive.org/web/20250601002505/https://lobste.rs...
"Lobsters is hosted on three VPSs at DigitalOcean: a s-4vcpu-8gb for the web server, a s-4vcpu-8gb for the mariadb server, and a s-1vcpu-1gb for the IRC bot. DNS is hosted at DNSimple and we use restic for backups to b2. (Setup details are available in our ansible repo.) Lobsters is cheap to run, so we don't take donations."
If their triage system is good than overwhelming them with duplicate non-specific "things are wonky" error reports might not hurt, but it def doesn't help.
https://status.heroku.com/incidents/2822
Update
Heroku continues to investigate and remediate an issue with intermittent outages.
Posted 3 hours ago, Jun 10, 2025 14:20 UTC
Issue
Beginning at 06:03 UTC, Heroku is having intermittent outages which are currently being investigated.
Posted 4 hours ago, Jun 10, 2025 13:07 UTC
Investigating
Engineers are continuing to investigate an issue accessing Heroku services.
Posted 4 hours ago, Jun 10, 2025 12:58 UTC
Investigating
Engineers are continuing to investigate an issue accessing Heroku services.
Posted 8 hours ago, Jun 10, 2025 09:19 UTC
Investigating
Engineers are investigating an issue with the Heroku platform.
Posted 9 hours ago, Jun 10, 2025 08:04 UTC
It has been there for over 9 hours now.
I would definitely shame them -- do they not even bother looking at status pages before giving you information? How has Heroku not distributed this information to any customer-contacting staff already?
It's crap like this that disturbs me even more than the outage in fact. I know bad outages can happen even to the best of us, but your account manager not knowing about it and not even bothering to try to find out before giving you bad information is the mark of mediocrity at best.
Access to check logs or perform other Heroku CLI functions is very intermittent. Some of our services work sometimes. Others don't. When I can occasionally access the logs, it seems like there are issues pinging other services. The apps aren't able to push their logs to DataDog.
It's a mess, and there's not much we can do right now, which is frustrating. And the fact that the Heroku status page itself is broken is embarrassing.
But then, about 7 hours in (~1:40 PM EDT) for about 30 minutes all web requests were failing with a 503, logged as H99 "Platform error". No requests made it to our dyno during that period.
We can't failover to contingency without making API calls - the read replica has to be forked and promoted. I've always comforted myself by saying "well, the data plane and control plane have always been separate and aren't impacted by the same events". No longer.
My guess is this may have been some kind of trouble-shooting that caused this issue, but it doesn't matter. The point is I can no longer ignore this global single point of failure and really the only answer is to migrate away. Which I am reluctant to do, and I don't know that it will increase reliability.
We have never had application downtime as a result of a Heroku issue in the last 8 years this has been in production. 30 minutes in 8 years isn't bad. But if it were down for 3 days I'd have the same options: fuck and all.
Even the status page is failing
Our servers are up till now
And the official Heroku status page says it's all fine.
I feel bad for companies with a lot of users
My best guess: The authentication issue affects them more than it does us.
1. Our platform can't access any external API services from that dyno specifically. 2. Can't get into dashboard.heroku.com to login to our services/infrastructure. 3. heroku-cli doesnt work so cant access any logs from our dyno. 4. We're an enterprise client and had next to no update or no response from our account manager albeit we're right at the bottom of the pecking order but no reason.
"Update 6: Posted Tue, 10 Jun 2025 11:27:51 UTC
The third-party provider disabled a single server, which did not yield a positive outcome. We are continuing our investigations with the provider."
I wonder, do they mean this whole shitshaw was caused by just one fucking third party server?
Even their status page (https://status.heroku.com/) doesn't work normally. This is incredible.
> 18:22:11 UTC
We’ve identified the likely trigger of the issue and are making strong progress toward a resolution. A recent automated update disrupted network connectivity on affected instances.
The Heroku dashboard is now back online. Customers can now attempt to recycle their dynos on the dashboard.
https://status.heroku.com/incidents/2822
https://status.salesforce.com/generalmessages/10001540
> The Salesforce Trust site is currently experiencing a service disruption. During this time, users will be unable to access the Trust site or receive notifications about service-impacting incidents or maintenances. We will continue to provide updates here until the issue is resolved.
And the heroku status page actually has recent upates, although they are just repeating this every hour:
> Heroku continues to investigate and remediate an issue with intermittent outages.