Enterprise customers, remember to email your sales rep and ask for them to report on their contracted uptime with you that you are allowed to do as per contract.
They wont do this unless you ask hoping you don't notice the outages.
It creates lots of internal pain - they have no automation internally for reporting on this.
This is the only way anything will ever change. GitHub is _easily_ the most unreliable SaaS product. There's not a week whereby we aren't affected by an outage. Their reputation is mud.
petcat · 4h ago
> GitHub is _easily_ the most unreliable SaaS product.
Some of us are stuck using Atlassian and BitBucket and it is by far worse in every way.
zemo · 3h ago
I used GitHub for years in the tech industry, then went into the games industry and used BitBucket and hated it, thought it was such a downgrade. Now I'm back in the tech industry and using GitHub, and I miss BitBucket.
lesuorac · 3h ago
Starting to pretty much desire tools that can do nothing but are just fast at their core competency.
Looking at PRs in github and then when toggling to the "files" tab it chocking up or being like "I don't want to display this file because its more than 100 lines" is like wtf you're whole point is to show me modified files.
FredPret · 2h ago
Like the Unix philosophy, but for businesses.
Come to think of it, this might be great advice for life in general: do one thing very well, and be modular (aka play well with others)
Analemma_ · 23m ago
The thing is, I wish the world worked this way, but it doesn't, especially in tech. If you "do one thing well", the cloud hyperscalers will use their billions to copy whatever that is, and add your "one good thing" into their bundled subscriptions or cloud plans. At which point, any rational CTO will go "why should we pay for this, when we're already getting it via AWS/O365/whatever, and with better integration with our existing tooling to boot?"
I don't think "do one thing well" can succeed in this world, which is why Atlassian, Dropbox, etc. keep on launching things like office suits even though that makes no sense considering their core competencies. It's the only way not be streamrolled by FAANG.
agloe_dreams · 2h ago
I'm about to sound crazy.
Github's problem is that it isn't a SPA. It is a massive Ruby on rails project that is all server-rendered. Everything you do needs to be synchronous and almost everything requires a reload.
A react or angular app with great restraint would be dramatically faster at all of this as viewing a file is just an API call - not a page reload. They are stuck with their hands tied as loading large data would cause the whole page load to be delayed - thus silly limits.
Many things should not be webapps... but an app on the web like this...probably should.
cyrilou242 · 1h ago
> Everything you do needs to be synchronous and almost everything requires a reload.
this is pretty incorrect, you may want to look into the concept of "partials" in SSR.
maybe you meant everything requires a roundtrip ? but SPA would not solve most of the roundtrips necessary in github given many interactions in the github app require authn/authz checks.
would you care getting into more details ?
Also, 'old' github was known to be very fast an reliable and was indeed a ruby on rails SSR app.
Since a few years ago github started to introduce react and more client side logic and it correlates with more issues and more slowness in the frontend. It only correlates, but still.
lpapez · 1h ago
You can have parts of the web app rendered on the client, and still keep the rest of the app the same. Rewrite the diffs and previews, keep the rest as-is.
There is no excuse for possibly the most used feature of Github to suck so badly.
bpye · 3h ago
> then went into the games industry and used BitBucket and hated it
At least it wasn't Perforce?
nickitolas · 3h ago
Is games not a part of tech?
krs_ · 3h ago
Of course, but there are some oddities in tool use compared to other industries. At my job we use Perforce for version control for example, which I think is more common in the game industry than other solutions for whatever reason. Naturally everyone here hates it.
psygn89 · 3h ago
I liked the UI of BitBucket more. Stuff I accessed frequently like commits and branches were tabs across the top, easy to reach from any page. Branch dropdown sort order was by most recently updated unlike GitHub where I have to search for it. Easy to diff files. This was like 4 or 5 years ago though, maybe it has gone through modernization/enshittification. GitHub feels a bit fragmented and it tries to be more performant by virtualizing some things but while BitBucket in some sense was more rudimentary and showed it all (bogging my machine down some), it allowed me to CTRL + F easily with more confidence whereas with virtualization I've had issues with it finding things and I couldn't 100% trust it.
dosethree · 3h ago
Bitbucket is butt
pmontra · 30m ago
Two customers of mine have been using Bitbucket and other Atlassian products. I remember a problem a few months ago, nothing else. Maybe I've been lucky, no accesses during the outages.
sigio · 3h ago
It's also _by_far_ the cheapest, and most git-work can be done offline, so while it's inconvenient, it's not stopping all operations here.
ta1243 · 3h ago
Depends how intergrated you are. If you're using it as a code repository fine, if you're tying your workflows into it with pull requests, actions, maybe a third party CI which ties to it, and use it as part of operations then it's a major problem
I just approved a PR which added a user to one of our AWS accounts for example, if github is down then that PR can't be approved, the update can't work and the user can't access that account
Nicook · 1h ago
I mean you can always work around it with the correct permissions right.
JanMa · 1h ago
I used to maintain a self hosted instance of BitBucket and the user experience of it was actually very nice. We shut it down when Atlassian deprecated the self-hosted licenses. Moving to GitHub and GitHub Actions felt like a downgrade in more than a few ways
weberer · 3h ago
I've only had good experiences with Gitlab.
guywithahat · 3h ago
Funny, I've had the worst experience with gitlab. Bitbucket is the best experience I've had so far
__natty__ · 3h ago
Agree, atlassian products are much worse in terms of reliability.
enraged_camel · 3h ago
We use Jira. It's a horrendous product, but I cannot remember the last time it was down, unlike Github.
ta1243 · 3h ago
I use self-hosted jira, it's a great product, but I have full control over my teams tasks and workflows and as a tiny team we make them work for us (subject, description, comments, occasional linking to other tickets, assigned to, and status of "open", "blocked" or "done")
Most of the problems I hear about are micromanaging product managers. That's not the fault of the tool itself per-se.
hkt · 3h ago
Amen, BitBucket outages are regular and showstopping. Self hosting is genuinely more reliable.
bargainbin · 2h ago
Ah, you clearly haven’t worked with the self hosting teams that I have.
I hope I never again have to explain to someone that you can’t just “restore the code from the weekly database backup” because the code is in the file system they just osmosed.
ta1243 · 3h ago
Noooo!! It's impossible to get three nines uptime when you self host something on a single raspberry pi, I read it on the internet so it must be true.
at-fates-hands · 3h ago
Isn't Sourcetree an Atlassian product? I used that for a while when my company outlawed Github and it seemed to do a good job.
zulban · 31m ago
> GitHub is _easily_ the most unreliable SaaS product.
I'm no fan of Microsoft either but when you say ridiculous things it's hard to take you seriously.
cddotdotslash · 3h ago
Companies should automate this. Write their own outage monitoring, feed the results, plus the cumbersome format you have to send to the provider, into an LLM, have it spit out an email requesting SLA credits or whatever the contract specifies.
Probably not worth it for low cost services, but if you’re paying GitHub $x millions per year, maybe it is.
colinbartlett · 3h ago
Some customers of my product, StatusGator, do this with our API. They can extract the outage data -- including the time when we detect the outage before its acknowledged. And then use that to get SLA credits.
WD-42 · 2h ago
I think that was sarcasm
ta1243 · 3h ago
Why would I trust you to report
Its great that your specific product does this, but as a whole I have to monitor the service separately to keep you honest (well not you specifically, I'm sure you are honest and do as much as you can to be honest, but not every company is), and of course to monitor the problems I have which you don't detect.
edoceo · 1h ago
Gotta get a monitor for the monitor, then a manger for the monitors, and a manager to manage the managers of monitors.
dehrmann · 2h ago
Obviously you should use a SaaS for SaaS uptime monitoring. No need to build this yourself.
FinnKuhn · 1h ago
You can also self-host something like Prometheus or Uptime-Kuma.
colechristensen · 3h ago
They intentionally underreport outages. Everybody does. When your performance metrics for your customers, managers, and individual contributors all include uptime, what you get isn't better uptime but lies about uptime.
RandallBrown · 3h ago
I've had much better luck with GitHub than GitLab and whatever Atlassian is calling their source control these days.
Still sounds like good advice though.
xyst · 1h ago
All this does is make some poor offshore schmuck toil away for hours trying to get this data.
If companies begin to _cancel_ their contracts with MSFT/GH because of a breach of SLAs, then maybe conditions will improve.
Reality: companies locked into multi year deals with MSFT including a MSFT-shit suite and windows licenses.
Migrating away from it will be expensive. MS knows this. Thus the reason why nothing will change.
sneak · 3h ago
The way you change this is not by pressuring Microsoft to be better or less of a shitty company.
The way that you change this is by switching to a different forge host, or by self-hosting Gitea.
I do so, and it’s simple and painless and cheap, and this quarter my uptime is better than this multinational’s.
xyst · 1h ago
this, but unfortunately many Fortune 500 are so deep in the MSFT stack. It would take many years to migrate off.
CoderJoshDK · 3h ago
With continued reliability issues and the CEO stepping down, now feels like such an opportune time for a competitor to start taking market share. I sure am rooting for it!
For the longest time, I thought that there was absolutely no way for some of these corner stone companies (slash tech) to be toppled. And I’m very impressed with their ability to destroy consumer trust!
shrinks99 · 2h ago
Between Tangled, GitLab, Codeburg (Forgejo), and Gitea there's quite a lot of decent alternatives now compared to when GitHub first sold to MS. Having the entire world of FOSS integrated in one development platform was convenient but I'm more excited by the possibilities for more innovation in the space.
colechristensen · 3h ago
Sadly gitlab lacked focus especially on software quality.
WhyNotHugo · 6m ago
The linked page doesn’t say much. Just “increased latency”. No more details than that. Am I missing something?
evilmonkey19 · 4h ago
I'm a fan of github but lately i'm seeing a lot of issues like these... Also they don't have yet support for IPv6 (surprising).
awestroke · 4h ago
The refusal to support IPv6 is embarrassing at this point
kevingadd · 3h ago
Had to buy an IPv4 address for a VPS the other day in order to clone some git repositories. Couldn't believe it. Costing their customers money when they should be able to support v6 by now.
hk1337 · 3h ago
What VPS are you using that doesn't come with both IPv4 and IPv6?
miyuru · 3h ago
There are plenty of low end providers that support IPv6 only.
At that scale price of IPv4 is the highest cost of the VPS.
Here is a list of providers I created back in 2022.
Hetzner charges extra for IPv4 address, as I believe most of them do. I know because I went through the same crap.
dehrmann · 2h ago
It seems more like a weird Hetzner thing that they won't give you a IPv4 NAT gateway.
jorams · 39s ago
They charge €0.50 per month to add an IPv4 address. A shared IPv4 NAT gateway introduces a whole lot of problems for them just to support customers who need IPv4 but don't want to pay a tiny amount for it.
drozycki · 1h ago
How would a server-side NAT know which Hetzner customer it should route a request to? It has an encrypted packet arriving at this shared address on port 443. You can route a shared address to the proper service based on the HTTP Host header but that can only be done by the customer using their encryption key, so no sharing an address between customers. Home LAN NAT only works because the router can change the source port used by the request so that responses are unambiguously routed to the right client.
lelandbatey · 1h ago
I don't think they're saying they should support incoming connections on such a NAT, I think they're saying that servers behind the NAT would be able to make outgoing connections (e.g. to access shared resources).
jraph · 3h ago
There are options where you pay 1€/IPv4/month and IPv6s are free.
ta1243 · 3h ago
AWS charges for ipv4 doesn't it?
hk1337 · 3h ago
In regards to an EC2, AFAIK, not necessarily. You pay extra for an elastic IP (IPv4) which is the equivalent to a static IP but the EC2 is assigned an IPv4 address and an IPv6 when IPv6 is enabled.
oasisbob · 2h ago
Beginning in early 2024, AWS began charging for every IPv4 address in-use on your resources.
I thought a recent downtime was contributed to rolling out the initial prep for IPv6 support.
depr · 4h ago
Given that they are probably at least partly on Azure, this makes it less surprising because Azure has the worst IPv6 implementation of the 3 large cloud providers.
77pt77 · 2h ago
Possibly stupid question but, how can someone mess that up?
What is wrong/missing?
gmerc · 4h ago
Is this a good time to mention forgejo as a self hosted alternative?
yogsototh · 2h ago
I host a forgejo instance and I feel it is great to embrace the distributed nature of git.
But be aware if you intend to host it you will need to protect it from recent AI web scrapers. I use anubis but there are other alternatives.
jeltz · 4h ago
How does it compare to Gitlab?
dzogchen · 4h ago
Much simpler. Much less features. Completely open source and not only the core.
jeltz · 4h ago
Any important feature from Gitlab you feel is missing? I personally think Gitlab has way more features than I need but maybe there are some important ones I would miss.
lucb1e · 3h ago
I use git pretty basically, just as a revision system where the hosted version has some nice-to-haves on top like rendering markdown, permalinking bits of code to people, and being able to open tickets and contribute code. Gitea/Forgejo/Codeberg does all of that and I haven't run into any missing features. It's also a lot easier to navigate than GitLab, but I'll admit that's probably just a matter of me not being used to it
Having self-hosted Gitea after considering GitLab, I can also say that the resource consumption of Gitea is a tiny fraction of that of GitLab's. I don't get the impression that their employees care about self hosters beyond as a gateway for enterprise sysadmins to get it running quickly before doing some big installation
styanax · 3h ago
A tough question as everyone's needs are different; I might recommend you create an account on https://codeberg.org as it's the largest, most popular instance of Forgejo running with many FOSS projects hosted there.
Codeberg devs have to disable some features (pull mirrors e.g., only push is allowed to prevent abuse) and they use some custom code (abuse mitigation - spam, etc.) but in general you're getting the latest Foregjo experience "test drive" which only gets better when self hosting when you can use all the features.
makkesk8 · 4h ago
Running Gitlab in any kind of scale beyond a single server is a major PITA.
And it's very poorly optimized.
appease7727 · 3h ago
Gods, for some reason GitLab consumes 5-10% of a CPU at all times. I spent weeks trying to get it to calm down to reduce our AWS spend. Absolutely no changes no matter what I tried. On my 2013 Xeon server at home it's even worse.
GitLab is great, I really do enjoy working with it. I hate running it.
12_throw_away · 2h ago
Yeah, I love gitlab as a user - but as an admin, the performance feels like something out the 90s. I had to use the gitlab-rails REPL console for something a couple of weeks ago. Even on a server with tons of headroom, it took *10 minutes* to start up?
jeltz · 3h ago
That I am well aware of. I hate running Gitlab. I jsut wonder what different features different people are missing.
homebrewer · 4h ago
Depends on your needs. Last time I checked, Gitlab wanted money for e.g. assigning multiple PR reviewers, which is available in gitea/forgejo.
The real issue with gitea/forgejo compared to Gitlab is their terrible CI, which is (to some approximation) a clone of GitHub Actions, also a dumpster fire for those of us proficient with/preferring the UNIX command line. You'll probably need a separate CI runner, like Woodpecker or Drone.
jeltz · 3h ago
Gitlab CI is so much better than Github Actions. With some additional improvements Gitlab CI could become amazing.
zamalek · 3h ago
CI is one area that it's "lacking." I quote that because, honestly, all the bells and whistles in these CI yamls are starting to hurt. Woodpecker[1] (what forgejo uses) is strikingly simple.
The only thing worse than "it's down" is hearing "it's down again"
Godspeed, IR workers...
jeltz · 4h ago
Last time it was down was Friday last week when they broke some of the raw content URLs.
gdcbe · 3h ago
Maybe they should focus less on "agentic" and more on just keeping their core product solid... I suppose that doesn't rhyme with growth at all cost... zzz sad... it is
taude · 4h ago
git was designed as a distributed vcs for high-latency connected developers with plenty of ability to work offline.
I don't think I've really been impacted by any of the outages. Maybe I wait an extra hour to merge a feature or something, in which case I actually get to eat lunch and browse HN, doesn't feel quite as catastrophic for me, as some of you.
petcat · 4h ago
The problem is that people design their entire development and release lifecyle to be dependent on Github. A lot of times they can't even push code hotfixes to production without it. It's a terrible SPOF for a lot of engineering orgs.
lucb1e · 3h ago
We also started having customers since a few years that declare GitHub fully trusted, as in, it is simply not worth considering what the impact would be if that vendor gets compromised. I can't name names, but this includes a vendor that aims to prevent supply chain attacks (technically language-agnostic; in practice aiming to be the solution chosen by one of the biggest programming language's package manager)
> can't even push code hotfixes to production without it. It's a terrible SPOF
GitHub's availability impact is the least of my concerns these days. It'll be a really tough year for society worldwide if we need to rebuild loads of infrastructure after some threat actor got into github and managed to change key pieces of code without being detected a couple of years. Having seen how hospitals handle updates, they might get lucky and be old enough to not be affected yet, or have a really tough time recovering due to understaffed IT
No clue how to even begin solving this since our OSes are likely all pulling dependencies from GitHub without verification of the developer's PGP key, if the project even has that and applies it correctly. I guess I can only recommend being aware of the problem and doing what you can in your own organization to reduce the impact
taude · 3h ago
The hotfixes makes sense.
dymk · 3h ago
GitHub is everything in addition to the git hosting. Issue tracking, code review, CI, artifact hosting, wiki+docs, kanban board.
taude · 3h ago
That makes sense, I wasn't sure how "all in" everyone is on these other features that are only github centric.
Joe_Cool · 3h ago
It also has pretty neat support for emailing patches.
And it's practically impossible to lose data as long as any single dev still has an intact .git directory.
Nobody is preventing the devs from just setting up a second "upstream" and pushing to both github and gitlab (for example) or any other service at the same time.
koiueo · 3h ago
GitHub isn't just git, it's also a CI, a project management tool/issues tracker...
Thev00d00 · 4h ago
Vibe coded a bit too hard.
bob1029 · 4h ago
I've been working on a self-hosted alternative to GitHub and I am curious what HN finds to be the most important features. I think Code, Issues and PRs are the critical aspects, but I don't know what typical workflows look like for others these days.
It seems like there are some teams that have figured out a way to turn GH into a labyrinth of CI/CD actions that allegedly produces a ton of business value, but I don't know how typical this kind of use case is. If the average GH user just needs the essentials, I could easily focus those verticals down and spend more time on things like SAML/OIDC, reporting APIs, performance, etc. I suspect there aren't a whole lot of developers who are finding things like first party AI integration to be indispensable.
gazook89 · 3h ago
Any time I see this topic brought up, two things are always mentioned: the "hub" part, meaning the discoverability and social aspect, and the "network effect" of having everyone use a single service (so everyone already has an account and they don't have to create additional account for every self-hosted project.
tracker1 · 3h ago
Agreed, it's definitely the network cohesion that keeps GH together. Especially for FLOSS. For advanced features, there are some niceties that say Azure DevOps offers that GH Enterprise still lacks, though it feels like there's some convergence on the backend.
I like GH Actions myself, though sometimes it can get a little cumbersome with self-hosted workers on private/enterprise projects. I'm a big fan of automation and like to keep related bits closer together. As opposed to alternatives that have completely detached workflows/deployments.
I like what I've used of Forgejo (Git, Issues+Board, Wiki), and have hosted it on servers and localhost easily. I haven't tried its CI features yet.
Codeberg is a cloud site for open source projects that runs Forgejo.
Forgejo is a fork of Gitea, which is another option, especially if you want commercial support, but I haven't tried it yet.
I also kinda like GitLab, both the cloud one and the enterprise on-prem version. And their issue label features work more easily with the board than Forgejo's (automatically moving issues between columns based on scoped labels). Though their pricing tiers have been unfortunate at times (I don't know latest).
justinrubek · 3h ago
I don't think it's the features that are lacking from the alternatives. It's the network effects.
sneak · 3h ago
I use and love Gitea. Why not contribute to that or the other fork vs making another?
zanfr · 4h ago
well I wish nothing but the best for the great people at Microsoft, this small family business.
halfmatthalfcat · 4h ago
Github reliability has been abysmal the past couple of weeks.
Aldipower · 1h ago
Used and using Gitlab with success in all of my companies. It is maybe a bit boring and a little bit slow, but it is enterprise and has everything you need. Included time on the market..
alphazard · 3h ago
Everyone impacted by this outage should take a look at Radicle, which was also on the front page today.
I'm not affiliated with Radicle in any way. I am just impressed with the technology.
jonator · 3h ago
If you're reading this there's a great opportunity to pull a Linear move and disrupt the entrenched players with a 10x better UX. Although the hardest nut to crack here are the network effects.
demarq · 3h ago
What’s the hardest part of making a git hosting service?
I used to work at a place that had a second copy of all the git repositories on a server, available over ssh. We could push there and then, at deploy time, instruct the servers to pull from that repo instead of bitbucket.
If I were to set up the same thing again today, I'd add some automation to automatically keep it in sync with github as well as automate the servers so that they'd attempt to pull from both on every deploy.
I share this as an idea for those who need an emergency or backup deploy mechanism.
gethly · 4h ago
I've always used Bitbucket as it allows private repositories, so Github was never something I wanted to use. But it is one of the most important websites in the world for tech people and should be run better than this, especially being now owned by the largest tech company in the world. On the other hand, it just shows that centralisation, or over-reliance on one thing or service, is always the worst idea. But people are very lazy and so we keep running in these circles ad infinitum.
clintonb · 4h ago
Do you mean free private repos? GitHub has allowed free private repos for a few years now.
gethly · 25m ago
I have been using bitbucket before it got acquired by atlassian, which was in 2010, so i am sure github changed quite a bit since then. Even though i do have a github account, i don't use it for anything else other than creating or commenting on issues for other repos. So i have no clue about present day github capabilities as i am satisfied with bitbucket and had no need to explore github in depth.
Man, I remember feeling like that change was somewhat recent. 10 years ago was about 2016. I'm not ready to accept this.
acedTrex · 4h ago
Day ends in Y
sigio · 3h ago
Just completed one or our (quarterly) github exports before this hit... If people are looking into extracting everything from their organisations, i've published the scripts I use: https://github.com/sigio/github-export
asim · 3h ago
Too many unnecessary features at this point perhaps. Self hosting might be a good alternative. Don't need the UI.
legobmw99 · 4h ago
Day one of the new management structure off to a rocky start
xyst · 4h ago
always been "rocky" since the acquisition.
This is just perfect comedic timing
irthomasthomas · 3h ago
Any one else smile for a moment when they saw their issues list empty? Don't rush to fix it ;)
This is just embarrassing for them, especially whoever is in charge of pull requests.
thrdbndndn · 4h ago
Coincidentally reddit is also down.
mock-possum · 4h ago
Well at least there’s a silver lining then
ngoldbaum · 4h ago
I wonder why the github status page has an atlassian cookie request pop-up.
mbb70 · 3h ago
The status page is a SaaS product called StatusPage, acquired by Atlassian in 2016.
metalliqaz · 4h ago
Dear sages of HN, what is the scrappy new upstart project that will rise up to take Github's place? Because this is becoming intolerable.
john01dav · 4h ago
I think that the answer here is eventual widespread use of many smaller options, from FSF's Savannah to self-hosted gitea/forejo/gitlab.
Git is in a good position to support this.
cAtte_ · 4h ago
not gonna happen. society loves monopolies because they're convenient
john01dav · 4h ago
We're starting to see the pain of such monopolies. Note that I included a hosted option (Savannah) in my list. It doesn't take everyone leaving github to break the monopoly, just enough to make it not a monopoly.
meepmorp · 4h ago
It's kind of weird that we've collectively decided on a distributed version control system, while centralizing where we keep the repositories and metadata.
ryandv · 4h ago
This is because the dream of true P2P decentralization died with NAT and IPv4 exhaustion.
metalliqaz · 3h ago
I think it's just a simple reality that most projects don't actually need or want a decentralized development process. In my experience, most projects are looking for a single, high-reliability canonical source that is in control of project leadership. Most projects are only developed by a small group, maybe even only one person.
keybored · 18m ago
^ Comment on the Nth 100+ GitHub Down thread (every thread is like that).
Maybe everyone here is just using it as an excuse to chatter about forges or GitHub being down too much etc. and it has no impact. But if it does and people are honestly fretting they can mirror their repos. Then no one needs to worry that much (except for their cursed CI setups) the next time it happens.
And that’s a benefit of peer-to-peer repos right there.
meepmorp · 3h ago
Yeah, I think what people really liked about git (vs many other VCSs at the time) was cheap and easy local branching, not the distributed nature.
keybored · 17m ago
Local and distributed is the same difference to me compared to subversion.
tracker1 · 3h ago
I think the distributed support is pretty nice for easy-ish mirroring. Even to a relatively bare git+ssh target on a self-hosted server. No specialized services required. I mean, just for VCS.
JyB · 3h ago
I’m really wondering what internally causes this. No one likes it when they have outages, but it keeps happening. Is this a culture thing? Like pressure to ship features fast? Some under staffed teams or lack of ownership on some crucial components?
tracker1 · 3h ago
Definitely all good questions. I've noticed that there seems to be a bit of convergence between Azure DevOps and Github (Enterprise) and am curious how co-mingled the teams or management are at any given level or not.
I'm mixed on the cultural changes at MS and have historically preferred GH's approach. I'm hoping MS moves closer to GH than the reverse. I'm not working with/at either company and don't really have a lot of insights to offer other than observations from the outside.
ab71e5 · 4h ago
Ah was wondering why pull requests did not show up
dustypotato · 4h ago
Why does it show me a cookie banner from Atlassian?
alexjplant · 4h ago
...because they use Statuspage, an Atlassian product?
j45 · 4h ago
The cloud is someone else’s computer.
I wonder if self hosting is more reliable. How much does a private and firewalled git* instance need updates?
tracker1 · 3h ago
If you don't need any features beyond a backup location for git, all you need is an SSH server with FS support. All you need is git on the remote server to initialize a server directory, and you can target that with git+ssh directly. Works well enough as a backup/mirror repository.
If you want to self-host for more features (CI/CD, PRs, etc.) there's GitLab, Gitea, and forgejo that I'm aware of. I think GitLab is a bit heavy duty for most self-hosting usage myself though. I actually appreciate the online/cloud and commercial options.
When you self-host, it becomes your job to fix it when it breaks.
j45 · 2h ago
Yup, as you mentioned, there's other alternatives to Gitlab lcaolly that are decent.
I'm finding myself liking and using gitlab more and more when I come back to it every 6-8 months.
I don't know how I'd be able to trust only a cloud for my source code and devops/CI/CD. At least a mirrored setup in a private or hybrid cloud on another provider as a failover that isn't with the same cloud provider.
homebrewer · 4h ago
Our gitea uptimes are measured in months. The only downtime is during non-working hours for upgrading gitea & the underlying OS, which take about 5 seconds of work and another 15 seconds of waiting for it to upgrade the database and restart.
j45 · 2h ago
That's great to hear, taking this downtime to install them all and try them out.
At the very least a few backups and mirrors running once I get them syncing.
john01dav · 4h ago
I have gitea running in my basement. You can see how often it updates on the gitea github (since they don't self host since they haven't written an importer yet). You could automate this, but I haven't spent enough time updating to make that make sense.
numbsafari · 4h ago
If you just need a hosted git server, maintenance is mostly just making sure you have a proper storage/backup strategy in place.
Before gitlabs, github, etc. it was common to host your own code repos on-prem. The thing now, though, is that there's a lot of add-on functionality for how teams flow from using github over just hosting a git server.... so it's not really apples v oranges, anymore.
marcosdumay · 3h ago
> The thing now, though, is that there's a lot of add-on functionality
I don't get why people use those.
I understand free software projects that don't want to run any infrastructure, but why companies push their building and deployment out of premises when all you need is one trusted computer somewhere. Why do people insist on trusting cloud computers more than the ones they can kick?
john01dav · 4h ago
You can use a self hosted forge, like gitea, forejo, or gitlab.
j45 · 2h ago
Gitlab can handle a lot of CI/CD hosted locally for free that github actually charges for.
Git was originally local only too. People would run their own source code repos, it was trivial to run and maintain for the most part for most basic to intermediate use cases.
I had clients who insisted source code (mine or theirs) couldn't be on a public cloud provider. It's not that unreasonable or uncommon.
e-dant · 1h ago
I like github
meepmorp · 4h ago
Actually, the big problem for me is that github doesn't have issues at the moment.
boesboes · 4h ago
Yeah, i was pretty glad to see i have zero open issue remaining today!
jsk2600 · 4h ago
At least I can enjoy seeing zero issues on my projects, once in a lifetime...
TonyTrapp · 4h ago
Yep, issues are having issues, resulting in user not seeing issues
ryandv · 4h ago
Funny, because just yesterday I was downvoted for pointing out (among other things) GitHub's less than stellar reliability these past few years.
When are the AI vibe coders going to create a GitHub replacement? With 1000x AI productivity a lean startup should easily outcompete the incumbents, no?
numbsafari · 4h ago
Yeah, especially if you deploy it on Azure...
nimbius · 4h ago
recent developments in the C levels not withstanding, github has been a dumpster fire for quite some time now with no sign of improvement.
social features via atprotocol, native jujutsu support, stacked PRs, interdiffs, light weight to selfhost, nix-based CI.
Aldipower · 1h ago
Gitlab of course. It's mature.
amelius · 4h ago
When a cool job becomes boring IT work.
joshred · 3h ago
Sooo....
What's the consensus on gitlab?
tracker1 · 3h ago
I've used their public site for a few private projects, mostly in habit from when private projects in GH were limited to paid accounts. The collaboration was a bit better at that time imo.
I'm not sure that I would choose it for self-hosting over gitea, forgejo or straight up ssh+git on a remote system, which works well enought for a personal backup target.
belter · 4h ago
Did they upgrade to GitHub 11 ?
CoastalCoder · 4h ago
Thanks to internal politics, that only possible if they have a Technical Product Manager II (TPM).
yoyohello13 · 3h ago
Just saying. Our company has been on self-hosted Gitlab for years. We have one devops guy who spends like an hour a month managing the server. Never one outage.
rvz · 4h ago
No CEO of GitHub to contact about this outage this time.
numbsafari · 4h ago
Contact Satya Nadella.
siva7 · 4h ago
The guy who framed Github as a stellar example of how Microsoft will let operate its aquisitions independently?
numbsafari · 4h ago
He is (and was) literally the CEO.
Yeah, he can give whatever title he wants to his subordinates, but the "CEO" of GitHub has been a mid-level VP for quite some time.
rs_rs_rs_rs_rs · 4h ago
Just Github? I'm having problems logging in into Dropbox as their captcha setup fails.
havkom · 4h ago
Does this matter much? Git is a ”distributed source control system”. Are you using git in the wrong way? :)
tracker1 · 3h ago
Well, considering git is only version control, and github does much more... pull requests, social interaction, workers/workflows, ci/cd etc. It's kind of a big deal.
Unless you have some sort of decentralized method of doing CI/CD and pull requests I'm unaware of?
bootsmann · 4h ago
I am surprised at how much github is willing to let themselves go for being a product that can nowadays be replaced by someone over the weekend. This is the second major outage within like 2 weeks.
taude · 4h ago
Depends how much customization you've done to your cicd and how heavily you use custom GHA, and other plugins. No way my org could move off in a weekend.
tracker1 · 3h ago
Agreed... you could probably move a single project over pretty easily on a weekend, or several depending on how integrated you are.
reaperducer · 4h ago
for being a product that can nowadays be replaced by someone over the weekend.
If one person can move a company move off of GitHub in a weekend, you're too small a company for Microsoft to care about.
bootsmann · 4h ago
It was less about moving your company away and more about the core functionality being extremely simple to reproduce.
sashank_1509 · 4h ago
I’m at Google, we have a million + file codebase. Every piece of code is snapshotted (no need to commit, automatic snapshot whenever a change is made). Every line of code has its own unique URL (this is for your branch too, not just overall). Background tests running nonstop hourly. Never seen any downtime. The infra at Google is insane
sequin · 2h ago
GitHub is a completely different scale. If Google hadn't wasted its opportunities to become the prime source code platform they'd also have issues.
chaz6 · 3h ago
Without manual commits, is there any way to record the intention of each change?
sashank_1509 · 3h ago
Yes you can still commit as a logical delineation of changes. And every commit is automatically a PR, folding multiple commits is possible but not the common workflow, they prefer just reviewing and approving each commit.
Balinares · 3h ago
There are manual commits, they're just called changelists.
This is the only way anything will ever change. GitHub is _easily_ the most unreliable SaaS product. There's not a week whereby we aren't affected by an outage. Their reputation is mud.
Some of us are stuck using Atlassian and BitBucket and it is by far worse in every way.
Looking at PRs in github and then when toggling to the "files" tab it chocking up or being like "I don't want to display this file because its more than 100 lines" is like wtf you're whole point is to show me modified files.
Come to think of it, this might be great advice for life in general: do one thing very well, and be modular (aka play well with others)
I don't think "do one thing well" can succeed in this world, which is why Atlassian, Dropbox, etc. keep on launching things like office suits even though that makes no sense considering their core competencies. It's the only way not be streamrolled by FAANG.
Github's problem is that it isn't a SPA. It is a massive Ruby on rails project that is all server-rendered. Everything you do needs to be synchronous and almost everything requires a reload. A react or angular app with great restraint would be dramatically faster at all of this as viewing a file is just an API call - not a page reload. They are stuck with their hands tied as loading large data would cause the whole page load to be delayed - thus silly limits.
Many things should not be webapps... but an app on the web like this...probably should.
this is pretty incorrect, you may want to look into the concept of "partials" in SSR. maybe you meant everything requires a roundtrip ? but SPA would not solve most of the roundtrips necessary in github given many interactions in the github app require authn/authz checks.
would you care getting into more details ?
Also, 'old' github was known to be very fast an reliable and was indeed a ruby on rails SSR app. Since a few years ago github started to introduce react and more client side logic and it correlates with more issues and more slowness in the frontend. It only correlates, but still.
There is no excuse for possibly the most used feature of Github to suck so badly.
At least it wasn't Perforce?
I just approved a PR which added a user to one of our AWS accounts for example, if github is down then that PR can't be approved, the update can't work and the user can't access that account
Most of the problems I hear about are micromanaging product managers. That's not the fault of the tool itself per-se.
I hope I never again have to explain to someone that you can’t just “restore the code from the weekly database backup” because the code is in the file system they just osmosed.
I'm no fan of Microsoft either but when you say ridiculous things it's hard to take you seriously.
Probably not worth it for low cost services, but if you’re paying GitHub $x millions per year, maybe it is.
Its great that your specific product does this, but as a whole I have to monitor the service separately to keep you honest (well not you specifically, I'm sure you are honest and do as much as you can to be honest, but not every company is), and of course to monitor the problems I have which you don't detect.
Still sounds like good advice though.
If companies begin to _cancel_ their contracts with MSFT/GH because of a breach of SLAs, then maybe conditions will improve.
Reality: companies locked into multi year deals with MSFT including a MSFT-shit suite and windows licenses.
Migrating away from it will be expensive. MS knows this. Thus the reason why nothing will change.
The way that you change this is by switching to a different forge host, or by self-hosting Gitea.
I do so, and it’s simple and painless and cheap, and this quarter my uptime is better than this multinational’s.
For the longest time, I thought that there was absolutely no way for some of these corner stone companies (slash tech) to be toppled. And I’m very impressed with their ability to destroy consumer trust!
At that scale price of IPv4 is the highest cost of the VPS.
Here is a list of providers I created back in 2022.
https://blog.miyuru.lk/ipv6-hosting-2022/
https://aws.amazon.com/blogs/aws/new-aws-public-ipv4-address...
What is wrong/missing?
But be aware if you intend to host it you will need to protect it from recent AI web scrapers. I use anubis but there are other alternatives.
Having self-hosted Gitea after considering GitLab, I can also say that the resource consumption of Gitea is a tiny fraction of that of GitLab's. I don't get the impression that their employees care about self hosters beyond as a gateway for enterprise sysadmins to get it running quickly before doing some big installation
Codeberg devs have to disable some features (pull mirrors e.g., only push is allowed to prevent abuse) and they use some custom code (abuse mitigation - spam, etc.) but in general you're getting the latest Foregjo experience "test drive" which only gets better when self hosting when you can use all the features.
GitLab is great, I really do enjoy working with it. I hate running it.
The real issue with gitea/forgejo compared to Gitlab is their terrible CI, which is (to some approximation) a clone of GitHub Actions, also a dumpster fire for those of us proficient with/preferring the UNIX command line. You'll probably need a separate CI runner, like Woodpecker or Drone.
[1]: https://woodpecker-ci.org/
Godspeed, IR workers...
I don't think I've really been impacted by any of the outages. Maybe I wait an extra hour to merge a feature or something, in which case I actually get to eat lunch and browse HN, doesn't feel quite as catastrophic for me, as some of you.
> can't even push code hotfixes to production without it. It's a terrible SPOF
GitHub's availability impact is the least of my concerns these days. It'll be a really tough year for society worldwide if we need to rebuild loads of infrastructure after some threat actor got into github and managed to change key pieces of code without being detected a couple of years. Having seen how hospitals handle updates, they might get lucky and be old enough to not be affected yet, or have a really tough time recovering due to understaffed IT
No clue how to even begin solving this since our OSes are likely all pulling dependencies from GitHub without verification of the developer's PGP key, if the project even has that and applies it correctly. I guess I can only recommend being aware of the problem and doing what you can in your own organization to reduce the impact
Nobody is preventing the devs from just setting up a second "upstream" and pushing to both github and gitlab (for example) or any other service at the same time.
It seems like there are some teams that have figured out a way to turn GH into a labyrinth of CI/CD actions that allegedly produces a ton of business value, but I don't know how typical this kind of use case is. If the average GH user just needs the essentials, I could easily focus those verticals down and spend more time on things like SAML/OIDC, reporting APIs, performance, etc. I suspect there aren't a whole lot of developers who are finding things like first party AI integration to be indispensable.
I like GH Actions myself, though sometimes it can get a little cumbersome with self-hosted workers on private/enterprise projects. I'm a big fan of automation and like to keep related bits closer together. As opposed to alternatives that have completely detached workflows/deployments.
Codeberg is a cloud site for open source projects that runs Forgejo.
Forgejo is a fork of Gitea, which is another option, especially if you want commercial support, but I haven't tried it yet.
I also kinda like GitLab, both the cloud one and the enterprise on-prem version. And their issue label features work more easily with the board than Forgejo's (automatically moving issues between columns based on scoped labels). Though their pricing tiers have been unfortunate at times (I don't know latest).
https://news.ycombinator.com/item?id=44874945
I'm not affiliated with Radicle in any way. I am just impressed with the technology.
If I were to set up the same thing again today, I'd add some automation to automatically keep it in sync with github as well as automate the servers so that they'd attempt to pull from both on every deploy.
I share this as an idea for those who need an emergency or backup deploy mechanism.
This is just perfect comedic timing
Git is in a good position to support this.
Maybe everyone here is just using it as an excuse to chatter about forges or GitHub being down too much etc. and it has no impact. But if it does and people are honestly fretting they can mirror their repos. Then no one needs to worry that much (except for their cursed CI setups) the next time it happens.
And that’s a benefit of peer-to-peer repos right there.
I'm mixed on the cultural changes at MS and have historically preferred GH's approach. I'm hoping MS moves closer to GH than the reverse. I'm not working with/at either company and don't really have a lot of insights to offer other than observations from the outside.
I wonder if self hosting is more reliable. How much does a private and firewalled git* instance need updates?
If you want to self-host for more features (CI/CD, PRs, etc.) there's GitLab, Gitea, and forgejo that I'm aware of. I think GitLab is a bit heavy duty for most self-hosting usage myself though. I actually appreciate the online/cloud and commercial options.
When you self-host, it becomes your job to fix it when it breaks.
I'm finding myself liking and using gitlab more and more when I come back to it every 6-8 months.
I don't know how I'd be able to trust only a cloud for my source code and devops/CI/CD. At least a mirrored setup in a private or hybrid cloud on another provider as a failover that isn't with the same cloud provider.
At the very least a few backups and mirrors running once I get them syncing.
If you want to self-host a code forge... start here: https://forgejo.org/docs/latest/admin/upgrade
I don't get why people use those.
I understand free software projects that don't want to run any infrastructure, but why companies push their building and deployment out of premises when all you need is one trusted computer somewhere. Why do people insist on trusting cloud computers more than the ones they can kick?
Git was originally local only too. People would run their own source code repos, it was trivial to run and maintain for the most part for most basic to intermediate use cases.
I had clients who insisted source code (mine or theirs) couldn't be on a public cloud provider. It's not that unreasonable or uncommon.
When are the AI vibe coders going to create a GitHub replacement? With 1000x AI productivity a lean startup should easily outcompete the incumbents, no?
https://www.githubstatus.com/history
[1] https://forgejo.org/ [2] https://codeberg.org/
social features via atprotocol, native jujutsu support, stacked PRs, interdiffs, light weight to selfhost, nix-based CI.
What's the consensus on gitlab?
I'm not sure that I would choose it for self-hosting over gitea, forgejo or straight up ssh+git on a remote system, which works well enought for a personal backup target.
Yeah, he can give whatever title he wants to his subordinates, but the "CEO" of GitHub has been a mid-level VP for quite some time.
Unless you have some sort of decentralized method of doing CI/CD and pull requests I'm unaware of?
If one person can move a company move off of GitHub in a weekend, you're too small a company for Microsoft to care about.