The Path to Medical Superintelligence

> Microsoft AI Diagnostic Orchestrator (MAI-DxO) correctly diagnoses up to 85% of NEJM case proceedings, a rate more than four times higher than a group of experienced physicians.

> Clinicians in our study worked without access to colleagues, textbooks, or even generative AI, which may feature in their normal clinical practice.

1. As I understand, it's very common for doctors to fall back on reference material in their practice, especially for the most complex cases. If all access to resources was cut off (as seems to be implied by the second quote), the comparison seems somewhat unfair.

2. What were the publication dates of the case records? I can't find this information, and it makes a difference if the NEJM case studies were in the LLMs' training data.

I was doing a comparative analysis of the acquistion strategies of various "big tech" firms and was a little startled that I missed Microsoft's 2022 acquistion of Nuance, largely for its speech recognition systems aimed at the medical sector:

https://news.microsoft.com/source/2022/03/04/microsoft-compl...

As any AI researcher knows, if you have a model that does 4x better than the naive baseline (the humans, in this case), you are likely looking at overfit, not real-life performance. This study is just slop, and you can tell so by the mere fact that they did not submit a paper, but just published a PR article.