I got fooled by AI-for-science hype–here's what it taught me

Interesting discussions tend to avoid “AI” in favour of specific terms such as “ML”, “LLM”, “GAN”, “stable diffusion”, “chatbot”, “image generation”. These terms refer to specific tech and applications of that tech, and allow to argue about specific consequences for sciences or society (use of ML in biotech vs. proliferation of chatbots).

However, certain sub-industries prefer “AI” precisely because it’s so vague, offers seemingly unlimited potential (please give us more investment money/stonks go up), and creates a certain vibe of a conscious being useful when pretending not to be working around IP laws and creating tools based on data obtained without relevant licensing agreements (cf. the countless “if humans have the freedom to read, therefore it’s unfair to restrict the uses of a software tool” fallacies, often perpetuated even by seemingly technically literate people, in pretty much every relevant forum thread).

To the extent you care about chess and Go as human activities, progress there is real.

there are some other scientific computing problems where AI or neural-network-based methods do appear to be at least part of the actual state-of-the-art (weather forecasting, certain single-molecule quantum chemistry simulations).

i would like the hype of the kind described in the article to be punctured, but this is hard to do if critics make strong absolute claims ("aren't useful in any field whatsoever") which are easily disproven. it hurts credibility.

It's also very hard to prove a negative. If you predict "AI will never do anything of value" people will point to literally any result to prove you wrong. TFA does a good job debunking some recent hype, but the author cannot possibly wade through every hyperbolic paper in every field to demonstrate the claims are overblown.

It’s people making money off hype until it dies and move on to the next scam-with-some-use.

This is "paying the toll", otherwise one will be accused of being a "luddite."

> Where are the AI-driven breakthroughs

> are we just using AI to remix existing general knowledge, while making no progress of any sort in any field using it?

The obvious example of a highly significant AI-driven breakthrough is Alphafold [1]. It has already had a large impact on biotech, helping with drug discovery, computational biology, protein engineering...

[1] https://blog.google/technology/ai/google-deepmind-isomorphic...

literally last week

https://deepmind.google/discover/blog/alphaevolve-a-gemini-p...

They exist all over science, but they are just one method among many, and they do not really drive hypotheses or interpretations (even now)

Define breakthrough. When is the improvement big enough to count as one?

Define AI. Are you talking about modern LLM, or is old school ML also in that question?

I mean Googles AI-company had with AlphaFold and other project quite the impact.

> Or are we just using AI to remix existing general knowledge

Is remixing bad? Isn't many science today "just" remixing with slight improvements? I mean, there is a reason why we have theoretical and practical scientists. Doing boring Lab-work and accidentally discovering something exciting is not the only way science is happening. Analysing data and remixing information, building new theories, is also important.

And don't forget, we don't have AGI yet. Whatever AI is doing today, is limited by what humans are using it for. Another question is, whether LLM is not normalized enough already that we do not see it as very special any more, if it's used somewhere. So we might not even see it if AI has significant impact on any breakthrough.

The only thing that seems to live up to the hype is AlphaFold, which predicts protein folding based on amino acid sequences, and of which people say that it actually makes their work significantly easier.

But, disclaimer, this is only from second-hand knowledge, I'm not working in the field.

Maybe too much room, but it's hard to predict if AI tools will overcome their limitations in the near future.

> Or are we just using AI to remix existing general knowledge, while making no progress of any sort in any field using it?

AIUI they are generally not talking about LLMs here.

If you're looking for breakthroughs due to AI, they're not going to be obviously attributable to AI I think. Focusing on the biology related foundation models ... The ability to more quickly search through the space of sequences->structure, drugs, and predicted cell states (1) like with the biology based foundation models will certainly lead to some things being discovered/rejected/validated faster.

I heard about this vevo company recently so it's on my mind. Biology experiments are hard and time consuming, and often hard to exactly replicate across labs. A lot of the data is of the form

a) start in cell state X (usual 'healthy' normal state) under conditions Y (environment variables like temperature, concentrations, etc.)

b) introduce some environmental pertubation P like some drug/chemical at some concentration, or maybe multiple pertubations at once

c) record trajectory or steady/final state of cell.

This data is largely hidden within published papers in non-standardized format. Vevo is attempting to collate all of those results with considerations for reproducibility into a standard easy-to-use format. The idea being that you can gradually build up a virtual sort of input-output (causal!) model that you can throw ideas for interventions against and see what it thinks would happen. Cells and biology are obviously enormously complicated so it's certainly not going to be 100% accurate/predictive, but my experience with network models in quantitative biology plus their proclaimed results make me pretty confident it's a sound approach.

Thus approach is clearly "AI" driven (maybe I would call this ML) and if their claims are anything close to reality, this is an incredibly powerful tool for all of academia and industry. You can reduce the search space enormously and target your experiments to cover the areas the virtual model doesn't seem so good, continuously improving it in a sort of crowd-sourced "active learning" manner. A continuously improving experimentally backed causal (w.r.t. perturbations) model of cells has so many applications. Again, i don't think this will directly lead to a breakthrough, but it can certainly make the breakthrough more likely and come faster.

There are many other examples like this that are some combination of 1) collating+filtering+refining existing data into an accessible easily query-able format

2) combining data + some physically motivated modeling to yield predictions where there is no data

3) targeted, informed feedback loop via experiments, simulations, or modeling to improve the whole system where it's known to be weak or where more accuracy is desired.

Assuming it all stays relatively open, that's undeniably a very powerful model for more effective science.

And that's just one approach. In physics ML can be use for finding and characterizing phase transitions, as one example. In the world of soft matter/biophysics simulation here's a few ways ML is used:

a) more efficient generation of configurations (Noe generative models). This is a big one, albeit still kind of early stages. Historically(simplifying), to generate independent samples in the right regions of phase means you need to integrate the system for long enough to hit that region multiple times. So regions of the space separated by rare transitions will take a loooong time to hit multiple times. The solution was simply longer simulations. Now, under some restrictions, you can leverage and augment existing data (including simulation data) to directly generate independent samples in the regions of interest. This is a really big deal.

b) more efficient, complex and accurate NN force-fields. Better incorporation of many-body and even quantum effects.

c) more complex simulation approaches via improved pipelines like automated parameterization and discovery of collective variables to more efficiently explore relevant configuration space.

Again, this is tooling that improves the process of discovery & investigation and thus directly contributes to science. Maybe not in the way you're picturing, but it is happening right now.

1) vevo https://www.tahoebio.ai/

> To investigate AlphaEvolve’s breadth, we applied the system to over 50 open problems in mathematical analysis, geometry, combinatorics and number theory. The system’s flexibility enabled us to set up most experiments in a matter of hours. In roughly 75% of cases, it rediscovered state-of-the-art solutions, to the best of our knowledge.

> And in 20% of cases, AlphaEvolve improved the previously best known solutions, making progress on the corresponding open problems. For example, it advanced the kissing number problem. This geometric challenge has fascinated mathematicians for over 300 years and concerns the maximum number of non-overlapping spheres that touch a common unit sphere. AlphaEvolve discovered a configuration of 593 outer spheres and established a new lower bound in 11 dimensions.

https://storage.googleapis.com/deepmind-media/DeepMind.com/B...

(this is an LLM driven pipeline)

You have no idea what you are talking about. Every day there is plenty of research published that used AI to help achieve scientific goals.

Now LLMs are another matter, and probably a ways off before we reap the benefit beyond day to day programming / writing productivity.

> Besides protein folding, the canonical example of a scientific breakthrough from AI, a few examples of scientific progress from AI include:1

> Weather forecasting, where AI forecasts have had up to 20% higher accuracy (though still lower resolution) compared to traditional physics-based forecasts.

> Drug discovery, where preliminary data suggests that AI-discovered drugs have been more successful in Phase I (but not Phase II) clinical trials. If the trend holds, this would imply a nearly twofold increase in end-to-end drug approval rates.

Sure, it is often flag-planting, but when these papers come from big corps, you cannot "just ignore them and keep on" even when there are obvious flaws/issues.

It's a race over resources, as a (former) researcher on a low-budget university, we just cannot compete. We are coerced to believe whatever figure is passed on in the literature as "benchmark", without possibility of replication.

I agree it's a problem across all of science, but AI seems to attract more than it's fair share of researchers seeking fame and fortune. Exaggerated claims and cherry-picking data seem even more extreme in my limited experience, and even responsible researchers end up exaggerating a bit to try and compete.

Thing is we just see that it's copy pasting stack overflow, but now in a fancy way so this is sounding like "I asked Google for a nearby restaurant and it found it in like 500ms, my C64 couldn't do that". It sounds impressive (and it is) because it sounds like "it learned about navigating in the real world and it can now solve everything related to that" but what it actually solved is "fancy lookup in a GIS database". It's useful, damn sure it is, but once the novelty wears off you start seeing it for what it is instead of what you imagine it is.

Edit: to drive the point home.

> claude just generated that

What you think happened is AI is "thinking" and building a ontology over which it reasoned and came to the logical conclusion that this script was the right output. What actually happened is your input correlates to this output according to the trillion examples it saw. There is no ontology. There is no reasoning. There is nothing. Of course this is still impressive and useful as hell, but the novelty will wear off in time. The limitations are obvious by this point.

Everyone is a rational actor from their individual perspective. The people hyping AI, and the people dismissing the hype both have good reasons.

The is justification to see this new tech as ground breaking. There is justification to be weary about massive theft of data and dismissiveness of privacy.

First, acknowledge and respect that there are so many opinions on any issue. Take yourself out of the equation for a minute. Understand the other side. Really understand it.

Take a long walk in other people’s shoes.

LLMs are undeniably treading onto that territory. Who knows how far in they will make it, but the wall is breached. Which is unsettling to down right scary depending on your take. It is a real threat to a skill that many have honed for years and for which is very lucrative to have. Programmers don't even need to be replaced, having to settle for $100k/yr in a senior role is almost just a scary.

I've seen this over and over again and been on different sides of the question on different technologies at different times.

To me, this is same as it ever was!

I agree that this is useful! It will even take natural language and augment the script, and maybe get it right! Nice!

The AI is combing through scraped data with an LLM, and conjuring forth some imagemagick snippets into a shell script. This is very useful, and if you’re like most people, who don’t know imagemagick intimately, it’s going to save you tons of time.

Where it gets incredibly frustrating is tech leadership seeing these trivial examples, and assuming it extrapolates to general software engineering at their companies. “Oh it writes code, or makes our engineers faster, or whatever. Get the managers mandating this, now! Also, we need to get started on the layoffs. Have them stack rank their reports by who uses AI the best, so that we are ready to pull the trigger.”

But every real engineer who uses these tools on real (as in huge, poorly written) codebases, if they are being honest (they may not be, given the stack ranking), will tell you “on a good day it multiplies my productivity by, let’s say, 1.1-2x? On a bad day, I end up scrapping 10k lines of LLM code, reading some documentation on my own, and solving the problem with 5 lines of intentional code.”

Please, PLEASE pay attention to this details that I added: Huge, poorly written codebases. This is just the reality at most software companies that have graduated from series A startup. What my colleagues and I are trying to tell you, leadership, is that these “it made a script” and “it made a html form with a backend” examples ARE NOT cleanly extrapolating to the flaming dumpster fire codebases we actually work with. Sometimes the tools help! Sometimes, they don’t.

It’s as if LLM is just another tool we use sometimes.

This is why I am annoyed. It’s incredibly frustrating to be told by your boss “use tool or get fired” when that tool doesn’t always fit the task at hand. It DOES NOT mean I see zero value in LLMs.

scientists don't need to be efficient, they need to be correct. Software bugs were already a huge cause of scientific error, and responsible for lack of reproducibility, see for example cases like this (https://www.vice.com/en/article/a-code-glitch-may-have-cause...)

Programming in research environments is done with some notoriously questionably variation in quality, as is the case for the industry to be fair, but in research minor errors can ruin results of entire studies. People are fed up and come to much harsher judgements on AI because in an environment like a lab you cannot write software with the attitude of an impressionist painter or the AI equivalent, you need to actually know what you're typing.

AI can make you more efficient if you don't care if you're right, which is maybe cool if you're generating images for your summer beach volleyball event, but it's a disastrous idea if you're writing code in a scientific environment.

Where ML generally shines is either when you have relatively lots of experimental data with respect to a fairly narrow domain. This is the case for machine learned interatomic potentials MLIPs which have been a thing since the '90s. Also potentially the case for weather modelling (but I do not want to comment about that). Or when you have absolute insane amounts of data, and you train a really huge model. This is what we refer to as AI. This is basically why Alphafold is successful, and Alphafold still fails to produce good results when you query it on inputs that are far from any data points in its training data.

But most ML for physics problems tend to be somewhere in between. Lacking experimental data and working with not enough simulation data because it is so expensive to produce. And also training models that are not large enough, because inference would be too slow, anyway, if they were too big. And then expecting these models to learn a very wide range of physics.

And then everyone jumps in on the hype train, because it is so easy to give it a shot. And everyone gets the same dud results. But then they publish anyway. And if the lab/PI is famous enough or if they formulate the problem in a way that is unique and looks sciency or mathy, they might even get their paper in a good journal/conference and get lots of citations. But in the end, they still only end up with the same results as everyone else: replicates the training data to some extent, somebody else should work on the generalizability problem.

The only realistic evaluations of "AI" so far are those that admit it's only useful for experts to skip some boring work. And triple check the output after.

In particular the ability of auto regressive transformer based networks to produce sequences speech while being immutable still shocks me whenever I think about it. Of course, this says as much about what we think of ourselves and other humans as it does about the matrices. I also think that the weather forcasting networks are quite shocking, the compression that they have achieved in modeling the physical system that produces weather is frankly.... wrong... but it obviously does actually work.

One typical use case is that the simulation data takes months to generate. So for experimental use cases, it is very slow. So the idea was, to train a model that can learn the underlying physics. The model will be small enough so that inference won't be prohibitively expensive. So you can then use the ML model in lieu of the classical physics based model.

Where this usually fails is that while ML models can be trained well enough to replicate the training data, they typically fail to generalize well outside of the domain and regime of the training data. So unless your experimental problems are entirely within the same domains and regimes as the training data, your model is of not much use.

So claims of generalizability and applicability are always dubious.

Lots of publications on this topic follow the same pattern: conceive of a new architecture or formalism, train an ML model on widely available data, results show that it can reproduce the training data to some extent, mention generalizability in the discussion but never test it.

I used to work in academia and was involved in NSF-funded programs, so I have mixed feelings about this. Waste and inefficiency are rampant. BTW I'm not talking about failed projects or research that were dimmed "not important", but things like buying million-dollar equipment just to use up the budget, which then sits idle.

That said, slashing NSF funding by 50% won’t fix any of that. It’ll just shrink both the waste and the genuinely valuable research proportionally. So it's indeed a serious blow to real scientific progress.

I don’t really have a point, just want to put it here. Also to be fair this kind of inefficiency isn't unique to academia; it’s common anywhere public money is involved.