It is time to stop teaching frequentism to non-statisticians (2012)

On the subject of prioritizing EDA:

I need to look this up, but I recall in the 90s a social psychology journal briefly had a policy of "if you show us you're handling your data ethically, you can just show us a self-explanatory plot if you're conducting simple comparisons instead of NHST". That was after some early discussions about statistical reform in the 90s - Cohen's "The Earth is round (p < .05)" I think kick-started things off.

Definitely. It always amazes me that in many situations, I'm applying some stats algorithm just to conclude: let's look at these data some more...

Yes. And the same for DS/ML people also, please. The amount of ML people that can meaningfully drill down and actually understand the data is surprisingly low sometimes. Even worse for being able to understand a phenomena _using data_.

You can never state the full nature of your experiment. Even the simplest experiment under the most controlled conditions has a bunch of unknown unknowns - you can never be certain that you didn't get a bad batch of reagents or a bit flip in your data or just royally screwed something up. Unless you're omniscient, there are always unknown unknowns lurking somewhere in your method.

In the case of your survey, you can't really state anything about the students you sampled with absolute confidence. You might have been asking a question about something that a lot of people are inclined to lie about. You might have been subconsciously biased and subtly influenced people towards the "right" answer. Slight inconsistencies in the wording of the question might radically alter how people respond. The student you tasked with conducting the interviews might have just stayed in bed and fabricated the data.

Bayesianism gives us an incredibly powerful framework for reasoning about this innate and unavoidable fog of uncertainty; frequentism largely pretends that it doesn't exist. The ongoing replication crisis shows why this is not merely pedantry, but the single most urgent issue in science.

Well even for simple things there's a large difference. Say you toss a coin N times and observe heads x times. What is the probability of your next toss coming up heads?

A frequentist arguably would say the question doesn't really have any meaning since probabilities are about long run frequencies of things occuring. They might do various tests or tell you the probability of that outcome under various probabilities for heads.

A Bayesian would make an initial assumption about the probability of any given probability, and then compute a posterior using the likelihood function the frequentist may have, and give you a distribution for what you should believe about the what the true probability of heads is on your next coin toss.

In general, the latter is more meaningful and informative. There's also pretty good arguments that any coherent method of representing credences is isomorphic to probability, see Cox's theorem.

> If we state the full nature of our experiment, what we controlled and what we didn't... how can it be a "degree of belief"?

Here's a question for you: what's the millionth digit of Pi? (In the standard decimal expansion, of course.)

This is a question (kind of an "experiment") which literally has only one, constant answer, that is theoretically knowable. But unless you search online, or happen to know the answer, you actually don't know which of the digits 0-9 it is. And I can just as easily ask about the 10^100th digit of Pi, which is, again, a constant - and yet no one knows what it is.

So using the Frequentist approach to statistics doesn't make much sense - there's no repeated experiment with possible different outcomes.

But there is a real sense in which your answer should be "it's one of the digits 0-9 with 1/10 probability each". That answer makes sense, because the answer isn't unknowable, just unknown, and probability reflects your lack of knowledge and degree of belief.

I think Bayesian methods have made ground in sciences such as Sociology, Psychology and Ecology, which are mostly observational, but still attempt to make models with intepretable parameters.

With observational studies, representing confounders and uncertainty is a primary concern, because they are the most important source of defeater. Here, Bayesian software such as brms, Stan, pyMC, become a flexible way to integrate may sources of uncertainty. Although, I suspect methods like SEM still dominate for their use cases.

Personally, I find myself using Bayesian methods in a similar bag-of-tricks way that I use Frequentist methods mostly because its difficult to believe that complex phenomena is well described by either, so I use whatever makes the case best.

NHST, which is part of frequentist statistics, is wrong, plain and simple. It answers the wrong question (what's the probability of the data given the hypothesis vs. what's the probability of the hypothesis given the data), and will favor H1 under conditions that can be manipulated in advance.

There is a total lack of understanding of how it works, but people think they know how to use it. There are numerous articles out there containing statements like "there were no differences in age between the groups (p > 0.05)". Consequently, it is the wrong thing to teach.

That's apart from the more philosophical question: what does it mean when I say that there's a 40% chance that it team A will beat team B in the match tomorrow?

Frequentism methods are strictly less general. For example Laplace used probability theory to estimate the mass of Saturn. But with a frequentist interpretation we have to imagine a large number of parallel universes where everything remains the same except for the mass of Saturn. That's overly prescriptive of what probability means. Whereas in Bayesian statistics what probability means is strictly more general. You can manipulate probabilities even without fully defining them (maximum entropy) subject to intuitive rules (sum rule, product rule, Bayes' theorem), and the results of such manipulation are still correct and useful.

Frequentist methods are unintuitive and seemingly arbitrary to a beginner (hypothesis testing, 95% confidence, p=0.05).

Bayesian methods are more intuitive, and fit how most be reason when they reason probabilistically. Unfortunately Bayesian computational methods are often less practical to use in non-trivial settings (usually involves some MCMC).

I'm a Bayesian reasoner, but happily use frequentist computation methods (max likelihood estimation) because they're just more tractable.

Frequentist stats are wrong. They are built entirely on a flawed, non-consistent premise. The only reason for their use is because Bayesian approaches were computationally unfeasible for a long time, but that is no longer the case.

> Why not just use a blog site, medium or substack?

Because it looks more credible, obviously. In a sense it's cargo cult science: people observe this is the style of science, and so copy just the style; to a casual observer it appears to be science.

Two reasons:

1. Preprint servers create DOIs, making works better citable.

2. Preprint servers are archives, ensuring works remain accessible.

My blog website won't outlive me for long. What happened to geocities could also happen to medium.

> It’s weird how random people can submit non peer reviewed articles to preprint repos.

Assuming that you are referring to the Arxiv, they can't:

https://info.arxiv.org/help/endorsement.html

Why the gatekeeping. Only what is said matters, not who says it.

I would start with github. Arxiv, show hn, twitter, TikTok , Super Bowl ad if you are going for max look at me, i am not wearing diapers effect.

>It’s weird how random people can submit non peer reviewed articles to preprint repos.

It is weird how people use a platform exactly how it is supposed to be used.

I've published stats papers on Bayesian methodology, consider myself a kind of Bayesianist, and stances like the link essay, and many of the posts here, make me sad and a bit frustrated. As you're saying, it's become an ideological, not functional, debate. Different inferential paradigms have their benefits and costs, and Bayesian methods have their own problems.

Whenever you make an inference, it's a gamble, in a sort of literal sense. The choice of frequentist versus Bayesian methods basically is a bet about whether the bias due to your prior is small enough to offset the variance due to the lack of one. If the bias due to a "misguided" prior is high enough, you'll end up making a worse inference than not using a prior.

You can, of course, use an optimally conservative prior, a reference prior, but for many cases what you're left with is a uniform prior, and therefore, frequentist inference. It's not always the same as a uniform prior, but in practice is often the same. I think the philosophical arguments against frequentist lines of thoughts are often unresolvable and involve strawmen characterizations of frequentism often derived from misperceptions of young adults learning any sorts of stats for the first time.

There's also a very strong argument against strong priors in competitive situations, such as in achievement tests for admissions or some such thing. Imagine making priors for your test score based on your demographic background, for instance. Technically this might be ok from a subjective Bayesian perspective but I think almost no one would agree this is acceptable, and the reasons why are telling about statistical inference more generally.

Many of the other arguments equally apply to Bayesian methodology too. The worst problems of NHST are not actually about the tests, they're about how they are used and interpreted, and would creep up if everyone was using credibility intervals too.

I kind of think Bayesian methodology should be taught more at an earlier stage, but it would be irresponsible in my mind to not teach frequentist methodology at the same time.

> You have a jar with five green, three blue marbles. First random marble you pick is green, what is the chance you get blue next? There is no use for bayes here.

There’s no use for Frequentism there either, it’s basic probability theory.

You're right that the dice are literally intended to represent a hidden variable. The difference between the frequentost and the Bayesian is that the frequentist only accounts for the hidden variable, whereas the Bayesian also accounts for the pre-test probability of the sun exploding. Since the probability of the sun exploding is 1 in a bazillion, multiplying that by 35 still gives you a very very low post-test probability that the sun has gone nova because 35 in a bazillion is still pretty unlikely

It is like running a casino versus gambling as an individual.

If I was running a casino and presiding over tens of thousands of bets daily, I would use frequency statistics to guarantee the house always wins.

If I was an individual gambler and determining my odds at a particular bet, I would use Bayesian.

It depends what the null hypothesis is here, but by construction, under a reasonable null, the p-value for an appropriate test would not be acceptable under a Frequentist framework.