Pretraining a LLM with less than $50 budget which outperforms Google BERT

Comments (1)

spindump8930 · 5h ago

The title makes it sound nice but the reported results are worse than random baselines on several benchmarks, including ones to claim superiority over BERT. At a glance, Hellaswag, boolq, winogrande are all at or below random guessing. At best this is a fun model with broken evaluation. At worst this is medium spam for clout farming - which won't work on anyone who can read the tables.

Ask HN: Who is hiring? (September 2025)

Ask HN: Who wants to be hired? (September 2025)

Ask HN: Why does Seattle feel so risk-averse compared to the Bay Area?

LinkedIn seems to be leaking Google Docs

Ask HN: Best foundation model for CLM fine-tuning?

Understanding Android's Boot Process

Ask HN: Tools for Crossword Puzzle Generation?

Ask HN: The government of my country blocked VPN access. What should I use?

Ask HN: How do you fight YouTube addiction and procrastination? I'm struggling

Ask HN: Do custom ROMs exist for electric cars, for example Teslas?

Tell HN: Use "-f**k" to kill Google AI Overview

Change Tracker: Monitor+revert file edits from Claude/AI agents(in-memory VCS)

Ask HN: Why hasn't x86 caught up with Apple M series?

Ask HN: What to learn for math for modeling?

Tell HN: My advice after I applied to 450 positions before getting hired

Ask HN: Which Open Source License to Choose for a Python Language Server

Ask HN: Did Developers Undermine Their Own Profession?

Hacker News Alternativies

Ask HN: How can I recover and run my old mobile game from the 2010s?

Ask HN: How much are you guys paying for AI coding tools monthly?

JDeploy 5.0: Deploy Java Desktop Apps to ARM64 Windows and Linux with One Click

FUGC: Understand the GC in Fil-C

Ask HN: What great book are you currently reading or recently finished?

Looking for Info on Aerosol Pathogen Detection

Ask HN: Anyone using their own custom text editor?

Ask HN: Best self-hosted wiki solution in 2025? Mediawiki or something else?

Ask HN: Any Android Engineers Here?

Ask HN: What do you think about GFW?

Ask HN: Best codebases to study to learn software design?

Ask HN: What to Do with Old iPads?

Ask HN: Should we stop worrying that AI will replace developer jobs?

Ask HN: What options do I have for self-hosted end to end encrypted group chat?

Ask HN: What to do when you suspect your interview is with a state operative?

Pretraining a LLM with less than $50 budget which outperforms Google BERT

Comments (1)