Show HN: AI Peer Reviewer – Multiagent system for scientific manuscript analysis

I would say the exact opposite. Let the machine do the easy/simple/boring stuff. Let the human peer reviewer do the big question. That's what the human is good at and excited about and it's what the machine will not be good at. The question is a philosophical one: Is this a good idea? Is it relevant? Is it important? This is highly subjective and needs folks in the field to build consensus about. Back in my PhD days, I'd have loved if a machine could have taken care of the simple stuff so humans could focus entirely on the big questions.

(A machine could point to similar work though.)

Thanks for the thoughtful feedback. That’s very helpful.

We didn’t think too deeply about the term “AI peer reviewer” and didn’t mean to imply it’s equivalent to human peer review. Based on your comments, we’ll stick to using “AI reviewer” going forward.

Regarding security: there is an open-source version for those who want full control. The free cloud version is mainly for convenience and faster iteration. We don’t store manuscript files longer than necessary to generate feedback (https://www.rigorous.company/privacy), and we have no intention of using manuscripts for anything beyond testing the AI reviewer.

On novelty. totally agree it’s a core part of good peer review. The current version actually includes agents evaluating originality, contribution, impact, and significance. It’s still v1 of course but we want to improve it. We'd actually love for critical thinkers like you to help shape it. If you're open to testing it with a preprint and sharing your thoughts on the feedback, that would be extremely valuable to us.

Thanks again for engaging, we really appreciate it.

My view, and how I conduct my peer reviews, is that I do not care to make decisions of whether a question is important/interesting or not. I feel like my job is to judge the paper on rigor, and whether it fails (purposely or from ignorance) to address/acknowledge the relevant literature.

I'm also a PhD[0] and researcher who has worked in various fields, including national labs too (You DOE?)

I mostly share the same sentiment, and I see a similar issue with the product. The current system is not in its current poor state due to lack of reviewers, it is due to lack of quality reviewers and arbitrary notions of "good enough for this venue." So I wanted to express a difference of opinion about what peer review should be (I think you'll likely agree).

I don't think we are doing the scientific community any service by doing our current Conference/Journal based "peer review". The truth is that you cannot verify a paper by reading it. You can falsify it, but even that is difficult. The ability to determine novelty and utility is also a crapshoot, where we have a long history illustrating how bad we are at this. Several Nobel prize worthy works have been rejected multiple times due to "obviousness", "lack of novelty", and "clearly wrong." All three apply to the paper that led to the 2001 Nobel Prize in Economics[1]!

The truth of the matter is that peer review is done in the lab. It is done through replication, reproduction, and the further development of ideas. What we saw around LK-99[2] was more quality and impactful peer review than what any reader for a venue could provide. The impact existed long before any of those works were published in venues.

I think this came down to forgetting the purpose of journals. They were there when we didn't have tools like ArXiV, OpenReview, or even GitHub. Journals were primarily focused on solving the logistic problem of distribution. So I consider all those technical works, "preprints", and blog posts around LK-99 replications as much of a publication as anything else. The point is that we are communicating with our peers. There's always been prestige around certain venues, but primarily people did not publish to them. The other venues checked for plagiarism, factual errors, and any obvious errors. Otherwise, they continued with publication.

This silly notion of acceptance rates just creates a positive feedback loop which is overburdening the system (highly apparent in ML conferences). The notions of novelty and impact are highly noisy (as demonstrated in multiple NeurIPS studies and elsewhere), making the process far more random than acceptable. I don't think this is all that surprising. It is quite easy to poke flaws in any work you come across. It does not take a genius to figure out limitations of works (often they're explicitly stated!).

The result of this is obvious, and is what most researchers end up doing: resubmit elsewhere and try your luck again. Maybe the papers are improved, maybe they aren't, mostly the latter. The only thing this accomplishes is an exponentially increasing number of paper submissions and slowing down of research progress as we spend time reformatting and resubmitting which should instead be spent researching. The distribution of quality review comments seems to have high variance, but I can say that early in my PhD they mainly resulted in me making my works worse as I chased their comments rather than just trying to re-roll and move on.

In this sense, I don't think there's a "lack of reviewer" problem, so much as we have an acceptance threshold problem with an arbitrary metric. I think we should check for critical errors, check for plagiarism, and then just make sure the work is properly communicated. The rest is far more open to interpretation and not even us experts are that good at it.

[0] Well my defense is in a week...

[1] https://en.wikipedia.org/wiki/The_Market_for_Lemons

[2] https://en.wikipedia.org/wiki/LK-99

Thanks for the feedback. Totally agree. It’s a real shame we don’t have more historical peer review data. It would be great if research was fully transparent.

We did find a few datasets that offer a starting point: https://arxiv.org/abs/2212.04972 https://arxiv.org/abs/2211.06651 https://arxiv.org/abs/1804.09635

There’s also interesting potential in comparing preprints to their final published versions to reverse-engineer the kinds of changes peer review typically drives.

A growing number of journals and publishers, like PLOS, Nature Communications, and BMJ—now publish peer review reports openly, which could be valuable as training data.

That said, while this kind of data might help generate feedback to improve publication odds (by surfacing common reviewer demands early), I am not fully convinced it would lead to the best feedback. In our experience, reviewer comments can be inconsistent or even unreasonable, yet authors often comply anyway to get past the gate.

We're also working on a pre-submission screening tool that checks whether a manuscript meets hard requirements like formatting or scope for specific journals and conferences, hoping this will save a lot of time.

Would love to hear your take on what kind of feedback you find useful, what feels like nonsense, and what you would want in an ideal review report... via this questionnaire https://docs.google.com/forms/d/1EhQvw-HdGRqfL01jZaayoaiTWLS...

I wish my own manuscripts would be that important...

Regarding security concerns: there is an open-source version for those who want full control. The free cloud version is mainly for convenience and faster iteration. We don’t store manuscript files longer than necessary to generate feedback (https://www.rigorous.company/privacy), and we have no intention of using manuscripts for anything beyond testing the AI reviewer.

I guess the paper would be complete enough to publish as a preprint at the stage where this specific service is most useful

The AI pseudoagent[1], unless sentient and proficient in the chosen field of expertise, is not a peer. It's just a simulacrum of one. As such, it can only manifest simulacra of concepts such as "biases", "fairness", "accountability", etc.

The way I see it, it can function, at best, as a tool of text analysis, e. g. as part of augmented analytics engines in a CAQDAS.

1. Agents are defined as having agency, with sentience as an obvious prerequisite.

Does peer review typically select for the same demographics as the author?

I was joking, but probably so, to the extent that 80% of peer reviewers are men and 80% of authors of peer reviewed articles are men[0]

0. https://www.jhsgo.org/article/S2589-5141%2820%2930046-3/full...

Thanks for sharing. We’ll take a closer look. There’s definitely something we can learn from it.

System prompts / review criteria cannot be "leaked" because they are open-source (full transparency). Focusing heavily on monetization at this stage seems shortsighted...this tool is a small (but longterm important) step of a larger plan.

Right now, the core workflow takes about 8 minutes locally, mostly because we haven’t optimized anything yet. There’s plenty of low-hanging fruit: parallelization, smarter batching, etc. With a bit of tuning, we could bring it down to 1–2 minutes. That said, we’ll probably want to add more agents for deeper feedback and quality control, so the final runtime might go up again. At this stage, we're figuring out what’s actually useful, what isn’t, what additional signals we should look at, and what the right format and level of detail for feedback should be. The cloud version includes a manual review step, partly to control costs, partly because we’re still discovering edge cases. So the current 1–2 day turnaround is really just a safety net while we iterate. If we decide to continue with this, the goal is to deliver meaningful feedback in minutes.

That eight month delay is caused by backlog, a human reviewing an individual paper takes about a day.

Totally agree! The tool is designed to provide early-stage feedback, so experts can focus their attention on the most relevant points at later review stages.

Here is a description of how it works: https://github.com/robertjakob/rigorous/tree/main/Agent1_Pee...

> I came to the realization “Code Reviews Are Dead”.

If that's how it works at your company then run as fast as you can. There are many reasonable alternatives that won't push this AI-generated BS on you.

That is, if you care. If you don't then please stay where you are so reasonable places don't need to fight in-house pressure to move in that direction.

Oh my god.. The horror.. Please do not let this be my future..

wild times

IMO that's what this focus on. Language. That's what LLMs excel at. Perhaps branch out to providing localized papers to markets like China or France (hah, sorry).

Judging the actual contents may feel like the holy grail but is unlikely to be taken well by the actual academic research community. At least the part that cares about progressing human knowledge instead of performative paper milling.

Haha fair point, domain name was a 5-second, “what’s available for $6” kind of decision. Definitely not trying to go full corporate just yet

Thanks for the heads-up. We'll raise the file size limit shortly.

Best feedback so far!

You're right: In the current version each "agent" essentially loads the whole paper, applies a specialized prompt, and calls the OpenAI API. The specialization lies in how each prompt targets a specific dimension of peer review (e.g., methodological soundness, novelty, citation quality). While it’s not specialization via architecture yet (i.e., different models), it’s prompt-driven specialization, essentially simulating a review committee, where each member is focused on a distinct concern. We’re currently using a long-context, cost-efficient model (GPT-4.1-nano style) for these specialized agents to keep it viable for now. Think of it as an army of reviewers flagging areas for potential improvement.

To synthesize and refine feedback, we also run Quality Control agents (acting like an associate editor), which reviews all prior outputs from the individual agents to reduce redundancy and surface the most constructive insights (and filter out less relevant feedback).

On your point about nitpicking: we’ve tested the system on several well-regarded, peer-reviewed papers. While the output is generally reasonable and we did not discover "made up" issues yet, there are occasional instances where feedback is misaligned. We're convinced, however, we can almost fully reduce such noise in future iterations (Community Feedback is super important to achieve this).

On the code side: 100% agree. This is very much an MVP focused on testing potential value to researchers, and the repeated agent classes were helpful for fast iteration. However, your suggestion of switching to template-based prompt loading and dynamic agent registration is great and would improve maintainability and scalability. We'll 100% consider it in the next version.

The _determine_research_type method is indeed a stub. Good catch. Also, lol @ the JS comment hashes, touché.

If you're open to contributing or reviewing, we’d love to collaborate!

Hi azalemeth, I'd be happy to run the script on your manuscript and debug the issue. Could you send me your contact info via X (https://x.com/robertjakob) or LinkedIn (https://www.linkedin.com/in/robertjakob/) so I can send the feedback report back to you once it's ready? Feedback would be super appreciated.

Once you receive the report, we'd really appreciate your feedback on what we can improve via https://docs.google.com/forms/d/1EhQvw-HdGRqfL01jZaayoaiTWLS...

thanks! We'll get back to you with the feedback report (currently receiving loads of submissions).

Fair point.

We're in very early MVP mode, trying to move fast and see if this works. We pushed a Cloud version to support users who don't want to run the GitHub script themselves. That said, you're absolutely encouraged to run it yourself (with your openAI key) — the results are identical.

For context: we're two recent ETH Zurich PhD graduates.

Robert Jakob: https://www.linkedin.com/in/robertjakob Kevin O'Sullivan: https://www.linkedin.com/in/kevosull

Going to add contact information immediately.

Thanks again for the feedback — it's exactly what we need at this stage.

Even extremely privacy-conscious authors could submit their paper to the service at the same time they publish their preprint v1, then if the service's feedback is useful, publish preprint v2 and submit v2 as the version of record.

It seems you have the option to run the tools yourself (with an OpenAI API key). The cloud version is for convenience. I agree that a privacy/usage policy is necessary.

I agree. I've worked at a national lab before and I immediately thought this service is a massive security risk. It will definitely be hard for some scientists to use these kind of cloud services, especially if their research truly is cutting edge and sensitive. I think many people will just ignore things like this because they want to keep their jobs, etc.

Just upload an already published paper to test it