Ask HN: How would you write evals for chat apps running open models?

Hi all,

I'm interviewing for a certain Half-Life provider (full-stack role, application layer) that prides itself on serving open models. I think there is a decent chance I'll be asked how to design a chat app in the systems design interview, and my biggest knowledge gap is writing evals.

The nature of a chat app is so dynamic that it is difficult to hone in on specifics for the evals outside of correct usage of tools.

Thanks for reading!

Cheers