Robustly Improving LLM Fairness in Realistic Settings via Interpretability