Trump offered to let detained Korean workers stay, but almost all heading home