Activity annealing leads to a ductile-to-brittle transition in amorphous solids