Why spotting cosmic visitors is harder than you think