Once the ends take on a flair from being pounded on your kinda screwed.should not take but a solid firm hit to have it slide/move within the hole.A LARGE HEAVY hammer hit is far better than lighter tap taps.
Yup, Quality HS steel or CARBIDE drill at low rpm and the roll pin will come right out. Drilled at a size LESSOR than pins O.D. obviously.Just hold it down in a padded vise using a drill press or milling machine.