Dennis, the hole cutter (image below) I made accepts the common 1/8" shank drill bits (dremel, etc) so you would just pick the right size bit. The goal was to make it a one step drill and cut operation so you don't have to find center again. I don't use these inserts so this was a "can I do this" type of exercise. The easiest way to do this might be to make an adapter to an commercial hole cutter (usually 1/4" drill) to accept either 1/8" shank drill bits or attach something like a small collet.
I think making these with a lathe is a little tricky, though I'm no expert.
I'm still surprised that the laser cut tolerances that your vendor gave you are so loose. All the cheap things I've bought that were laser cut fit together well. I would expect tolerances in the .05 mm range not .5 mm