Oh, I see you returned the rifle. Good move.
This is a known problem and there is a decent fix for the up and down wobble. It's even ok that the plastic detent wedge is "wallered out" (great term!) and loose in its socket, like mine is, too. I cut a piece of K&S brass tubing from a hobby shop to fit over the detent bar to thicken it. Choose a diameter tubing that will fit tightly over the bar, cut the tube to length, then cut a space out of the side of the tube so it can snap over the bar side-on and stay. I got this idea from someone else here in this forum. With a thicker detent bar, the wedge pushes harder against the bar and forces the detent pin hard against the side of its socket, which reduces the up and down slop. I did this last year and my Gamo may have two tubes over the bar, I can't remember. I hope this helps others with the same problem.