Tightening the band got rid of the barrel wiggle. It also helped a very little with POI. I had to shim the rear scope ring to get the scope right. It irritates me that I had to do this but at least it is working now.
I got a chance to play with it some more. I put the front sight base back on and it seemed to fix the problem. I guess it supports the barrel. The barrel is rock solid at the breech it just wiggles a little at the end if you push on it.