To get the right answer, you have to imagine there is an expansion (rarefaction) wave within the barrel, and the pellet is riding the front of that wave. If you do this, you find a well-behaved "theoretical" maximum velocity which depends on the specific heats of the gas.
In your derivation of your formula you made use of the isothermal expansion assumption, even if you didn't state it explicitly. That happened when you assumed the force was equal to P0*A for "any" position of the pellet in the bore.A formula like yours may in fact give good estimates of max velocity, but that doesn't make it a theoretical maximum. Deriving the theoretical maximum is a lot more complicated because you must take into account the non-stationary of the expansion within the bore, and this is no longer an algebraic problem.By the way, if I am not mistaken, implicit in your formula is an efficiency equal to [gamma - 1], where gamma is the ratio of specific heats, which in air is 1.4. This means that, per your formula, 40% of the thermal energy of the gas that enters the bore gets converted into kinetic energy of the pellet.I almost forgot, the reason your formula blows up when the mass of the pellet approaches zero is because, in your formula, vel = srt(2 P0 A x/M), where M is the pellet mass, P0 the reservoir pressure, A the bore area, and x the bore length. As M approaches zero, your velocity goes to infinity like 1/sqr(M), and this is forbidden by the physics of gases (because in that case the gas molecules would not be able to keep up with the pellet!). Same with x, the bore length.
You are almost there. The formula v = SOS sqr[2/(gamma -1)] applies to "steady state" flow. This fact is absolutely critical. In a PCP we don't have steady flow.The correct formula is v = SOS 1/(gamma - 1), which applies to "unsteady" expansions (for gamma = 1.4 this corresponds to Mach about 5.) The speed of sound in this formula is the one in the reservoir.You may ask, if the first formula represents energy conservation, does this mean the second formula violates energy conservation? What people, even experienced fluid dynamic practitioners, tend to forget is that in unsteady flow the total temperature is NOT preserved. In an "unsteady" rarefaction wave (which is what you have in the bore) the energy of a fluid parcel is not constant (Bernoulli's equation is "not" valid in unsteady flow!) Unfortunately, you very rarely find any discussion of the difference between these two regimes in fluid dynamic texts.By the way, although gamma can be significantly higher in a PCP environment, this is not essential to the basic difference between steady and unsteady max velocity.
doing this invalidates your assumption that the energy imparted to the pellet is P0*A*x.
Neglecting the KE of the gas behind the pellet (in a PCP) makes a difference of a couple of Joules in the pellet energy (for a 12 fpe, 4.5 mm gun).
Fanno flow is strictly "stationary" flow, this is "not" what you have in a PCP. Think about a situation that would apply to "stationary" 1-D flow and you will see under what conditions your conclusion applies. It is actually interesting that there is a difference between stationary and non-stationary maximum values. This has to do with the transmission of energy between non-stationary fluid parcels, which doesn't happen in stationary flow. It is subtle but important distinction.