I would agree in most small game cases, where just poking a hole through the animal, that the small calibers are enough. But here in the west, it was won with big slow moving chunks of lead, killing the buffalo. The larger meplat of a big caliber projectile, really smacks the animal good, knocking it down. If you use a small caliber high BC slug, on too big an animal, it is going to have the same effect as shooting an animal with a field point instead of a broad-head. If you are back yard plinking between houses, then in my opinion even a 25 cal might be too big, having more than enough energy to penetrate fence slats and house walls.
It really comes down to intended purpose and since I live where people are spread out I go big because we are not regulated to the size nonsense that some locations have. Having the large meplat projectiles ensures that I do more than poke a hole in the animal. I know a lot of people harp on shot placement but often those people do not practice enough the actually place the shot consistently.
I have shot plenty of deer, elk and caribou of the years, using archery, modern firearms and black powder to know that if the shot does not take out both lungs and the heart or wipe out the brain pan the animal can run and that is where a big piece of lead knocking it down really helps out in hastening the kill.