I imagine prey type is a significant determinant. Can it be taken down by a single hit with a heavier, but slower weapon? Or is rapid fire from a lighter weapon more effective?
Prey type is definitely a factor, but it's hard to predict how that plays out in practice.
Hunting scenarios rarely allow for "rapid fire". You generally get one shot (per hunter), then the prey bolts. Warfare, though, might be a different story.
My guess is that a larger, heavier weapon is likely to make a larger wound and result in more damage. That means more "stopping power", which would be important if there is a risk of a large animal charging at you. A really heavy spear can also be used hand-held to fend off an attack, and won't bend or break. (There's a reason why spears were the traditional weapon for boar-hunting in Europe.)
In theory an arrow can compensate for lower mass with increased velocity. A simple bow would get you two to four times the projectile velocity of an atlatl, depending on size, with maybe a quarter of the projectile weight. So in terms of kinetic energy (mass times the
square of velocity) a bow could easily be advantageous. How that works in practice with flint tips, flexible shafts, and thick animal hides is not entirely clear, but from watching Tod's Workshop on Youtube ("arrows versus armor") I can say that a lot of the projectile's mass is in the shaft, which, being flexible and fragile, won't transfer energy well. A bigger stone point should make a big a difference, and there's a limit to how big an arrowhead can be without completely unbalancing the arrow.
A smaller projectile that travels faster should give you more range, which is an important safety factor. It's also probably easier to fire a bow without breaking concealment than it would be to use an atlatl. You
might even have a chance of taking multiple shots before the prey moves out of sight or out of range (or before it can close the distance to stomp you).
My understanding is that hunting large game (deer-sized and up) with a bow often means "wound it with an arrow, then follow its trail for hours while it slowly bleeds to death." If the prey is large enough compared to your arrowhead, it may stop bleeding before you can catch up to it. Smaller game, though, wouldn't have that problem. So for something like rabbits, a bow looks like a much better choice than an atlatl.
So... which game are you hunting? Are you hunting the same game all the time? What's your hunting style - how much risk are you taking? Are you hunting alone or in groups? Is your prey alone or in a herd? How much concealment do you / the prey have? Is your prey cornered, or are you sneaking up on it? Is your weapon choice flexible enough to handle all the likely variations?
And what about the risk that your hunting party runs into a hostile band of humans? If you're carrying only one type of ranged weapon, it better be a good one. Maybe the bow's ability to carry more ammo would be an important factor here. Range too, depending on the environment. (Range doesn't matter too much in thick forests.)