Given same axle/sheave friction a larger pulley will give more leverage over the friction, and so thereby be more efficient; and bend line less; and given the same amount of rope running thru each pulley; a larger pulley will make less revolutions: have a longer life/ less wear. Similarly a larger wheelbarrel wheel on same axle will be more efficient (but s-lightly harder to start rolling); and last longer on tire tread and axle/bearings due to the distance modifier.
If lowering a load thru pulley; with the load on 1 end and control totally separate on the other end; the force must crank thru pulley; so pulley diameter bending line makes a bigger difference in how line is leveraged. In my imagery, a load position on 1 end of line thru pulley, that the control on equal/opposite end that also connects back to load (as in tree climber's DdRT); so that load(climber) belays/controls self; pulley size makes less difference on the leveraging of the line(but not on the efficiency of the pulley). i see this because in this setup; the force transfer can be across the air to the other leg of line support (dead ending at saddle); not so much thru pulley/support; especially as that positicon becomes more frictional. This small factor is why a 100# man should not descend on a friction hitch in SRT, but a 300#man can descend on same in DdRT(even though the tree climber has 2/1 over self, on self lift in DdRT).
But, ye are write to say, that the axle on a small pulley is small; especially pulled by your weight forces perpendicular to the spread supports of the cheek plates support on the axle.
i think that 8/1 is about perfect/ no leverage strength loss example we see for supple braids, 10/1 on 3 strand, but this is all relative to line's resistance to bending(because the resistance to bend is leveraging). Thus steel cable requires more (and specially shaped sheave to support the 'loaded belly' of the cable). Kevlar etc. can easily get up to 30x (which cable exceeds) and also needs specially shaped/grooved sheaves. Leveraging by bend in Standing part is different than by bight), not a strength loss, but load force increase; to leave less useable range of rope tensile rating remaining. The rope leveraging around the bight of a small turn/pulley is an actual strength loss, by compressing fibers on inside of bight. The compressed / push fibers don't support the load activity in this tension only device/rope, so only the outer arched fibers are tensioned/ helping to carry the load. Thereby, to leave less useable range of rope tensile rating remaining; just by different means.