For one, a larger mill of 36" will work on a shorter bar, you just need to adjust it down to the size of bar you have (so you can use a 20" bar with it). But the reverse is not true: you can't increase the size of a 24" mill to cut wider than that. (or as you noticed, a 24" mill will only give 20" of cut space since all two-point connected mills lose about 4").
(*I say "two-point connected" because if you are considering the "small log mill" by Granberg, it only connects at one point. More on this in a moment)
So, if you ever intend to cut a wider log and/or use a longer bar, you might want to go with a bigger mil. And a 661 can handle a 36"bar fine. (and though you say now that you have no intentions to do so, this often is not the case later).
In regards to the small log mill by Granberg that connects only at one point and is rated 20", actually, you can cut wider than that since its only connected at the base of the bar, and the tip is not connected. On the one hand this actually allows you to use any sized bar (even a 60"), so if you only foresee cutting the occasional big log, this could be good. But on the other hand, the fact that it is not connected on both ends gives a lack of support (I've heard of them snapping at the welds due to being under uneven pressure, mainly in wider than 20" wood with someone pushing too hard) as well by being without a tip connection there is more of a likelyhood that the bar will cut uneven, particularly as the wood/bar get wider (so on say a 24" wide board, it may be 1/4" or more thicker on one side than the other).
Though if you only want to do occassional milling now and then for a few boards, and either intend to plane them or aren't overly worried about being a bit off in thickness, then the small log mill could be a good start as it costs way less than any two point connected mills. Granberg also sells conversion kits to make the small log mill into a two point mill, as well as extension frames to increase the width up to any size you want (though, these upgrades will probably cost more in the long run than just buying a 32 or 36" mill from the outset). Or, as I did, you can just go out and get some angle iron, a few nuts&bolts, a length of half-inch pipe, and a u-bolt and just convert a small log mill into a two-point mill yourself for about $10. I even attached it through the nose sprocket, allowing for a bit more width than the real Granberg set-up.
