I don't think it's an issue of your soil being too hard, I think it's more to do with adequate moisture and fertility. Big potatoes take a lot of nutrients and water.
The area where I plant potatoes is quite heavy. Heavy enough that I thought I was going to snap my pitchfork handle this fall while I was digging them. I plant potatoes 4-6" deep, hill them once, and that's it. I've always managed to grow nice big potatoes, but I always fertilize them well and irrigate them when needed. Except for this year, I was too busy and didn't notice how dry the garden was getting. Some people may remember my post from a few months ago of wilted strawberries. After that I kept on top of watering but I think that was enough to impact their size, as when I dug them a couple weeks ago they were smaller than normal.