I've done plenty of rough physical jobs. Moving van swamper, infantryman, technician, assault pioneer, office furniture mover, etc, etc...
Each was very satisfying (until it wasn't) for the most part. But the best part is that I can now, as a flabby old geezer, out lift, out last, out anticipate and basically out perform most people half my age because as you do physical thing, you develop a sense for the spatiality of the objects, a feel for their momentum, a sixth sense for knowing the leverage points and a general know how about moving yourself with/around/along/over stuff. And you get a handle on the tools you can use to really leverage your power.
Some people can't. They simply can't. Spatial awareness escapes them in the way that pitch escapes someone who can't hold a tune.
But I think they're rare.
Start, and go with it. You'll discover your abilities - and they're not very contingent on brute strength.
Sure, lift weights. But work with wheels, pulleys, boxes, lumber, shovels, axes, ropes, fulcrums, levers, furniture, ladders, etc.
And enjoy!