About 90% of tree roots are in the top 18" of soil.
Tree roots will go as deep as they can go and still find oxygen in the soil (coming from the air above).
So if you put 4-5" of soil over the roots, you are probably killing the bottom 4-5" of the roots...IF they were evenly distributed in the soil profile (they are not), you'd be killing about 20% of the roots.
Assuming the soil that you are putting on top will be worked over pretty well with equipment to make it smooth, you are compacting the new soil more than the existing soil. That means air and water will have a more difficult time getting through.
Also, the soil you are digging out probably isn't really top soil - is it? Sure, the top few inches is, but below that is no longer top soil ... just dirt with a lot less micro-biotic activity.
Those last 2 lines mean you are probably killing well more than 20% of the trees' roots.
Will the trees recover? Maybe. Maybe not. How healthy are they to start with. Will you see a drought in the next couple of years - because that can be another major factor leading to a more rapid decline? How about a period of extended saturation? - that is as bad, or worse, than drought. These are things a healthy tree may recover from. A tree that just lost 40% of its roots probably won't.
If there are a few roots sticking above ground and you just level the ground around those - not covering wide areas, that is different than 4" of soil over the entire root system.