Of course you do because that is what is what responsible folks do. Use common sense. That is why if applicable you have buffer strips along water and sensitive areas. Of course that is not always possible. Everyone's situation is different. I share 3 fence lines with a real hard core environmentalist. Her and her husband have a beautiful place that they have put a lot of hard work, sweat, and years into. I want the fences clean but I sure do not want to harm any of their precious plants. I say that without sarcasm as I have no interest in harming anything that does not need killed. There is one fence in particular that I sprayed gently this spring. It needs touched up but I am very skittish. They have a beautiful grass buffer along a portion of it that the mow with a lawnmower. Yes I could safely hit it with 2-4-D but the fence is full of grass so that really would not fix the issue. Now once I turn the corner from the property line and head west it is game on as everything needs to die and there are no neighbors or houses for 1/2 mile or more
As for killing healthy mature trees with a light dose of Glyphosate or 2-4-D I say hogwash. I have never seen it. I have trees around field edges that I want to save but when hand spraying the edges I still hit the low hanging leaves and limbs. It curls the leaves back and some fall off but the tree does not die, The low hangers are in the way anyway. I have a medium sized Cherry that the wind took down laying in a bean field now. It is mostly snapped off at the base but still has some connectivity. It is fully alive and I have sprayed part of it. The other part is still thriving. Some folks make mountains out of mole hills