I like the sound of that,I can't picture how you mean though(its me,I know )Is the punch thicker than the drilled hole?I can't picture when you say use as a wedge... Scuse my dumb moment!
Grind it with a die grinder or dremel not 100 percent through just leave a little room for error so you dont hit the crank then knock it loose with a chisel.