Sometimes instead of a dowel i'll use a chopsrick or a wooden kabab skewer with an end cut flat.

I got a brass hammer and punch set. The largest punch and light tap with the hammer work great with out damaging.