Rosa rugosa has been used to make impenetrable, thorny hedges, and should be hardy in New England. At least some varieties are shade tolerant, but the advice about giving roses 6-8 hours of sunlight daily is to promote more blooms... rose plants will be healthy with less sun, just not bloom as much. If you've got deep shade, this probably won't work.
http://homeguides.sfgate.com/s...
Rugosa roses are invasive in some parts of the Northeast, so you might want to research the likelihood of that happening in your area. It would not be cool to introduce them to the woods, and the hips are well-loved by various birds and critters.
I like the ideas of Aralia spinosa (Devil's Walking Stick) and Smilax, but I worry that these will easily end up in other parts of your yard where you don't want them.
A 'Beware of Dog' sign might help even if you have no dog, as Mindi suggests.
Good luck,
Virginia