Break out your dremel cuz this is gonna be fun. My car did the exact same thing. Sounds kinda like rocks in a tin can eh? People on B15sentra forums told me to cut off the heat shielding from the exhasut.
After 2 agonizing hours with a dremel, working under my car in blistering heat, the SOB was finally off.
The piece I removed was a some sheet metal that surrounded the exhaust pipe right after the header. The piece is held in with one or 2 bolts (mine were rusted to hell). Once they were gone all was quiet again.