I created a small tool to permanently delete TFS Work Items from Team Foundation Server Database. I found some old tools, but they didn't work with TFS 2013 and the source code was not available. So, I decided to create a new one and share the source code.
Download the tool from Visual Studio Gallery
Visit the source code on CodePlex
This tool physically deletes work items from a Team Foundation Server. Use it at your own risk