Go to Settings-Manage My Voices, and swipe left on the voice you don’t want. The Delete button will appear.
Left swipe is a standard iOS gesture for deleting an row in a table.