If you still have the old installation it may be worthwhile to save the configuration in a different place and after that delete MP3tag.
It may also be required to adjust the path for the shell extension if there was any.
But after that, it should be possible to delete the folder.
See also the documentation: