I doubt that the Windows Explorer is the right tool to edit the tags ...
Anyway: please check ((in MP3tag with Alt-T) what kind of tag-version you have now in the file(s) - the window title of that dialog shows the tag versions of all included tags.
I suspect that WE reduced the tags to Version 1 which does not support albumart.
Also check (in MP3tag with Alt-T) that you only have mp3 tags but not accidentally APE tags.
Then: if you have used WMP as player you probably also have the file folder.jpg in each album folder.
You can easily import these into the mp3 files with just one action (you have to create a new one, though).
In general: any deleted data is lost, usually, unless you have backups.
Yes, you are probably right that WE is probably not the appropriate tool to edit tags.
Within MP3tag with Alt-T, the MP3 files WITHOUT album art are typically ID3v2.4 (ID3v1 ID3v2.4) this information is visible at the top of the dialog window.
Within MP3tag with Alt-T - on FILETYPE - some corrupted files have AUDIO/MP3 - however, for some corrupted files FILETYPE is not visible at all! So I cannot check for APE tag.
Virtually all these are tracks are purchased from BEATPORT and (where I have followed the actions described initially i.e REMOVE TAG & UNDO) there is NO *.jpg files within WMP for the album art. I am unclear as to how these MP3 files have the album art "attributed/linked/associated" to each MP3 ?
From tracks purchased from DEFECTED (for example) there are file folders with *.jpg. They are okay.
Download the beatport zip file, extract it. You get a Cover &Preview#Single _ beatport.src file.
Copy it to this location.
Now check the Tag sources menu in Mp3tag for the new entry.
You can additionally download and start the Mp3tag Cover Preview tool. It will show you previews of the covers (unfortunately the beatport ones are very small)
You can also try the amazon or google scripts.