The windows-explorer takes this media-specific folder-icon directly from the covers that are embedded in the files. So if there is more that 1 file it shows an icon- catalog. You can avoid that with a file called folder.jpg in your folders. If this file is present the explorer will show only 1 icon according to this file.
Windows itself with it's media-player created folder.jpgs too but they are hidden and system. So as long as you do not activate to also see hidden and system-files in the explorer-options you won't see them as files. So in the past you used folder.jpgs too. For me and others this functionality of WMP was always an annoyance and therefore I avoided to use the WMP:
If you have embedded covers in your media-files MP3Tag can help you to create these folder.jpg-files in one-go.
-
I understand that you have separate folders for each album and your files have identic covers embedded per folder. So load all your files in MP3Tag and mark them. (CTRL-a)
-
Define an action of the type "Export cover to file". You can do this with
Actions->Actions (Quick)or if you want to preserve your action for future use withActions->Actions
As filename you writefolder. Don't mark "export duplicate covers".
The feature "desktop-ini" ist not very practicable to use for your purpose and wasn't made for media-folders by Microsoft.