When replacing an album cover with a new one, what happens to the old .jpg file?

Specifically, does MP3tag delete the old .jpg file?
It seems that the number of album art files is increasing while the number of songs (Titles) is staying he same.
I'm thinking the association between the Filename and the Album art file is changed but perhaps the old one is not deleted. Is this correct?

Thanks for thinking about this.

MP3tag has nothing to do with external files.
So, you would have to take care of them.
BTW: if you modify a single file how is it clear that the now embedded picture belongs to an album?

To ease the job: there is an action to export covers:

Yet, if the resulting file is different from the existing one but should get the same filename then an index is added to the name, e.g. folder(1).jpg - if yo specified folder as filename.

Please note that a fair number of players does not need external picture files to show a cover but that they really use the embedded picture.