Action to remove non-BMP characters

Hi there,

the media system in my car couldn't read any audio files from a USB stick. Turned out the reason was a few file names that contained Unicode characters outside the "Basic Multilingual Plane" (BMP), U+0000 till U+FFFF. That is, outlandish stuff like emojis: :eye::joy:...

It took me a while to figure out how to replace such characters using mp3tag, and I want to share the result.

To get rid of non-BMP characters in tags under *Windows*, I used the following "Replace with regular expression" action:

Field: _ALL

Regular expression: [\x{D800}-\x{DBFF}][\x{DC00}-\x{DFFF}]

Replaces matches with: _

Notice that the pattern doesn't describe non-BMP characters by a range of Unicode code points but rather it decribes what the two subsequent UTF-16 units - the "surrogate pair" - used for such characters look like. I suppose the reason why this (and only this) works is that the regex parser reads strings of wide characters (wchar_t), and on Windows wchar_t happens to be a 16 bit word interpreted as a UTF-16 unit.

I suppose that the expression above does *not* work on Mac. AFAIK on Mac, the internal encoding would be UTF-32 instead of UTF-16. Which implies that non-BMP characters are actually described by a range of Unicode code points, which makes the pattern simpler:

[\x{10000}-\x{10FFFF}]

But that's just my guess, I have no Mac.

Anyway, once I had all nasty characters replaced in the tags, I generated the file names from the fixed titles, and now my car's media system can play everything again.

Cheers,

Burkhard

1 Like