Request for to_plain_latin() conversion function

General Discussion
1

I'm new here, so I would like to say hello first ;-).

I would like to make a request for a conversion function that converts all national characters to Latin counterparts (if possible), leaving Latin and unknown characters intact. The first version might handle just European characters (I believe I read somewhere that this could be done with some Win32 API, but I am not sure), in the next versions the list of recognised characters could become longer.

I understand that this could be achieved with series of replacements right now, but this is not very comfortable, especially if you would like to define transformation for all European languages.

Rationale:
I would like to use this function in the "Convert -> Tag - Filename" context.
Although we normally use national characters in tags, the file names are much easier to handle without them, because especially when transferred between various servers and operating systems, the national characters often get lost or converted to something unreadable. And the media players often have very basic, lightweight OSes.

Please consider adding this function. Should be easy to implement, and I find it extremely usable.

2

Did you have a look at the
$validate()
function? It helps to create legal filenames.

Also, there is the $replace() function that allows you to create longish lists of pairs of characters that should replace each other.
This function could help you get those translations. Yet, you have to do it according to your requirements.
Both functions can be used in conjunction with the Convert-function in the menu as well as the format value actions.

3

Thanks for your answer.
Yes, I know about the two functions and I am aware that this is a workaround, yet creating a replace expression that is good for - say - all European languages would be quite a nasty job.

My request is motivated by the fact that this new function should be quite easy to implement and maintain for the developers, and a very useful "out-of-the-box" tool for the users.

That's only my suggestion anyway, let the Mp3tag developer(s) decide.

4

Hi, I found a nice python library that does the job of "latinising" various alphabets. It's called Unidecode:

A C port is available (C#, too).

Can this be included in mp3tag in the future?

5

After a brief look at this library, I'm not sure if this is really useful when they say:

A user expects a character to be transliterated in their language but Unidecode uses a transliteration for a different language. It’s best to not use Unidecode for strings that are directly visible to users of your application.

As an example from their FAQ:

German umlauts are transliterated incorrectly
Latin letters “a”, “o” and “u” with diaeresis are transliterated by Unidecode as “a”, “o”, “u”, not according to German rules “ae”, “oe”, “ue”. This is intentional and will not be changed. Rationale is that these letters are used in languages other than German (for example, Finnish and Turkish). German text transliterated without the extra “e” is much more readable than other languages transliterated using German rules

I doubt that many German users would agree with the last sentence.

German words with umlauts like
StÀdte -> Staedte -> Stadte
Dörfer -> Doerfer -> Dorfer
KĂŒhe -> Kuehe -> Kuhe
reduced to the base character are indeed less readable - at least for German readers.

6

It is a distorting difference if
Er machte Löcher (he made holes)
becomes
Er machte Locher (he made perforators)

Or Nöte (woes) vs. Note (note)
Apfel (singular) vs. Äpfel (plural)

Don't know, though, how many tags feature these words.

7

Another example:

Kaffee rösten (roast coffee)
EisentrÀger rosten (rusting iron girders)

One example that could be very offending:

Die Luft ist schwĂŒl (The air is humid)
Dieser Mann ist schwul (This man is gay)

8

Thanks for your feedback. Having in mind all the drawbacks, I still think availability of 'unidecode' might be useful. Please note that it would be used to change file names, not the tags. Filenames in a plain ASCII are just safe across various operating and file systems, and decently readable in most of them. And a 'decent readability' is enough for a file name in my opinion.

As for me... I would use this function if it were available.

(In fact, I made a small C# program using the C# port of this library, and am using it regularly, but haing this possibility integrated in mp3tag would be nice)
https://bitbucket.org/DimaStefantsov/unidecodesharpfork/src/master/

9

I'm just curious:
What kind of "conversion" would this tool do with Unicode characters like in this album if the filename for the titles would be converted?

image
image570×514 23.4 KB

10

Until such an integration, you could use it as new entry in Tools (if your C# program is an executable command line tool).

11

It's a GUI program, but I can rewrite it, no problem.
The python unidecode tool is available as a command line, too.

Btw. you can install it with:
pip install unidecode
and check on the files that you mention:
ls | unidecode

12

Sorry, I don't use python or PIP on my windows system :wink:

You could manually rename one of your example songs to
01 - :ringer_planet:
or
09 - :earth_americas:
and check the result of the conversion with your tool.

13

If I try to copy this to notepad or notepad++, I get:

01 - : ringer_planet :
or
09 - : earth_americas :

(I manually added spaces before and after colons, otherwise the forum treats and displays these strings as icons)

It's the same when I try to paste the characters into the linux (Putty) console, for instance:
mkdir : ringer_planet :

I would need to know the unicode numbers and use them, I guess...

How does dir command list these files?

14

Maybe I just don't understand the purpose of the library?
Should it convert any NON-ASCII-character to an ASCII-Character?
Or is the library name "Unidecode" just misleading for me?

image

CMD-Window with Font "Consolas":
image

If I redirect the output of DIR in a test.txt file, the content of test.txt looks like
image

15

Luckily, I found this album in my collection. Unidecode will not help in this case:

PS E:\Doc\MMedia\Music\Coldplay\2021 - Music Of The Spheres> ls -n
01 ⊔.flac
02 Higher Power.flac
03 Humankind.flac
04 -✧.flac
05 Let Somebody Go.flac
06 :heart:.flac
07 People of The Pride(Explicit).flac
08 Biutyful.flac
09 ❍.flac
10 My Universe.flac
11 ∞.flac
12 Coloratura.flac

PS E:\Doc\MMedia\Music\Coldplay\2021 - Music Of The Spheres> ls -n | unidecode
01 ?.flac
02 Higher Power.flac
03 Humankind.flac
04 -?.flac
05 Let Somebody Go.flac
06 ?.flac
07 People of The Pride(Explicit).flac
08 Biutyful.flac
09 ?.flac
10 My Universe.flac
11 ?.flac
12 Coloratura.flac

(note that the forum displays icons on the output of plain ls -n, but in the console they are not visible - I see rectangles instead)

16

Thank you for your test.

Just to be sure:
Is this the expected result or should "unidecode" convert such Unicode characters too?

17

I'm not sure and can only guess that the original intention of the author was to 'latinise' symbols that are alphabetic. Not all unicode symbols. So this result, although not satisfying, might be expected.

Anyway, if such a function is available in mp3tag, the user has a preview option before renaming files, so (s)he can decide to use it or not on a particular set of files. I believe that in most cases it would still be useful.