[WS] iTunes source script with multiple search criteria

[rboss]

Greetings to the web sources community.

I'm copying my original post from [General Discussion] to [Development>Web Sources scripts] to facilitate the sharing of my script, and to make it easier to discuss improvements / modifications.

---

Please note this is NOT a finished product. It's still in development, due to a large oversight on my part.
Details are in this post.

---

After using pone's WS iTunes script(s) for years, I've finally decided to take up scripting and give a shot
at creating a tool with the functions I wanted. And the main one was create a script that could be
used internationally, i.e. I could somehow define what country the script would query.

This made sense to me, since if there was an artist from a certain country, or if an album had a localized
special edition, I would want to get metadata info from iTunes from that country. There are already WS scripts for several countries, but personally I find them cumbersome. I don't need scripts for 20+ countries if I mainly use just 2 or 3 and seldom use the rest. So that was my starting point.

---

Instalation:

So just download the script(s) of your choice to your Mp3Tag '\sources' folder and on Mp3Tag go to Tag Sources > iTunes WS and select the script you want to use.

Instructions:

The "Search by:" box has the following fields:

"Country:" Simply state which country you would like the script to query. Please use correct country codes or a bad request error will be generated. List of available country codes is available at ISO 3166-1 alpha-2 - Wikipedia. Check beforehand if iTunes is available in your desired country.

"Language:" / [clear] / en_us / ja_jp
On the Itunes API page is mentioned the possibility of displaying results from a non-English alphabet (e.g japanese) in English, or displaying search results in japanese charaters from the english alphabet, if provided by the database. For always displaying results in English, the 'language' field must have the value 'en_us'; for always displaying results in japanese, the value must be 'ja_jp'. Leaving the field blank will display the results as is. Other/incorrect values will return a bad request error message.

NB: Though the ja_jp option exists and is available, I have never had a query whose results were ported to japanese characters if they were not already the original writing system. Please take note of this if/when using the 'ja_jp' option.

"Query:" / [clear]
Standard query field; script will load artist and album info onto the filed, if said info is already present in the files. Manual input is also available. For direct results, you can search for a collection ID number (on the link https://music.apple.com/us/album/[album name]/123456789 the number 123456789 would be the collection ID to put in the query field).

This script is intended to be used in tandem with localized versions; as a tool to perform one-off searches of seldom searched countries, instead of having 20+ WS scripts cluttering the Tag Sources menu.

So, please enjoy using this script, as much as I enjoyed making it (and leave a like if you think the script is worth it).

Thank you.

---

---

Current release(s):
- version 3.00 (check thread for history of changes)

Known issues:
- JPN - English returns latinized japanese, not translated.
iTunes is responsible for providing alternative language versions of their data. Their API page states 'English' language results. If you are unhappy with your results, contact Apple.

Files:
iTunes WS#[Any] - [Localized].src (6.7 KB) >> Blank 'Country' and 'Language' input fields
iTunes WS#JPN - English.src (6.7 KB) >> 'Country' preset to Japan (JP) and 'Language' preset to English (en_us)
iTunes WS#JPN - Japanese.src (6.7 KB) >> 'Country' preset to Japan (JP) and 'Language' preset to empty (original language)
iTunes WS#USA - English.src (6.7 KB) >> 'Country' preset to United States (US) and 'Language' preset to English (en_us)

[rboss]

Here's some addicional info regarding my script:

- first, this is not a copy of pone's WS scripts. I have used them for many years, and when I decided to try to crete my own script, I used pone scripts to learn the proper commands, and how a search is conducted, as well as a template. But it is my own creation, and I've added some feature I could not find previously (like multiple search fields).

- This is a work in progress, in beta stage. Since I'm trying to create an international iTunes script, the inability to query in foreign writing systems, to call this a finished product would be incorrect.

For any fellow script enthusiast, here is my current problem, with an example:
- Conduct a search in Itunes store- japan, for an album, using the term "らき☆すた", and list the first 20 results, in english.

The query link would be:
https://itunes.apple.com/search?entity=album&limit=20&lang=en_us&country=jp&term=らき☆すた

Since the API can't recognize japanese characters, I have to use url-utf-8 encoding, which encodes
"term=らき☆すた" into "term=%E3%82%89%E3%81%8D%E2%98%86%E3%81%99%E3%81%9F"

In WS script, this would be:
[IndexUrl]=https://itunes.apple.com/search?entity=album&limit=20&lang=en_us&country=jp&term=%s
[SearchBy]=らき☆すた, or rather
[SearchBy]=%E3%82%89%E3%81%8D%E2%98%86%E3%81%99%E3%81%9F
[Encoding]=url-utf-8

Which works fine. However, in order to have the search parameters as variables, I need:
[IndexUrl]=https://itunes.apple.com/search?entity=album&limit=20%s
[SearchBy]=&lang=en_us&country=jp&term=らき☆すた
[Encoding]=url-utf-8

which outputs
https://itunes.apple.com/search?entity=album&limit=20%26lang%3Den_us%26country%3Djp%26term%3Dらき☆すた

And this doesn't work. Because not only "らき☆すた" value gets converted into URL-UTF-8, but also the "&" and "=" symbols in "&lang=", &country=" and "&term=" which the API can't recognize

I can change the encoding to 'utf-8', but then I'd be back at square one, because "らき☆すた" isn't recognized.

Any help would be greatly appreciated. Thank you.

[Florian]

Thanks for describing the encoding issue. I've fixed this with Mp3tag v2.99c.

Please also note that you now need to use || as separator at SearchBy as described here

[rboss]

Thank you so very much.
I'm very glad I'm able to help improve Mp3tag.

I do remember the need to use "| |" as separator; I released my script for Mp3tag v2.99a (since it's the current release and the one I believe most users -myself included- have installed) and was planning to update to the "| |" version upon release of version 2.99b from beta to main.

But for development purposes the latest version would be best. I'll modify my script for version 2.99c (if necessary -I've just noticed it had been released), and modify my post accordingly.

Thank you for your reply.

[rboss]

Greetings to all.

Finally I was able to put some quality time into scripting, and release the latest version of iTunes WS script international.

Firstly, credits where credits are due. I'd like to thank Florian for taking the time to notice my work, and pointing me in the right direction.
Also, with the release of Mp3Tag 2.99c (beta), the biggest problem afflicting my script has been addressed:

2.99c (2019-12-11)

FIX: multi-field search for tag sources used encoding on whole parameter fragment when building URLs. (#47197 9)

This means that URL-UTF-8 encoding can now be used, which converts foreign characters, therefore making my script usable in any language supported by iTunes API.

List of changes:
> URL-UTF-8 implemented in multi-field search, which now properly converts only the necessary characters in the "Query" field (in version 2.99c)
> Double pipe "| |" used in [SearchBy] for multi-field search;
> [WordSeparator] value changed from "+" to "%20";
> Cleaning up of the code, making it easier to read and understand, and a cleaner debug file (for scripting purposes);
> Minor cosmetic changes to the script.

These changes mean that the current version of the script will only work in Mp3Tag version 2.99c (beta) and above. It will not work in the current (at the time of this post) stable version (2.99a).

On a personal note, it means a lot to me to be able to give back to a progrmam that I've been using for many years.

So, here is iTunes WS script international version. Enjoy.

iTunes_int_INTERNATIONAL_std_lang.src (6.8 KB)

Also, here are the localized versions for the U.S.A. (US), Japan (JP)- in original japanese writing kanji and kana, and a 2nd version for Japan, using converted romaji writing. Since these are the versions I most commonly use, I'm releasing them as well.

iTunes_loc_US_std_lang.src (6.8 KB)
iTunes_loc_JP_std_lang.src (6.8 KB)
iTunes_loc_JP_conv_lang.src (6.8 KB)

Enjoy.

[ceo54]

Thank you for the script. Just one request, will it be possible to add "composer" field to the tag?

Thanks.

[rboss]

Hello,

I've searched across the iTunes API and retested a few exemples of my library, but it doesn't look like iTunes has the "Composer" field available for processing. Can you provide an album page that has a Composer value (as in the url of the iTunes page, like _https://music.apple.com/jp/album/383274167?l=en)?

Or can you give some more details of what would you like to have my script do?

[ceo54]

Hello rboss,

Thank you for the response. Here's an example of the tags from iTunes:

Here's an album page which has a composer value:

MONTANA by French Montana

> can you give some more details of what would you like to have my script do?

Script works fine, it's just the composer tag is missing. Also some tag fields are uppercase, music purchased from iTunes never have tag fields in uppercase (see the screenshot)

I'm looking for a script that could imitate the tags as music purchased from iTunes. Or could get me as close as possible.

Thank you for your work rboss. Your help is highly appreciated.

[rboss]

Hello, ceo54

I searched the Itunes API directly for MONTANA, French Montana, and got the following raw data for track 2 (highlighted in your screen capture)

----
type: track
kind:song
artistId:475816358
collectionId:1489417548
trackId:1489417550
artistName:French Montana
collectionName:MONTANA
trackName:Suicide Doors (feat. Gunna)
collectionCensoredName:MONTANA
trackCensoredName:Suicide Doors (feat. Gunna)
artistViewUrl: link for the artist page on iTunes
collectionViewUrl: other link
trackViewUrl: yet another link
previewUrl: file.m4a
artworkUrl30:cover/30x30bb.jpg
artworkUrl60:cover/60x60bb.jpg
artworkUrl100:cover/100x100bb.jpg
collectionPrice:12.99
trackPrice:1.29
releaseDate:2019-09-13T07:00:00Z
collectionExplicitness:explicit
trackExplicitness:explicit
discCount:2
discNumber:1
trackCount:9
trackNumber:2
trackTimeMillis:213901
country:USA
currency:USD
primaryGenreName:Hip-Hop/Rap
contentAdvisoryRating:Explicit
isStreamable:true
-----

This is the ENTIRE information provided by iTunes for 'Suicide Doors' from the album MONTANA by French Montana.
My script simply parses this info into a format that Mp3Tag can work with.

As you can see, a 'Composer' entry is not present, despite it being on your purchased track.
With this I mean to say that iTunes does not have the best library for cataloguing music (even their own releases).
I'd like to give you an easy answer like "Sure! Here's a new version of the script with the Composer field. Have fun! " But I can only work with what the API provides. And that's not much.

In short, I can't add a composer tag to my script because iTunes doesn't provide it. If this changes, I'll update the script accordingly.

But after reading what you're trying to do, allow me a suggestion:

Check out the same album on MusicBrainz here, and Composer data (and other tag info) is present (and is the same as in your screenshot).
For my music library, I use iTunes, Musicbrainz, Discogs, and VGMdb catalogs. If one doesn't have what I want, usually some other will. So you should consider using multiple libraries to get the most info for your music, and then trim down what you don't need/want.
You'll find scripts for theses libraries in the Mp3Tag community too; and you can create custom actions to automate some of the work in sorting all the data and filter what you need.

Oh and reguarding the tag fields in uppercase- if you can open your iTunes purchased music in Mp3Tag and go to View>>Extended Tags, you'll find all the tag fields in all caps. But if you check tag info in Windows (right-click>>properties) all the tag fields are in small caps.
I can't say with 100% certain, but I think tag fields are processed/displayed differently by different programs/OSes.
So again, this is not something I can control or modify. Apologies.

If this isn't the answer you're looking for, I hope that at least it can help guide you to it.

Good luck.

[ceo54]

Thank you for the detailed response rboss.

I understand until the API supports composer and copyright tags, there isn't any to retrieve those. Is there any way I could contact Apple and ask them to include those tags in their API?

Thanks again. Highly appreciated.

[rboss]

Here is the link for the iTunes API link. There should be a contact link in it somewhere.

Good luck.

[ceo54]

No contact information there unless I missed something. They should have at least included that.

Thank you rboss for everything. Your contribution and your help is highly appreciated.

[gval]

Hi This Script it's Almost Perfect,

1. Not Sure Country and Language works, maybe I did not respect what is expected ?

Country : US|JP|DE Language : us|jp|de ..?

2. Online Dataset and Local Files does not match, it takes a Huge time to Map them, Is there a way with json to automatically makes maching Online Parsed tracks with Local Filname ?

[rboss]

Hello gval,

1. Not Sure Country and Language works, maybe I did not respect what is expected ?

From this link for an album in iTunes:
_https://music.apple.com/us/album/hello-world-single/1469518210?l=en&lang=ja_jp

the country field should be filled with the 2-letter country id. In the link above the country is
_https://music.apple.com/US/album/hello-world-single/1469518210?l=en&lang=ja_jp

(US) which stands for United States.
Germany(DE),Brasil(BR), Great Britain(GB), etc. also work. When in doubt, go to the album page link of the entry you're searching for and parse the country id from the link. If it still doesn't work, please reply with the link you're after and I'll check too.

As for language, there are only two options (as per the iTunes API description page): english (en_us) and japanese (ja_jp).
But I will be honest with you: outside for having country:JP and language:en_us (output japanese album in english characters -romagi- , or country:JP and language:ja_jp (output japanese album in japanese characters -kana and kanji-, I have never seen the API providing info in different languages for any other country code; just Japan. The API simply doesn't provide the info.

2. Online Dataset and Local Files does not match, it takes a Huge time to Map them, Is there a way with json to automatically makes maching Online Parsed tracks with Local Filname ?

JSON not implemented I've seen it and I'm interested in converting my script to use JSON, but that's a long time from now. Apologies.
I have that problem too; usually I just order them by track duration, and then do a manual check to make sure the parsed tracks and local files are properly mapped.

Hope this helps

[rboss]

It's been a while since my last release, and I've been working on a new version of the script (ever since I heard about regular expressions). It's been ready for some time, and I haven't found any major bugs, so I'm releasing a new version of my WS multiple search criteria script.

What's new:

• Full revision of the code, implementing regular expressions whenever possible;
  (this was something I've been wanting to do for a long time, and had to learn about regular expressions to do it, but I'm happy with the results)
• Increased album search maximum limit from 100 to 200 entries;
• Addition of "Country" field to album query response;
• Addition of DebugWriteInput command for debugging purposes (disabled by default);

For now, here is the international version; I can make local country versions (as before) if requested.

iTunes_INTERNATIONAL_release_20220320_0.src (6.1 KB)

Thank you and enjoy.

[rboss]

Hello everyone,

Here is a new version of the iTunes script, with many improvements. This is not a revision of the previous script, but a new one built almost from scratch. Major changes include:

• Full rewrite - now using JSON
  Because the iTunes API returns queries using JSON-formatted data, this script was built with JSON at its core. This means less work to get the tag information from the query results, which means faster results.
  I compared a debug output file from this version to the previous, and it was ~90% smaller.

• Streamlined code
  I also reviewed and optimized the instructions used, and removed unnecessary and unwanted fields. Which leads to....

• Explicit results
  The previous versions collected censored fields for Album title and Track title. I made the decision to change this and get the 'explicit' versions of those fields. This doesn't mean that I cut corners or was lazy and didn't want the trouble - in fact the script contains instructions for both censored and explicit tags, but only the explicit tags are active. And I included the iTunes Explicit|Clean classification on the search results, and the content advisory rating tag for tracks, so you are informed on two levels of an explicit track/album. Of course, if you don't want those tags, just uncheck them.

TL/DR: If you have expletives in your music, now you (can) have expletives in your tags, with fair warning about them.

• Localized results
  The reason that I started this project. If there is an Special Edition album which only came out in one / a few countries, or if I want Japanese or Spanish tags, I don't need a script for every iTunes store there is. Consider this as a one-size-fits-all tagger for iTunes.

So just download the script(s) of your choice to your Mp3Tag '\sources' folder and on Mp3Tag go to Tag Sources > iTunes WS and select the script you want to use.

I am releasing 4 versions of the script. They are identical to each other, but have the "Country" and "Language" fields pre-filled according to their intended use (but you can change them if you want):

• 'USA - English' returns tags from the American localized iTunes site, in English
• 'JPN - Japanese' returns tags from the Japanese localized iTunes site, in Japanese
• 'JPN - English' - same as the previous one, but returns results in English alphabet (if supported)
• '[Any] - [Localized]' - 'Country' and 'Language' left empty. Run an "empty" search and the iTunes API will default to USA / English

There is also some deactivated code (besides the explicit fields) left in the script(s). This is intentional - some tags are filled according to my preference, but this might not be the case for others, so I left other options coded. For example I prefer the 'Date' tag in 'YYYY-MM-DDTHH:MM:SSZ' format. If you know what you are doing, you can modify the script to return 'YYYY-MM-DD' or just 'YYYY'.

Enjoy your tags!

iTunes WS#[Any] - [Localized].src (6.7 KB)
iTunes WS#JPN - English.src (6.7 KB)
iTunes WS#JPN - Japanese.src (6.7 KB)
iTunes WS#USA - English.src (6.7 KB)

[rboss]

Hello again,

I wasn't expecting to be needing to release another version so soon, but I just noticed that, with version 3.22 and its major revision of JSON tools, the international iTunes script works abnormally i.e. it will still give results, but incorrect ones.

Here is a quick patch that fixes most of the problems. For normal use it will work fine, but if you edit the script to enable the censored field "TRACKS_CENSORED", then in the remote event that the iTunes' API returns results with missing values, the script (as of the current version) will not be able to identify and work around them as it did before.

For fellow WS developers/enthusiasts, the 'problem' lied in the 'json_select_many' function, which apparently now only returns elements denoted by the second parameter if they exist, whereas in version 3.21 it would return an empty value if the element didn't exist.

Since this change happened in Mp3tag 3.22, I took the opportunity to implement the new [MinAppVersionWin], that all but guarantees that this fix will only work with version 3.22 (and above).

I'll try to release a final version as soon as I'm able to do so.

---

iTunes WS#301[Any] - [Localized].src (6.7 KB)
iTunes WS#301_JPN - English.src (6.7 KB)
iTunes WS#301_JPN - Japanese.src (6.7 KB)
iTunes WS#301_USA - English.src (6.7 KB)