[WS] iTunes source script with multiple search criteria

Development Web Sources Scripts
1

Greetings to the web sources community.

I'm copying my original post from [General Discussion] to [Development>Web Sources scripts] to faciitate the sharing of my script, and to make it easier to discuss improvements / modifications.

Please note this is NOT a finished product. It's still in development, due to a large oversight on my part.
Details are in this post.

After using pone's WS Itunes script(s) for years, I've finally decided to take up scipting and give a shot
at creating a tool with the functions I wanted. And the main one was create a script that could be
used internationally, i.e. I could somehow define what country the script would query.

This made sense to me, since if there was an artist from a certain country, or if an album had a localized
special edition, I would want to get metadata info from Itunes from that country. There are already WS scripts for several countries, but personally I find them cumbersome. I don't need scripts for 20+ countries if I mainly use just 2 or 3 and seldom use the rest. So that was my starting point.

Instructions:

"Search by:" / [clear]
The "Search by:" box has the following fields:

"Country:" Simply state which country you would like the script to query. Please use correct country codes
or a bad request error will be generated. List of available country codes is available at https://en.wikipedia.org/wiki/ISO_3166-1_alpha-2

"Language:" / [clear] / en_us / ja_jp
On the Itunes API page is mentioned the possibility of displaying results from a non-English alphabet (e.g japanese) in English, or displaying search results in japanese charaters from the english alphabet, if provided by the database. For always displaying results in English, the 'language' field must have the value 'en_us'; for always displaying results in japanese, the value must be 'ja_jp'. Leaving the field blank will display the results as is. Other/incorrect values will return a bad request error message

NB: Though the ja_jp option exists and is available, I have never had a query whose results were ported to japanese characters if they were not already the original writing system. Please take note of this if/when using the 'ja_jp' option.

"Query:" / [clear]
Standard query field; script will load artist and album info onto the filed, if said info is already present in the files. Query also loads the itunes' unique album identification code (if available), which dramatically increases search times and displays always correct results, if provided with correct info. Manual input is also available.

This script is intended to be used in tandem with localized versions; as a tool to perform one-off searches of seldom searched countries, instead of having 20+ WS scripts cluttering the Tag Sources menu.

So, please enjoy using this script, as much as I enjoyed making it (and leave a like if you think the script is worth it).

Thank you.

Current release:
- beta version 200191130_0

Known issues:
- [Query field] only supports UTF-8-compatible characters (A-Z, a-z, 0-9). Foreign writing systems (eg. japanese kanji/kana) or special characters (eg. Γ§, Γ©, ΓΆ) will return an invalid url error. Queries in English should be safe, as well as [CollectionId] number queries

iTunes_INTERNATIONAL_beta20191130_0.src (7.0 KB)

2

Here's some addicional info regarding my script:

- first, this is not a copy of pone's WS scripts. I have used them for many years, and when I decided to try to crete my own script, I used pone scripts to learn the proper commands, and how a search is conducted, as well as a template. But it is my own creation, and I've added some feature I could not find previously (like multiple search fields).

- This is a work in progress, in beta stage. Since I'm trying to create an international iTunes script, the inability to query in foreign writing systems, to call this a finished product would be incorrect.

For any fellow script enthusiast, here is my current problem, with an example:
- Conduct a search in Itunes store- japan, for an album, using the term "γ‚‰γβ˜†γ™γŸ", and list the first 20 results, in english.

The query link would be:
https://itunes.apple.com/search?entity=album&limit=20&lang=en_us&country=jp&term=γ‚‰γβ˜†γ™γŸ

Since the API can't recognize japanese characters, I have to use url-utf-8 encoding, which encodes
"term=γ‚‰γβ˜†γ™γŸ" into "term=%E3%82%89%E3%81%8D%E2%98%86%E3%81%99%E3%81%9F"

In WS script, this would be:
[IndexUrl]=https://itunes.apple.com/search?entity=album&limit=20&lang=en_us&country=jp&term=%s
[SearchBy]=γ‚‰γβ˜†γ™γŸ, or rather
[SearchBy]=%E3%82%89%E3%81%8D%E2%98%86%E3%81%99%E3%81%9F
[Encoding]=url-utf-8

Which works fine. However, in order to have the search parameters as variables, I need:
[IndexUrl]=https://itunes.apple.com/search?entity=album&limit=20%s
[SearchBy]=&lang=en_us&country=jp&term=γ‚‰γβ˜†γ™γŸ
[Encoding]=url-utf-8

which outputs
https://itunes.apple.com/search?entity=album&limit=20%26lang%3Den_us%26country%3Djp%26term%3Dγ‚‰γβ˜†γ™γŸ

And this doesn't work. Because not only "γ‚‰γβ˜†γ™γŸ" value gets converted into URL-UTF-8, but also the "&" and "=" symbols in "&lang=", &country=" and "&term=" which the API can't recognize

I can change the encoding to 'utf-8', but then I'd be back at square one, because "γ‚‰γβ˜†γ™γŸ" isn't recognized.

Any help would be greatly appreciated. Thank you.

3

Thanks for describing the encoding issue. I've fixed this with Mp3tag v2.99c.

Please also note that you now need to use || as separator at SearchBy as described here

4

Thank you so very much.
I'm very glad I'm able to help improve Mp3tag.

I do remember the need to use "| |" as separator; I released my script for Mp3tag v2.99a (since it's the current release and the one I believe most users -myself included- have installed) and was planning to update to the "| |" version upon release of version 2.99b from beta to main.

But for development purposes the latest version would be best. I'll modify my script for version 2.99c (if necessary -I've just noticed it had been released), and modify my post accordingly.

Thank you for your reply.

1 Like
5

Greetings to all.

Finally I was able to put some quality time into scripting, and release the latest version of iTunes WS script international.

Firstly, credits where credits are due. I'd like to thank Florian for taking the time to notice my work, and pointing me in the right direction.
Also, with the release of Mp3Tag 2.99c (beta), the biggest problem afflicting my script has been addressed:

2.99c (2019-12-11)

FIX: multi-field search for tag sources used encoding on whole parameter fragment when building URLs. (#47197 9)

This means that URL-UTF-8 encoding can now be used, which converts foreign characters, therefore making my script usable in any language supported by iTunes API.

List of changes:
> URL-UTF-8 implemented in multi-field search, which now properly converts only the necessary characters in the "Query" field (in version 2.99c)
> Double pipe "| |" used in [SearchBy] for multi-field search;
> [WordSeparator] value changed from "+" to "%20";
> Cleaning up of the code, making it easier to read and understand, and a cleaner debug file (for scripting purposes);
> Minor cosmetic changes to the script.

These changes mean that the current version of the script will only work in Mp3Tag version 2.99c (beta) and above. It will not work in the current (at the time of this post) stable version (2.99a).

On a personal note, it means a lot to me to be able to give back to a progrmam that I've been using for many years.

So, here is iTunes WS script international version. Enjoy.

iTunes_int_INTERNATIONAL_std_lang.src (6.8 KB)

Also, here are the localized versions for the U.S.A. (US), Japan (JP)- in original japanese writing kanji and kana, and a 2nd version for Japan, using converted romaji writing. Since these are the versions I most commonly use, I'm releasing them as well.

iTunes_loc_US_std_lang.src (6.8 KB)
iTunes_loc_JP_std_lang.src (6.8 KB)
iTunes_loc_JP_conv_lang.src (6.8 KB)

Enjoy.

1 Like
6

Thank you for the script. Just one request, will it be possible to add "composer" field to the tag?

Thanks.

7

Hello,

I've searched across the iTunes API and retested a few exemples of my library, but it doesn't look like iTunes has the "Composer" field available for processing. Can you provide an album page that has a Composer value (as in the url of the iTunes page, like _https://music.apple.com/jp/album/383274167?l=en)?

Or can you give some more details of what would you like to have my script do?

8

Hello rboss,

Thank you for the response. Here's an example of the tags from iTunes:

939Γ—745 170 KB

Here's an album page which has a composer value:

> can you give some more details of what would you like to have my script do?

Script works fine, it's just the composer tag is missing. Also some tag fields are uppercase, music purchased from iTunes never have tag fields in uppercase (see the screenshot)

I'm looking for a script that could imitate the tags as music purchased from iTunes. Or could get me as close as possible.

Thank you for your work rboss. Your help is highly appreciated.

9

Hello, ceo54

I searched the Itunes API directly for MONTANA, French Montana, and got the following raw data for track 2 (highlighted in your screen capture)

----
type: track
kind:song
artistId:475816358
collectionId:1489417548
trackId:1489417550
artistName:French Montana
collectionName:MONTANA
trackName:Suicide Doors (feat. Gunna)
collectionCensoredName:MONTANA
trackCensoredName:Suicide Doors (feat. Gunna)
artistViewUrl: link for the artist page on iTunes
collectionViewUrl: other link
trackViewUrl: yet another link
previewUrl: file.m4a
artworkUrl30:cover/30x30bb.jpg
artworkUrl60:cover/60x60bb.jpg
artworkUrl100:cover/100x100bb.jpg
collectionPrice:12.99
trackPrice:1.29
releaseDate:2019-09-13T07:00:00Z
collectionExplicitness:explicit
trackExplicitness:explicit
discCount:2
discNumber:1
trackCount:9
trackNumber:2
trackTimeMillis:213901
country:USA
currency:USD
primaryGenreName:Hip-Hop/Rap
contentAdvisoryRating:Explicit
isStreamable:true
-----

This is the ENTIRE information provided by iTunes for 'Suicide Doors' from the album MONTANA by French Montana.
My script simply parses this info into a format that Mp3Tag can work with.

As you can see, a 'Composer' entry is not present, despite it being on your purchased track.
With this I mean to say that iTunes does not have the best library for cataloguing music (even their own releases).
I'd like to give you an easy answer like "Sure! Here's a new version of the script with the Composer field. Have fun! :slight_smile: " But I can only work with what the API provides. And that's not much.

In short, I can't add a composer tag to my script because iTunes doesn't provide it. If this changes, I'll update the script accordingly.

But after reading what you're trying to do, allow me a suggestion:

Check out the same album on MusicBrainz here, and Composer data (and other tag info) is present (and is the same as in your screenshot).
For my music library, I use iTunes, Musicbrainz, Discogs, and VGMdb catalogs. If one doesn't have what I want, usually some other will. So you should consider using multiple libraries to get the most info for your music, and then trim down what you don't need/want.
You'll find scripts for theses libraries in the Mp3Tag community too; and you can create custom actions to automate some of the work in sorting all the data and filter what you need.

Oh and reguarding the tag fields in uppercase- if you can open your iTunes purchased music in Mp3Tag and go to View>>Extended Tags, you'll find all the tag fields in all caps. But if you check tag info in Windows (right-click>>properties) all the tag fields are in small caps.
I can't say with 100% certain, but I think tag fields are processed/displayed differently by different programs/OSes.
So again, this is not something I can control or modify. Apologies.

If this isn't the answer you're looking for, I hope that at least it can help guide you to it.

Good luck.

10

Thank you for the detailed response rboss.

I understand until the API supports composer and copyright tags, there isn't any to retrieve those. Is there any way I could contact Apple and ask them to include those tags in their API?

Thanks again. Highly appreciated.

11

Here is the link for the iTunes API link. There should be a contact link in it somewhere.

Good luck.

12

No contact information there unless I missed something. They should have at least included that.

Thank you rboss for everything. Your contribution and your help is highly appreciated.

1 Like
13

Hi This Script it's Almost Perfect,

  1. Not Sure Country and Language works, maybe I did not respect what is expected ?

Country : US|JP|DE Language : us|jp|de ..?

  1. Online Dataset and Local Files does not match, it takes a Huge time to Map them, Is there a way with json to automatically makes maching Online Parsed tracks with Local Filname ?
14

Hello gval,

1. Not Sure Country and Language works, maybe I did not respect what is expected ?

From this link for an album in iTunes:
_https://music.apple.com/us/album/hello-world-single/1469518210?l=en&lang=ja_jp

the country field should be filled with the 2-letter country id. In the link above the country is
_https://music.apple.com/US/album/hello-world-single/1469518210?l=en&lang=ja_jp

(US) which stands for United States.
Germany(DE),Brasil(BR), Great Britain(GB), etc. also work. When in doubt, go to the album page link of the entry you're searching for and parse the country id from the link. If it still doesn't work, please reply with the link you're after and I'll check too.

As for language, there are only two options (as per the iTunes API description page): english (en_us) and japanese (ja_jp).
But I will be honest with you: outside for having country:JP and language:en_us (output japanese album in english characters -romagi- , or country:JP and language:ja_jp (output japanese album in japanese characters -kana and kanji-, I have never seen the API providing info in different languages for any other country code; just Japan. The API simply doesn't provide the info.

2. Online Dataset and Local Files does not match, it takes a Huge time to Map them, Is there a way with json to automatically makes maching Online Parsed tracks with Local Filname ?

JSON not implemented :frowning: I've seen it and I'm interested in converting my script to use JSON, but that's a long time from now. Apologies.
I have that problem too; usually I just order them by track duration, and then do a manual check to make sure the parsed tracks and local files are properly mapped.

Hope this helps :slight_smile:

15

It's been a while since my last release, and I've been working on a new version of the script (ever since I heard about regular expressions). It's been ready for some time, and I haven't found any major bugs, so I'm releasing a new version of my WS multiple search criteria script.

What's new:

  • Full revision of the code, implementing regular expressions whenever possible;
    (this was something I've been wanting to do for a long time, and had to learn about regular expressions to do it, but I'm happy with the results)
  • Increased album search maximum limit from 100 to 200 entries;
  • Addition of "Country" field to album query response;
  • Addition of DebugWriteInput command for debugging purposes (disabled by default);

For now, here is the international version; I can make local country versions (as before) if requested.

iTunes_INTERNATIONAL_release_20220320_0.src (6.1 KB)

Thank you and enjoy.