Request: Multiple search criteria in Web Sources Framework

rboss · March 2, 2019, 3:59pm

Hello, and thank you Florian for this wonderfull program.

I am by no means a professinal programmer or coder, so please correct me if I'm mistaken/ wrong.

I've been playing with WS scripts (mostly Itunes WS scripts) and adapting them to my own tagging needs.
Lately I've been trying to find a way to edit the [IndexUrl] field in order to add more search queries but I'm coming up short.

On the Web Sources Framework help page all I find is an entry of "%s", which will be replaced by the "search criteria entered by user" (quote).

Is %s the only variable available? Is it possible to add more? Is it possible to create custom variables?

Here's an eample of what I'm trying to do:

(From my customized WS itunes script, originally from pone):

From my understanding, the %s variable will be filled with info from the [SearchBy] field, using [WordSeperator] to fill in blanks.

But I would also like to have another variable(s) (custom ou otherwise) for the 'country' info in [IndexUrl] and [AlbumUrl]. So I would get something like:

[Name]=iTunes
[BasedOn]=http://itunes.apple.com
[IndexUrl]=http://itunes.apple.com/search?country=%X&entity=album&term=%S
[AlbumUrl]=http://itunes.apple.com/lookup?country=%X&entity=song&limit=200&id=
[WordSeperator]=+
[IndexFormat]=%_preview%|%_url%|%Artist%|%Album%|%Explicit%|%Tracks%|%Year%|%Genre%|
[Variable_1,%X]= "country ID" (ou whatever)
[Variable_2,%S]=$if2(%albumartist%,%artist%)[ %album%] (already present)
[Encoding]=url-utf-8

Is this already acheivable? If so, how?

If not, could you consider this suggestion as a feature request?.

Thank you,
rboss

"May the Force live long and prosper with you" - Harry Potter

rboss · March 17, 2019, 7:59pm

Thats exactly what I want to do, but the issue is that I need 2 (or more) targets, as I tried to explain using the example in my previous post.

Any ideas?

rboss · September 28, 2019, 4:21pm

Well, apparently Florian either read my post, or read my mind.

As of 2019-09-20, Mp3Tag v2.98 came out, with some particular changes that interested me very much:

[2019-09-09] NEW: added support for including files from tag sources to reuse common parts.
[2019-09-08] NEW: added support for multi-field search for tag sources.
[2019-09-08] NEW: added support for more descriptive SearchBy parameters for tag sources.

Although not exactly what I requested, I have been working with this new feature, and have had some success in creating a WS script for Itunes DB with support for searching different countries.

Currently, my script does work, but it is very resource heavy; so I do not feel confortable sharing an unfinished product. Hopefully, in the coming weeks I will complete this project and share it with everyone.

rboss · October 13, 2019, 9:46pm

Hello, everyone.

A quick update:

Had 2 full rewrites of my script; but I was successful in reducing its overall resource load. It should now run as smoothly as other WS scripts;

My initial plan was to load Itunes' country list into the script, and then have the script compare each country code until a match was found. This caused my script to run very slowly and create a very heavy output (a debug file for the parser index script would normally be ~150mb.

Thankfully that is no longer the case.

I am now in the debug stage; after that I'll just need to do some cleaning of the code and it should be ready for release.

One thing I did discover during debug was that the information provided by the itunes website appears to be incomplete.

On the available countries and regions page there should be a list of all countries that support API search. However, there are some countries that the API can perform a query to, that are not listed in the page:

For example:

to query the Itunes API, for the term "a", and using United States as the search country/region:
"https://itunes.apple.com/search?country=US&entity=album&term=a"

Which correctly generates a query file populated with the search results.

The same search, using Antarctica (AQ) as a search country:
"https://itunes.apple.com/search?country=AQ&entity=album&term=a"
This query will connect to an error page indicating an error in the country field, which makes sense, since Antarctica isn't listed in the Itunes available countries and regions page.

But if I try the search with the Spanish region of Andorra (AD), for example:
"https://itunes.apple.com/search?country=AD&entity=album&term=a"

The query file IS generated (with 0 results, but the file is generated) which means the API search sucessfully connected with the database and returned a results file.

But the thing is, Andorra is also not listed in the Itunes available countries and regions page, just like Antactica.

So, I my conclusion is that the countries and regions page is either incomplete or outdated, and that I cannot garantee that all the countries in the list will provide query results.

That is to say, my script will be presented as is, and since the search country will be a manually input value, it may be prone to error.

As a side note, the country codes for United States, Antactica and Andorra were obtained from this site which has the official list of country abbreviation codes (also used by itunes API, as mentioned in their API search page.

I was planning to test my script against all countries in the countries' list, but because of this discovery, that test makes little sense now. I'll probably try a sample of a dozen or so countries and, given no errors are found, consider that stage of debugging done.

This will save some debugging time, so maybe in a couple of weeks I may release my script.

I'm enjoying this project very much, and hope to produce a good, stable and reliable script.

rboss · November 10, 2019, 7:45pm

After using pone's WS Itunes script(s) for years, I've finally decided to take up scipting and give a shot
at creating a tool with the functions I wanted. And the main one was create a script that could be
used internationally, i.e. I could somehow define what country the script would query.

This made sense to me, since if there was an artist from a certain country, or if an album had a localized
special edition, I would want to get metadata info from Itunes from that country. There are already WS scripts for several countries, but personally I find them cumbersome. I don't need scripts for 20+ countries if I mainly use just 2 or 3 and seldom use the rest. So that was my starting ponit. After several trials and errors (and learning experience), and also several rewrites, I finally have a product I feel confident in releasing publicly.

Instructions

"Search by:" / [clear]
The "Search by:" box has the following fields:
"Country:" Simply state which country you would like the script to query. Please use correct country codes
or a bad request error will be generated. List of available country codes is avvailable at https://en.wikipedia.org/wiki/ISO_3166-1_alpha-2.

"Language:" / [clear] / en_us / ja_jp
On the Itunes API page is mentioned the possibility of displaying results from a non-English alphabet (e.g Japan) in English, or displaying search results in japanese charaters from the english alphabet, if provided by the database. For always displaying results in English, the 'language' field must have the value 'en_us'; for always displaying results in japanese, the value must be 'ja_jp'. Leaving the field blank will display the results as is. Other/incorrect values will return a bad request error message

NB: Though the ja_jp option exists and is available, I have never had a query whose results were ported to japanese characters if they were not already the original writing system. Please take note of this if/when using the 'ja_jp' option.

"Query:" / [clear]
Standard query field; script will load artist and album info onto the filed, if said info is already present in the files. Query also loads the itunes' unique album identification code (if available), which dramatically increases search times and displays always correct results, if provided with correct info. Manual input is also available.

This script is intended to be used in tandem with localized versions; as a tool to perform one-off searches of seldom searched countries, instead of having 20+ WS scripts cluttering the Tag Sources menu.

So, please enjoy using this script, as much as I enjoyed making it (and leave a like if you think the script is worth it).

Thank you.

iTunes_INTERNATIONAL.src (6.7 KB)

Florian · November 15, 2019, 11:16am

Nice work!

I've just noticed that you're already using the new dialog for multiple search criteria. With v2.99b I've changed the delimiter at SearchBy to be two pipe symbols || (see this topic for discussion).

This means, that the SearchBy field for your tag source would now be

[SearchBy]=Country||||%s||Language||||&lang=%s||Query||[$if2([$meta(CollectionID)],$if2(%albumartist%,%artist%)[ %album%])]||&term=%s

I also invite you to create a dedicated topic for this web source at #development:web-sources-scripts so that others can find it more easily.

rboss · November 24, 2019, 2:01am

Thank you so much! For your message and for your work!

I'll update the script with your input, and open a new topic to share it with everyone, as soon as I'm able to.

Keep on making mp3tag the best tagger ever!