Web sources framework

Development Web Sources Scripts
#1

Hello!

The latest Development Build of Mp3tag features a web sources framework which allows tagging via almost any website which offers album information. Please note, that this part is still in beta stage.

To import tags from a website, a web source description file is needed, which contains general information about the source as well as parser scripts which are responsible for parsing the right data out of the WWW sites.

I've written the web source description file Discogs.src as an example how to use this framework and hopefully some of you will pick up the idea and write a source description file for e.g. your favorite online music catalog or a Lyrics site.

Please have a look at this brand-new help topic for documentation of the web source description file format and please don't hesitate to ask, if you have any questions!

Hope you'll like it and use it :slight_smile:

Best regards,
– Florian

4 Likes
Web Sources Archive
Frage zu Discogs.com
Possible Alternative
Selecting which tags to import from Discogs
#2

The new feature is available via the toolbar drop-down menu on the freedb toolbar icon. Please have a look at the screenshot :slight_smile:

Best regards,
~ Florian


Web sources
#3

Please let me know, if you've developed a custom web source description file :slight_smile:

Best regards,
~ Florian

#4

I've just created the Web Sources Archive with a brand-new AMG web source (which was contributed by an user of Mp3tag).

Enjoy :slight_smile:

#5

Hi,

I just tested the new feature and I am impressed one more time :stuck_out_tongue:. Seems like all three web sources are working well.

There are some things that catched my eye on this feature:

  • When non-standard tags are shown (like "style" in the allmusic dialog) it would be nice to be able to choose whether to use/tag them ore not
    Now that I think about it - maybe this would be nice for the FreeDB dialog as well in case you already have tags and only want to modify certain tags (with checkboxes in the columns or something).
    If this doesn't fit into the gui for usability reasons you could still add an option like "never use non-standard tags" to the preferences.
    Hmm - I think there was some reson why existing tags are deleted before tagging from freedb but can't think of it right now?! Modifying the .src files doesn't seem too easy on first sight... :ph34r:

-I like the feature to safe the album art as files since I like this better then to tag it to the music file (I didn't try Sebastians amazon tool yet). In the safe dialog it would be nice to be able to use variables like "%_directory%". Of course other variables like %album% would only work on the second go if no tags are initially present.

-As I have mentioned a while ago I would like to have a feature like "always write leading zeros for the track tag" in the preferences to safe one step in the tag editing. This came to my attention again when AMG didn't write leading zeros.
For some reason the freedb uses leading zeros when the track number tag had leading zeros before, the web sources seem always to use no leading zeros.

-maybe the "freedb" menu item should be renamed to "web sources" or something. :smiley:

Thanks again for another nice feature! :slight_smile:

best regards,
Vitamin

#6

Hi all,
only a question, can i use amazing "web sources" thought a proxy?

regard
Antonio

#7

Antonio,

I'll add proxy support to the next release! I simply forgot it :slight_smile:

Best regards,
~ Florian

#8

Would it be possible to search multiple sites or a list of sites in a serial fashion in case one site had no data?

FWIW- I've been using this feature EXTENSIVELY sorting out 400Gig of music and adding album art for nearly all of it. With all of the sources now listed there's VERY few albums I cannot get art for. This makes your program head and shoulders over any other IMO. Few programs make tracking this information down this easy or have so many sources. A VERY big thanks!

#9

BLKMGK that is basically the same feature I am hoping for, which is the ability to parse different html index pages.
This may be accomplished, for example, by adding some command to change [IndexUrl] (and then of course refreshing the html page)

The advantages i see are this:

  • Allows joining of multiple site search results
  • Solves the problem if the album list page can't be obtained in a single search link
  • Allows album list display method to be manipulated before it is actually displayed (like i can arrange the albums by date on the site first)
  • Many other possibilites

oh how i badly desire this :laughing:

#10

Wish list

  • Allows joining of multiple site search results
  • Solves the problem if the album list page can't be obtained in a single search link
  • Allows album list display method to be manipulated before it is actually displayed (like i can arrange the albums by date on the site first)
  • urlencoding function for search entry
  • regex for search entry
  • better documentation for src framework (tags names, functions, examples, ...)
  • proxy support
  • multiple cover art update or add

Thanks for this great tool

#11
  • better documentation for src framework (tags names, functions, examples, ...)

Seconded. E.g. "Web Sources Framework" should say how/where to install a description file.

BLKMGK, you say

I've been using this feature EXTENSIVELY sorting out 400Gig of
music and adding album art for nearly all of it.

Can I ask: did you do this one album at a time? E.g. for each album, you import/select just its tracks?

Or is there a way to do albums in bulk?

#12

just a little question:
is there a name for the program language used for the web source scripts?
maybe i can use my acquired knowlegde for a job application, but i need to know the name.

#13

Hi Florian,

I think it is a good idea to introduce some king of INCLUDE . command so certain code can be reused in Web Source files (.src).

I have mention this idea in Web Source Framework Discussion topic.
E.g. add following block to a separate file.

# Album
outputto "Album"
findline "class=\"profile"
findline "href=\"/artist"
regexpreplace "^.+</a>" "</a>"
findinline "</a>"
findinline " - "
sayuntil "</h1>"

INCLUDE this file in .src file. Changing included file would affect all related .src files.

Thank you,
Victor

#14

Export configs too.

#15

Hi,
Is it possible to assign multiple value to a multivalue field using the Web sources Framework?
Let's take the Genre Field an an Example: If I manually set the the genre of a track to genre1\\genre2, both genres are saves as individual values and can e.g. be used for grouping in players that support multivalue fields. But I wasn't able to achieve the same effect using the Web sources Framework. outputto "Genre" say "genre1\\genre2" just writes the backslashes to a single genre field (even If I use 4 backslashes because they are Escape Characters). It there another solution?
Thanks for your Reply,
Simon

#16

the answere is here:
multivalue tags for foobar2000 with discogs script

you have to press ctrl + s after running the script and the tag field must be displayed in the tag panel (which in case of genre is always true).

"\\" für Multi-Value-Tags funktioniert nur bei manueller Eingabe
#17

I think a few things need to be added to Web Source Script interface:

  1. User should be able to exclude certain tags in the media file's update popup regardless of the script. E.g. a Discogs/Amazon/MusicBrainz script can fill 15 tags although user wants to use the same script and just leave certain tags untouched.
    Each field/column should be somehow included/excluded from update (a checkbox maybe???). An option to save the selection as a variant could be provided.
    I think this "wish" was already requested?

  2. [SearchBy] popup window should be enhanced with an advanced mode (new button) to support multiple search fields. Something like discogs manual search http://www.discogs.com/search?q=&btn=&type=all.

I.e. search by artist, album, release ID.
[SearchBy]=&artist=$if2(%albumartist%,%artist%)$if(%album%,&release_title=%album%,)$if(%discogs_release_id%,&release_id=%discogs_release_id%,)
currently sets [SearchBy] to &artist=Talking Heads&release_title=Naked&release_id=2942139 in the WebSource popup.
Search fields/values could be displayed in a matrix.

-------------------------------------
Field          | Value
-------------------------------------
artist         | Talking Heads
release_title  | Naked
release_id     | 2942139
--------------------------------------

User could add/change/remove (predefined) selection.
Nothing changes on the WebSource itself. The advanced mode doesn't limit itself in any way. API has to support the listed fields.
We can go even further and provide dynamic solution: the list of all possible filter fields is populated in the script (a new variable). E.g. artist, format, label, year, etc.

-------------------------------------
Field          | Value
-------------------------------------
artist         | Talking Heads
release_title  | Naked
release_id     | 2942139
label          | N/A
year           | N/A
--------------------------------------

N/A means ignore this field from the [SearchBy] parameter.
So far there are 2 ways of joining the fields:

  • =&=...&=
    most of the sites use this format. e.g. artist=Talking Heads&release_title=Naked
  • : AND : ....... AND :
    This is used mainly by Musicbrainz api v2. e.g. artist:talking,heads AND release:naked.

The would be useful to not use a dedicated script file for each searchby (IndexURL and AlbumParse stay the same).
If someone is not positive for this addition because of available alternative (sort and search though the results) I reply in advance that the aim of mp3tag is to reduce the time of end-user otherwise why would we use mp3tag? The alternative was before mp3tag was created: fill everything manually.

#18

Yeah, I'm interested too, Florian. Is it officially "Web Sources Framework" not FloSkript or something...? :w00t:

#19

hey everyone.thank you guys for your great job.i just want this web source so badly.it is a great music database that most of my music is from there and i dont find them any where elese if anyone can make this for me.it would be great.

1 Like
#20

The new json based web_source just R_O_C_K_S :wink: :wink: :wink: :wink: :wink: ....