Request: Insert settings values into URL queries

Sorry to be so brazen with constant brain-farts and requests but I love the community’s creativity and pragmatism towards solving problems and @Florian is doing a fantastic job with furthering the capabilities of web sources and it’s been VERY much appreciated :cowboy_hat_face:

This is a follow up to the initial topic. I’ve had a bit of an over-think about it but can’t think of a definitive solution. :melting_face: I would seriously love to see other’s opinions, regardless of your technical experience with web sources, on what I think is a much-needed feature or on possible expansions that would also satisfy the requirement.

The problem:

We’re able to save and deploy a user’s settings as string-based customisations all over our scripts except into the initial URL query created by [SearchBy] before it’s sent to a website.

Data like language preferences and API keys are needed from the first step of running a web source. An example is the TMDB API[1], where no queries can be made without an API key:

https://api.themoviedb.org/3/search/multi?query=game+of+thrones?append_to_result=credits,season/1&api_key=...&language=en-GB

The current method is requiring users to manually insert such data into .src’s or constant maintenance of variations of .src’s[2]: storing it in a web source’s settings as a convenient, centralised access point would be preferable.

Considerations:

I’ve compiled a list of possible ways to approach the matter, ranging from basic additions to new parsers:

  1. Each settings key gets a persistent output buffer with %Var_key%.
  • To be used almost anywhere in a script including [SearchBy], [ParserScriptIndex] and [ParserScriptAlbum].
  • “Var_” is automatically prefixed by Mp3Tag to distinguish it from other outputs like how we use “Temp_”.
  • Could open up syntax possibilities like tag values being read in as “Tag_” or “Num_” to define strictly numeric values.
  • Doesn’t negate SayVar which is still preferable for readability.
  • Changing a value in [SearchBy] wouldn’t affect the value in .settings; this is merely quick access to temporary settings values much like tag %placeholders%.
  • This would need more than 5 fields in [SearchBy]? Hidden fields for values we don’t want users to see or edit? Can’t choose from lists? Not as elegant a solution.
  • Can’t validate a URL query before it’s sent between [SearchBy] and [ParserScriptIndex].
  1. Support for format string/function framework in [IndexURL] and [AlbumURL][3].
  • Requires implementation 1 or $var(%key%).
  • Allows more specific placements of data in a URL than %s.
  • Certain data like API keys are kept hidden, only viewable in settings.
  • % or empty values from %placeholders% might interfere with the URL structure.
  • Format strings get pretty lengthy using $if() conditions to validate data.
  • $var() could suggest broader support outside of web sources which isn’t the case (eg. $loop() only being available in Export).
  1. [SearchConfig], an optional [Key] adding settings fields to the bottom of [SearchBy] using name||%key%||%c||hide.
  • Requires implementation 1 or specific implementation for settings %keys% for this prompt alone.
  • Changing a value in [SearchBy] wouldn’t affect the value in .settings.
  • Could bring over the same field types seen in settings prompts such as lists and numbers with built-in validation.
  • Standard [SearchBy] fields could maybe utilise such field types.
  • Could open up future options like languages-REGIONs based on Mp3Tag preferences or external configs from an .inc file or a website[4] (would need a parser).
  • Likely limited to 5 fields (creating a maximum of 10 in [SearchBy]).
  • Hidden fields that we don’t want users to see or edit via prompt yet include in the URL? The fourth section set to ||1 could do this (empty or 0 is a normal “temporary” field).
  • Extra parsers would allow commands, therefore more control than this idea.
Other considerations
  1. Most search/parser [Keys] have a parameter to query for a website’s response (empty or =0), skip query and pass all output buffers (=1) or pass both (=2).
  • Allows editing/validating a URL before any queries.
  • [SearchBy] and [ParserScriptIndex] could become automated to pass values without user input. [ParserScriptAlbum] is the exception.
  • Output is only used by [ParserScriptIndex] to generate an index, debatable if still passed as likely needs reset between index prompts.
  • Input could give simpler access to previous [Key]’s query response (JSON/HTML) than relying on json “ON” or GoTo/Line commands.
  • [SearchBy]’s output buffers would be each field value including CurrentURL.
  • Sharing buffers between parsers is a great idea[5] but requires extra debugging to make sure Set gets all unwanted buffers). Would be useful for [SearchBy] values if changed from defaults as currently relying on parsing “dummy” parameters in URLs.
  • A command to determine this behaviour would not be possible for the first [Key].
  • This implementation then requires either of the following
  1. [ParserScript], based on a suggestion by @stevehero[6] as an optional [Key].
  • Allows commands like SayVar to process search/parser outputs before passing to the next parser with no prompts or user input.
  • Allows pre-fetching more data when one query isn’t enough and/or validation of [SearchBy] outputs…
  • Requires implementation 4.
  • Can’t load settings values into [SearchBy] which would solve the problem directly.
  • This requires more work than necessary and easier solutions exist.
  1. [ParserScriptIndex], [IndexFormat], [IndexURL] and [SearchBy] can be used multiple times.
  • These are connected via Input/Output to refine a search, eg. search an artist, choose from index of results, then choose from albums.
  • [IndexFormat] updates any [ParserScriptIndex]’s expected structure.
  • [ParserAlbumIndex]would only be deployed as the end of the script, where before-hand[IndexURL] becomes [AlbumURL] and output buffers become tags.
  • Could customise the flow of prompts to remove indexing and go straight to tags.
  • Indexes can already be “skipped” when one result is found, by chance or intentionally.
  • Potential for the flow of [Keys] to be command-driven by If, json_foreach or Do While to skip to other sections or retry sections until valid.
  • [Include]at the beginning could load multiple external files to be called via command at the end of each search/parser or ran each time to load in the next one. When no more called for, move to the next search/parser in main script.
  • Requires implementation 4.
  • Can’t load settings values into [SearchBy] which would solve the problem directly.
  • This is pretty demanding (but just throwing it out there).

I’m eager to hear other’s thoughts on a potential implementation as some considerations have more power to them outside of inserting settings but simpler solutions tend to be the best. I’d quite like to see option 3 but frankly will take whatever the community wants.


  1. [WS] TheMovieDB ↩︎

  2. Best way to handle region/language codes ↩︎

  3. Suggestion: 'Splice' command for WS - #3 by AreDigg ↩︎

  4. Languages
    Movie Certifications
    TV Certifications ↩︎

  5. Output reference between [Parser] blocks? ↩︎

  6. Add a [PreIndexUrl] function ↩︎

1 Like