\
**Category:** Web Sources Discussion\
**Created:** [March 19, 2018, 4:49pm UTC](https://community.mp3tag.de/t/cant-get-full-browser-html-source-for-imdb-titles/41140 "2018-03-19T16:49:00Z")\
**Posts on this page:** 9\
**Page:** 1
**Author:**  [@chuck](https://community.mp3tag.de/u/chuck)\
**Post date:** [March 19, 2018, 4:49pm UTC](https://community.mp3tag.de/t/cant-get-full-browser-html-source-for-imdb-titles/41140/1 "2018-03-19T16:49:00Z")
Hello,
I have an IMDB sources script to scrape movie metadata for my library. A few months ago it appears the structure of the returned HTML changed significantly. I am now working to fix the script. I have noticed that the HTML in the debug output is significantly different from the HTML source I see directly in my browser (i.e. Chrome), and the problem is there is a lot of data I want to scrape that is only available in the HTML source Chrome receives. I _presume_ IMDB is detecting the browser type and returning different HTML to the web sources framework. Does anyone have a suggestion on how to work around this so I can receive the richer HTML the browser receives?
Thank you
---
**Author:**  [@Florian](https://community.mp3tag.de/u/Florian)\
**Post date:** [March 19, 2018, 5:16pm UTC](https://community.mp3tag.de/t/cant-get-full-browser-html-source-for-imdb-titles/41140/2 "2018-03-19T17:16:30Z")
Hello Chuck,
are you familiar with the `curl` command? You could use this command to test whether sending a different user agent string would solve this.
Can you maybe post the URL for the request that gives the different output?
Kind regards
— Florian
---
**Author:**  [@chuck](https://community.mp3tag.de/u/chuck)\
**Post date:** [March 19, 2018, 5:27pm UTC](https://community.mp3tag.de/t/cant-get-full-browser-html-source-for-imdb-titles/41140/3 "2018-03-19T17:27:53Z")
Hello Florian,
Thanks for responding. Here is an example URL (Blade Runner 2049):
> **[Blade Runner 2049 (2017)](https://www.imdb.com/title/tt1856101/?ref_=nv_sr_1)**
>
> Directed by Denis Villeneuve. With Harrison Ford, Ryan Gosling, Ana de Armas, Dave Bautista. A young blade runner's discovery of a long-buried secret leads him to track down former blade runner Rick Deckard, who's been missing for thirty years.
I will experiment with cURL in parallel. How would I override the agent string in the mp3tag web sources framework as well?
Thank you,
Chuck
---
**Author:**  [@Florian](https://community.mp3tag.de/u/Florian)\
**Post date:** [March 19, 2018, 5:45pm UTC](https://community.mp3tag.de/t/cant-get-full-browser-html-source-for-imdb-titles/41140/4 "2018-03-19T17:45:07Z")
> [@chuck](#):
>
> How would I override the agent string in the mp3tag web sources framework as well?
This is not possible (yet). We'd first need to check whether this would address the issue at all.
There is a setting for sending Mp3tag's user agent, e.g., `Mp3tag/2.86` and you can check whether this already solves the issue by adding
`[UserAgent]=1`
to the upper header part of the tag source. By default, it's sending `Mozilla/5.0 (compatible)` currently.
P.S. Good choice of URL by the way 😃
---
**Author:**  [@chuck](https://community.mp3tag.de/u/chuck)\
**Post date:** [March 19, 2018, 6:00pm UTC](https://community.mp3tag.de/t/cant-get-full-browser-html-source-for-imdb-titles/41140/5 "2018-03-19T18:00:43Z")
Thanks Florian. Unfortunately, it does not appear the HTML I received is any different ☹ Any other suggestions?
---
**Author:**  [@Florian](https://community.mp3tag.de/u/Florian)\
**Post date:** [March 20, 2018, 9:50am UTC](https://community.mp3tag.de/t/cant-get-full-browser-html-source-for-imdb-titles/41140/6 "2018-03-20T09:50:52Z")
Can you tell me which are the most significant parts that are _missing_ when accessing the page through the Web Sources Framework?
You can wrap code in a code block using the `` icon from the toolbar here.
---
**Author:**  [@chuck](https://community.mp3tag.de/u/chuck)\
**Post date:** [March 20, 2018, 1:54pm UTC](https://community.mp3tag.de/t/cant-get-full-browser-html-source-for-imdb-titles/41140/7 "2018-03-20T13:54:34Z")
Here is an example, specifically plot keywords. The plot keywords are completely different.
First is what I see in my IMDB output from the web sources framework:
```auto
| Plot Keywords |
|
```
Next is the HTML I see in Chrome via view source
```auto
Plot Keywords:
short skirt
|
bare breasts
|
female nudity
|
micro mini skirt
|
miniskirt
| See All (575) »
```
---
**Author:**  [@Florian](https://community.mp3tag.de/u/Florian)\
**Post date:** [March 20, 2018, 6:53pm UTC](https://community.mp3tag.de/t/cant-get-full-browser-html-source-for-imdb-titles/41140/8 "2018-03-20T18:53:19Z")
I've tried it with a simple test script and got the very same keywords via the Web Source Framework as you're getting when using Chrome.
```auto
# ###################################################################
# Mp3tag Tag Source Test
# ###################################################################
[Name]=imdb.com
[BasedOn]=www.imdb.com
[PreviewUrl]=https://www.imdb.com/title/tt1856101/?ref_=nv_sr_1
[AlbumUrl]=https://www.imdb.com/title/tt1856101/?ref_=nv_sr_1
[WordSeparator]=%20
[SearchBy]=%title%
[Encoding]=url-utf-8
[ParserScriptIndex]=...
#
[ParserScriptAlbum]=...
# ###################################################################
# A L B U M
# ###################################################################
debugwriteinput "C:\Users\florian\Desktop\debug.out"
debug "on" "C:\Users\florian\Desktop\debug.txt"
```
---
**Author:**  [@chuck](https://community.mp3tag.de/u/chuck)\
**Post date:** [March 20, 2018, 9:52pm UTC](https://community.mp3tag.de/t/cant-get-full-browser-html-source-for-imdb-titles/41140/9 "2018-03-20T21:52:37Z")
Thanks Florian. That was the key I needed. The script I started with would use the IMDB "references" view for the AlbumUrl, which dramatically changed at the end of last year. This explains why it is different from what I saw in Chrome. The reason the data, such as keywords, is different is due to the data in the standard and references view being different (i.e. plot keywords).