How To Enable Javascript When Making An Httpwebrequest

I need to call a web page that has javascript. At the bottom of the page I have the following:

Solution 1:

What output do you want to convert? You can only scrape the static HTML, not the JavaScript-modified DOM.

Remember that HttpWebRequest does not interpret JavaScript.

Solution 2:

1. Use the HttpWebRequest as you have already did
2. After GetResponse and GetResponseStream, save the stream content a temporary file (e.g. using filename from Path.GetTempFilename() method)
3. Loads it up in The WebBrowser class.
4. Lets the page executes itself for a while.
5. Walk the web browser instance's representation of the DOM to get what you want.

Hope this helps.

Solution 3:

Javascript executes on the user-agent (client-side). You are providing a false user-agent string for the request. The user-agent you are "pretending" to be has a Javascript implementation. HttpWebRequest, of course, does not.

Solution 4:

I guess I need a way to execute the javascript that in included in the response, but do so outside of the browser.

You'll need to write your own jasvascript interpreter then.

The only alternatives I can think about is using any web browser engine like webkit, gecko, etc. to render the page for you at the server-side or searching for online service like browsershots that will render the page for you.

Solution 5:

Fix the page so it doesn't depend on JavaScript. Build on things that work.