Extracting data from website

**trash** · 05-05-13, 10:19 PM

I'm wondering if anybody knows of the best or simplest way to extract data from a web site and then take that data and say send it to a serial port.
Obviously there's a lot of applications for mining real time data from websites and using it to control various things.

An example is to look at the BOM website and look at the various weather stations looking for approaching wind gust and then controlling an external device by either
sending values to an external microcontroller to make the decision and control the hardware or the PC making the decision and sending the immediate command to the device.
The other thought was to reassemble the data and then send it to another third party web page.

It's not just the BOM, that's just one example. There are quite a few pages I'd like to mine real time data from.

**tristen** · 07-05-13, 12:47 AM

Interesting project, trash.

I had thought of doing much the same thing at one stage but decided to leave it for more pressing tasks at the time.

I believe that the term for monitoring such data on websites is "web scraping". See .

One way would be to write an application that examines the html code for a particular web page (web site) and to search for and list any changed variables. (The source html code for a web page can be readily examined in most popular web browsers).

All or a particular piece of data (variable) could be monitored in this way.

Out of interest, I'll compare html code for a page on the BOM website over several days to see what happens.

A quick Google for "Web-scraping software" found . It looks like possible starting point.

**simonm4** · 08-05-13, 09:25 PM

Trash, not sure how game you are but you could try some of the powershell cmds. Fairly easy to manipulate and break data out into other files or write out to a serial port. I haven't used this cmds specifically myself, but work with PS constantly in the "non hobby" hours of the weekdaze.. with a bit of planning you can do some pretty cool stuff.

As tristen suggested you could write a script that does a compare of previous content and then write out what you need to a com port.

Examples here:

Reading http content

Writing to com ports

**jwoegerbauer** · 11-05-13, 05:27 PM

If the data to be extracted are RSS formatted (means the data structure is known to you), with a simple MortScript "RSSReader.mscr" you can easily fetch these data and process them anyways.

**wwalford** · 11-12-13, 07:53 PM

I have personally done this extraction for real time stock data . I would read the HTML look for the the various fields and then save them into a Database. 3 things I struggled with:

1. Frames, the website had multiple frames so I had to find the original URL for each frame and process each frame separately.
2. Various changes to the HTML, because I did not own the website when ever the owner made slight changes to the HTML my code would fail. I had used various HTML parsers downloaded as well as written by myself. And they all struggled with some aspect of change to the HTML code.
3. IP blocking, I found that I could not read to quickly or too often as the IP got banned. Not all sites do this but some do.

What I was doing was not illegal it would be the same as me browsing each page and writing it down instead just my program did it for me. But i did stop it eventually as I found I started missing data so I then found it easier to by the stock data I was looking for. If the data comes in as an RSS it is super easy, if you using the original HTML good luck if you have a highly active website.