Performance: data pulling extremely slow

[rndsrc]

Currently, µcast uses NOMADS' grib filter to get GFS forecast data. However, the grib filter takes a few seconds to response and transfer back a few grid points. Because the NOMADS server can only accept a finite number of requests at a time, parallel requests from a single client would not improve the performance. This is the bottleneck of µcast.

The NOMADS website suggests a technique called partial-http transfer and provide scripts and examples for a special case. Because the perl scripts are running on the client side, some simple tests suggest that this would significantly improve the performance.

[wumpus]

I'm currently using 20 separate processes to download 20 stations (current + future) and that's working fine, perfect speedup. However, the NOMADS people have said that they're going to put a 120/minute rate limit on everyone stating April 20th. So I would like to switch to something better, and this looks pretty good, I hadn't seen the details for it before.