Roma77,
Your analysis is very interesting and I'm eagerly waiting to see when you have the full result. My number is D-04301-2XXXX and I'd love to see how they are doing for the cases in that neighborhood.
One quick piece of advice, if you choose to use it. I know nothing about the legality of this program, I'm not a lawyer and don't play one on the TV. I only have a technical input based on my experience with other sites. I don't expect DoL to be doing a great job in monitoring the server performance and hence I believe, it is unlikely that they will track you or block you from accessing the machine. That will be my guess; unless or until you create a problem for them. You clearly don't want to use multi-threading and put load on this site. Because, there is a high likelyhood that you will crash their server with load (I have experienced it first hand on a retail client website. I was doing it under a business understanding with them of course). If that happens, some one will clearly go looking at the log and may very well black-list your IP address.
A standard way for this type of web-crawling (it's pretty common on retail websites for comparison shopping or event tickets search websites) is to use a "politeness" factor. This means, they typically give 5-10 seconds between each request so that server can continue to function normally. I think, a minimum of 5 second delay will do you well. We can all wait for that program to finish (may be a week?) and look at the data as opposed to having you black listed and losing all this insight. Another piece of advise is "do not distribute your program". This can be a real issue and would be very similar to the "Denial of Service" attack worm/virus that is spread over the Net.
Having said all that, I'd love to get the information for my range of numbers, when you get a chance. Thanks.
