Data Consolidation Project (Thread #7)

Status
Not open for further replies.
picsarus
I'm still new here, just started playing with the spread sheet.
I think "Too many enquiries" error depends on your IP address and not on your browser. Unless you can control your IP address and change it when you want, you have to wait 24 hours.

thanks
 
Originally posted by picsarus

On a related subject - I am plodding through SRC01-180. Slow progress due to the large number of invalid numbers. What would be a really neat enhancement would be if the macro can automatically close the browser and re-open a new one and continue with the scan from where it left off due to the "too many enquiries" error. I looked at trying to do this, but my meager skills did not allow success. [/B]
Click to expand...

picsarus,

I had a liitle thought over this sometime back. One thing I had in mind at that point was, You need to keep two sheets
One which you are getting too many missing numbers.
Another one which you find do not have many missing numbers. Pick any day around src02 second half they seem to be lot better.

If you get 3 missing numbers the next query will knock you off if it is also missing. If you get the next query successful then you can continue. So keep a counter, if you get three missing number then switch over to the other sheet make your query for the next src and then come back. Or to be even safer if you get two missing switch over to the other for one query and come back. Essentially you will be scanning two sheets in parellal to some extent.

Even if you close the browser and reopen, the error will stay on for some time , I am not sure but it is something like an hour or so.

I never attempted putting it thro. May be worth a shot, if you can


-- pr
 
Harvydonald - the situation you are refering to is the max. limit per day (1000). You also get a "too many enquiries" error if you enter 7 or in some situations 3 invalid numbers in a row. In this case, if you close the browser and re-open a new browser window, you can continue to scan. Of course, you need to be monitoring the screen while the macro is running to be able to do this. One of the enhancements that Pradha put in is to note the last valid number, so that the macro can be restarted at the next number.

Pradha - your suggestion is interesting. What you are aiming for is to avoid the error altogether. But surely, if manually we can close the window, restart the browser and continue, we should be able to put it in the macro? BTW, you can close the browser, instantly open another window and continue - no need for a waiting period. I want to think about your suggestion some more.
 
For all the gurus, I am wondering if it is necessary that we can have a new field in the summary to show what is the percentage of RFE being sent out from TSC.

Just a thought. Thank you.
 
Originally posted by picsarus
Pradha - your suggestion is interesting. What you are aiming for is to avoid the error altogether. But surely, if manually we can close the window, restart the browser and continue, we should be able to put it in the macro? BTW, you can close the browser, instantly open another window and continue - no need for a waiting period. I want to think about your suggestion some more.
Click to expand...

picsarus,

When you get denied message after few invalid numbers (it was always 4 for me. Probably there is some scenario where it goes to 7) if you close the browser and open a fresh browser and make a query for a new number will it continue? It never did for me. It always blocked me for some time like an hour.

But I have never tried to stop one less like close after 3 invalid numbers ( ie just before getting blocked), then open a new browser and query. Is it the way it works.

If it works by closing and opening the browser it should be easy for you to copy that part of the code and plug it in. I was assuming it won't work that way.


--pr
 
new scan for 02258

as requested by HarveyDonald

I could not scan 54500 upwards since I kept getting invalid src numbers. BCIS website kept giving me the 'denied access' messages.
 
add to consolidted file

GoAstros/Pradha,

Please add the new 02258 scan excel file to consolidated spreadsheet . Its in the post above.

thanks,
 
Re: add to consolidted file

Originally posted by dvorak
GoAstros/Pradha,

Please add the new 02258 scan excel file to consolidated spreadsheet . Its in the post above.

thanks,
Click to expand...

Dvorak, Picsrus, Eb2, vidy, ansty, others - please comment on the splitting of the excel file.
Pradha has Once again done a marvellous job on Excel.
Though, his programming style makes the excelsheet look cool but Like before, i had my reservation abt splitting the 1 consolidated file.
Also i notice that now even Pradha has pointed out the complexity in maintaining 3-4 files.

But still , guys give this shot and then we can decide to go back to 1 file or split files. Once we decide then we can add your or any other contribution, Dvorak.

Also in regds to the 1000 hits per IP problem, sometimes, i then change my Ip settings, this is the address from where I get the new Ip & port.
http://atomintersoft.com/products/alive-proxy/proxy-list/?ap=1
 
Last edited by a moderator:
I also like one consolidate file, even though I am not an expert on the script, but I think the maintainess effort overweight any gains.
 
My thoughts:
Lets us consolidate the data into one file.
What we can do also is to just keep the "Formulas" for say the LAST 12 months only.
For months older than that we can put the ACTUAL values.
This will probably make the file smaller and will enable the excel application calulate the results faster.

So for all months preceeding Jan 03 we can remove the formulas and paste the values only.

What do you guys think ?
 
Hey all,

Even I think the simplicity of keeping one file out weighs the advantage of the saving in size when you breakup. However I couldn't resist opening this discussion in saving of some storage, though they are dirt cheap these days. (May be because my IT carreer always was some kind of admin.)

It may still become neccessary at a later ( very later) stage if many volunteer to get data for missing dates ( I hope so) and the consolidated file outgrows the excel limits or too long to download.


It was just an agenda of topic for discussion only. We should continue with _v102 in the meanwhile. That is why I specifically did not go with a revision number.

--pr
 
In that case, lets' take it from ver 102. Next person, pls add Dvorak's sheet for 02-258.
And please start the rescan from 01-279.
Also it would be good idea to post a message here to let everyone know when you are working on a rescan.
Later you can edit out that same message and post the consolidated file then.
 
better keep it in one sheet for now

But sooner or later we will have to split it out after people add lot more data to it. Maybe we can start discussing how to go about it.
Some points would be
1. versioning/revision numbers
some primitive form of version control to maintain data integrity. Maybe CVS.
Should the spreadsheet reside here or on another site like sourceforge (don't know if this would be eligible to be put in there). We can have the discussion thread here with links to maybe another repository where the spreadsheet lives.
2. How should it be split. Based on year?
3. Who will maintain each version.
4. Should we have a separate thread for each year.
5. Who will make sure that the macros are synched up.
6. Where will be summary stats across all the spreadsheets live.

Right now its up to a few members to maintain it. At some point in time others will have to pick it up. It would be great if we put in a semi formal process in place.

Ideal situation would be that TSC would appove everybody within 3 months and no need for this project. But thats wishful thinking.


My next scan will be for May 17, 2001.
SRC01175-XXXXX
 
Finding a web server with form automation

If only we could find a web server running a form automation service -- anywhere find one?
 
v103

Here is the next version.
Ran it overnight
Next person pls take it from SRC02-030 & the SRC # is 56235.
+ Added Dvorak's 02-258

203 approvals for Dec 03
31 approvals for Jan 04
 
new scan for 01175, ND: May 17, 2001

Some approvals in Dec and Jan.
Next person, please add it to consolidated sheet.

thanks,
 
v104

Here is the next version

+ SRC01-175 ( from Dvorak)
+ SRC02-041 ( initial scrs I got it from thread for 'perl gurus')
* ReScanned 01-139 to 01-256 and then 01-279 to 02-131 while following last night's match.
Next person can either take it from 02-133 or from 01-257 to 01-255.

214 approvals so far for Dec 03
58 approvals so far for Jan 03.

01-200 ( june 01 ) has 9 approvals in Jan so far.
So looks like they are looking at June 01 data now.
This probably means we don't have good sample of data for May 01.
 
Awesome job, guys.

I am sure this forum has been fostering a lot of project management talents as well as technical talents.

Anybody interested in forming a permanent body of IT management and research organization?
 
Originally posted by dfkj45
If only we could find a web server running a form automation service -- anywhere find one?
Click to expand...

dfkj45, what does this do? can you or anyone explain what is form automation service


Originally posted by dfkj45

Anybody interested in forming a permanent body of IT management and research organization?
Click to expand...

Sure dfkj, do you or anyone have any good ideas?
 
Status
Not open for further replies.
Top