New Simple/Robust Case Status Scanner

Discussion in 'California Service Center I-140' started by talkative, Oct 30, 2004.


What do you want next from the scanner?

  1. Simplify/document the Scanner

    5 vote(s)
  2. Add more scanning features

    10 vote(s)
  3. Add more reporting features

    9 vote(s)
  4. Add more interop features (import/export/merge)

    2 vote(s)
  5. Add a GUI

    16 vote(s)
  6. Fix more bugs

    3 vote(s)
  7. Forget about BCIS and Java, go get a life..

    5 vote(s)
Multiple votes are allowed.
  1. talkative

    talkative Registered Users (C)

    ---> New Simple/Robust Case Status Scanner <---

    I wrote a scanner with these objectives:
    • Simple and reliable
    • Should be able to initially scan a range and build a database
    • Once a database of interesting cases is found, refresh cases
    • Easy to configure and use
    • Merge somebody else's database into current database
    • Automatically rate proxies and select the better ones
    • Extract proxies from a URL to a web page having a proxy list
    • Tab seperated reports for easy data export (into excel etc)

    I don't know if I met all the above goals, but anyway, here's the scanner with source. I'm happy with it so far, it does what I wanted simply and easily.

    Basically you pick a range you're interested in (like the few days around when your case was filed) and run the scanner in "-search" mode. This will find cases that are of interest. Like all scanners, it uses a list of proxy servers to overcome the 200 request limit placed by the online status page.

    Once the database has entries that you're interested in, you can later run it in "-refresh" mode to pull updates. If you have fewer than 200 odd cases, then you can do away with proxies, as BCIS will allow that many queries.

    It will show a lot of messages about what's going on about proxy failures etc, but that's for information only. It will not corrupt the database in any way or halt progress. You can use the -flush switch to force the database to be saved every 'n' queries (default 10).

    Feel free to use this thread if you have feedback/questions. There are a lot of good scanners here, however I didn't find something that quite fit my requirements and so I wrote this. (Yes - BCIS has reduced me too into a useless frustrated wreck who spends half the day reloading status page to see if LUD changes).

    The scanner has a "-merge" option that allows you to merge other people's database into your own. So please post your databases here on this thread to benefit others as well. It has some handy reporting capability using "-report" option. But as always, you can directly look at the database which is a simple text file.

    To run the scanner, unzip the attached file and run the JAR file (examples below). If you'd like to modify it, the tscanner.jar file has source included in it.

    Please be nice to others. Do not do massive searches using public proxy servers. You'll exhaust their quota and nobody else can use these proxies for scanning for a whole day.

    Basic operation:

    • Take a look at the usage and some examples by running:
      java -jar tscanner.jar -help

    • Do a one-time search in the range and type of cases of interest:
      java -jar tscanner.jar -database mydb -search -type 140,485 -start WAC0400150000 -end WAC0400153000

      Above will cause the scanner to look at all cases in the specified range and write ONLY 140/485 cases to the database. Note: while scanning in "-search" mode, even though you specify case type, it will still need to fetch all cases. Only the types you specified will be saved and refreshed in the future.

    • Once you have identified the cases using -search, from then on, you need to only do a refresh to update cases that interest you:
      java -jar tscanner.jar -database mydb -refresh -type 140,485

    • Now you can ask for some reports:
      java -jar tscanner.jar -database mydb -type 140 -report detail
      java -jar tscanner.jar -database mydb -type 140 -report update:3
      java -jar tscanner.jar -database mydb -type 140 -report status

    • And you can merge databases:
      java -jar tscanner.jar -database mydb -merge friendsdb

    About BCIS query quota and proxies:
    Do not do "-search" more than once. It's wasteful as it examines each and every cases in the specified range. Only 10% or so cases are 140/485, so most of the queries go waste. Do a "-search" only once (the first time), and then onwards, simply do a "-refresh".

    For most of you who scan range of a few weeks, you won't even need proxies (and all the headache) once you're past your initial "-search" run. You can do a "-refresh" without any proxies daily and you'll likely never hit the BCIS quota.

    There's a "-suck" option to find new proxy servers. Jut point it to a web page with a proxy list and it will try to parse the page. In addition, if you specify a -proxy file, the quality information of these proxies will be saved as well. I normally use Alive Proxy or Stay Invisible.

    Please post your databases to so that others can benefit as well. Here is the thread to discuss scan results. You can easily and safely merge multiple databases using the "-merge" option to form a consolidated database.

    If you run into problems running the scanner, make sure you have a fairly new version of Java (JRE). It's available here (click the "Get it now" button).

    Last edited by a moderator: Dec 6, 2007
  2. NIW485waiter

    NIW485waiter Registered Users (C)

    I tried your program here are stupid questions,

    Since i don't have a java program in my WinXP computer, so i just went to DOS command and excute
    java -java .....

    The programe gave me either proxy failure, unexpected end from server, connection refused,

    Also programe didn't run without proxy file.

    Hope you can fix the bugs, thanks!
  3. talkative

    talkative Registered Users (C)

    > proxy failure, unexpected end from server, connection refused

    NIW485Waiter, you will need a list of proxies. The one I was using is probably stale by now. The proxies that you add must be working and must support HTTPS (CONNECT) protocol. I usually get mine at Alive Proxy or Stay Invisible

    > also programe didn't run without proxy file.

    I've modified it to run without proxies and updated the ZIP file in the original post. However, note that without proxies, you're limited to roughly 200 queries imposed by the BCIS website. [When you hit this quota, you'll get a soft failure: quota exceeded message]

    Last edited by a moderator: Nov 1, 2004
  4. IWantFree

    IWantFree Registered Users (C)

    Please post scanner result if you have run it.

    I ran old version of scanner, and got some results.
    why don't we post result on this threads.

    Case number--Cat----ScanDate--Tot--APV--RFE/TRAN---DENY--RCV/Pend
  5. disney2k

    disney2k Registered Users (C)

    Hi Talkative,


    Tried the following command:

    java -classpath c:\scanins\tscanner;c:\scanins\tscanner\tscanner.jar;c:\java\j2dsk1.4.1_01\jre\lib\rt.jar -jar tscanner.jar -database status.db -proxies proxy.lst -search -start WAC1234567890 -end WAC1234567899 -verbose

    getting the following error:

    Working.. To safely quit, type 'q' and hit enter [may take some time]
    Got error: unknown protocol: https

    What am I missing?
    Also, the new zip file you added has no proxy.lst file (the old one had it)?

  6. sonpapdi

    sonpapdi Registered Users (C)

    I tried .. it rocks

    I tried and it works fantastic .. but the only problem is it gives a lot of results which are not pertained to 140 or 485. So its a little hard to figure out the interested cases !!

    Thanks a lot talkative !!

  7. talkative

    talkative Registered Users (C)

    Sonpapdi, thanks a lot for your good words!

    As for filtering data, you can pass the "-type" switch with "-search", for example -database mydb -search -type 140,485 this will cause the scanner to only write 140 and 485 cases to database while doing a search.

    Then, once you have the cases of interest in your database, you should do "-refresh" instead of "-search" - this will cause it to find updates for only the cases in the database.

    • Do a one-time search in the range of interest:
      java -jar tscanner.jar -database mydb -search -type 140,485 -start WAC0400150000 -end WAC0400153000

      This will cause the scanner to look at all cases in the specified range and write ONLY 140/485 cases to the database.
    • Once you have identified the cases, next time you only need to do a refresh:
      java -database mydb -jar tscanner.jar -refresh -type 140,485

      This will pull updates for only the cases in your database, that is, the 140/485 cases in the WAC040015XXXX range.

    Finally, the "database" file is a simple text file, so you can look at this instead of the console output for data. It's better organized and easier to read.

    Let me know if this works for you!


    PS: You can pass the "-quiet" switch if you'd rather not see so much information while scanning
    Last edited by a moderator: Nov 1, 2004
  8. talkative

    talkative Registered Users (C)


    First of all, the case numbers are not correct. It needs to be of this format:

    yy=Two digit year, like 04
    ddd=BCIS day of the year, like 124
    ssss=Serial number, like 5124

    An example would be WAC0402351421 which would be 23rd working day in 2004, and serial number 1421.

    > unknown protocol: https

    You may be getting this error because you are explicitly setting a classpath and missing some system libraries. Try copying tscanner.jar to your working directory (c:\scanins\tscanner) and running it as:
    c:\scanins\tscanner> java -jar tscanner.jar <switches>

    If this doesn't help, it may be a JDK bug. Try using the latest JDK or JRE (1.4.2_06) from here: Sun JDK

    > Also, the new zip file you added has no proxy.lst file

    I removed it because proxy lists go out of date quickly. Try getting one from AliveProxy (lin in the original post). But first, you can try running without any proxies (without "-proxies" switch) to make sure everything is working.

    Let me know if any of these resolve your problem. If not, send me a private message and I can give you a version with additional debugging code to narrow down the problem.

  9. sonpapdi

    sonpapdi Registered Users (C)

    Rocks again !!


    Yes I corrected and now it works as it was desired earlier. This makes it an excellent program !! I am a happy customer :)

  10. NIW485waiter

    NIW485waiter Registered Users (C)

    yes, it works. Thanks!
    Only problem here is i could not find any proxy servers , either they was not working or "Daily quota exceed"
    Last edited by a moderator: Nov 1, 2004
  11. mish4gc

    mish4gc Registered Users (C)

    What is this bscis date of the year? Is this the RD from Jan 1st? How is this calculated?
  12. california_gc

    california_gc Registered Users (C)

    It is the julian date i guess

    7th jan is the 7th day
    30th dec is the 364th day (in a non leap year)
  13. mish4gc

    mish4gc Registered Users (C)

    Thanks california_gc, though, I don't think thats how it is, at least it does not work in my case. does it in urs?
  14. talkative

    talkative Registered Users (C)

    I think it's working days since the beginning of BCIS fisical year!
  15. tominavhech

    tominavhech Registered Users (C)

    In my case, it does not hold good (WAC040835XXXX)
    I filed my I-140 on Feb-3rd. According to ur logic it was the day 83 of the year??
  16. globus

    globus Registered Users (C)

    Weekends and holidays dont count and year starts from 10/01.
    Last edited by a moderator: Nov 1, 2004
  17. tominavhech

    tominavhech Registered Users (C)

    Thanks Globus. If they USCIS year starts on Oct 01st, it makes perfect sense!
  18. tominavhech

    tominavhech Registered Users (C)

    I have JDK 1.3 and for some reason wd not be able to install later version. Can somebody please put the latest I-140 updates for EB3 date range Feb-02-04 to Feb-04-04.
    Thanks in advance.
  19. hate140delay

    hate140delay Registered Users (C)

    Great job on the scanner, seems to work fine. However i am getting this issue, and sorry if this questions sounds silly. I am getting this error everytime -
    Proxy failure [soft: daily quota exceeded]: ProxyEntry
    I know there is a proxy setting. So, I got a list of proxies from AtomIntersoft, put it in the proxy file (called proxies), and gave this command -
    java -jar tscanner.jar -database mydb -search -type 140,485 -start WAC0324352071 -end WAC0324352158 -proxies proxies

    However it keeps giving the same error. I appreciate any help.
  20. talkative

    talkative Registered Users (C)

    as for the proxy problem, unfortunately, it seems most proxy servers' quota are exhaused quite quickly. Try looking also at - just list many servers, maybe you'll find one or two that still are working and have not exhaused their BCIS quota.

    thanks! it looks like you are scanning a small range of files, less than 200. So you don't even a proxy. Just leave out the "-proxy" switch and you can scan using your own local machine. If your local machine has run out of queries for today, you can try again after 24 hours.

    Important Note:
    Don't do a "-search" all the time. When you do a -search, it examines each and every case. It's wasteful. Just do a -search only ONCE. This will identify the 140 and 485 cases in your interest range. Then onwards, you need to do only a "-refresh". You can also set a smallar timeout if you have many proxies listed and most of them are bad (ie, "-timeout 3000" will give each proxy 3sec before it dumps it and tries the next one. It's 5sec by default)

    For most of you, if you run "-refresh" once a day, you probably won't even need any proxies.

    So once you get past the -search phase, you can forget all the proxy crap and simply run a "-refresh" to track cases of interest to you using your own machine's 200 query quota.
    Last edited by a moderator: Nov 1, 2004

Share This Page