The key documents here are your mother's birth certificate (which is proof of her U.S. citizenship), your birth certificate, the record of your mother's marriage to your father and their divorce decree. These documents do prove that you were born in wedlock to a U.S. citizen.
Now, about the proof that your mother satisfies the 10 year residency requirement prior to your birth, with at least 5 of those years after her 14th birthday. Based on the timeline you provided, it is clear that these conditions are indeed satisfied which does make you a U.S. citizen by birth. The challenge is to collect enough documents to convince the State Department or USCIS that this is indeed the case.
The fact that your sister's N-600 was approved is actually good news. I assume that she did not provide more documentation than what you listed above, and that was sufficient for USCIS in your sister's case.
A few suggestions regarding trying to collect extra supporting documentation:
1) Get your mother sign a notarized affidavit listing the dates of her residence in the U.S.
2) Try to get a few other people, such as other relatives and maybe family friends who knew your mother at that time to sign similar notarized affidavits confirming that they knew your mother during that period (prior to 1975) and that she was indeed living in the U.S. all that time.
3) Do try to get your mother's high school records. Even if the high school she went to no longer exists, the local state education authorities (such as city or county superintendant of education or some such official) should still have some kind of records confirming that your mother was a student there.
If your mother still has her high school diploma, that would help too.
4) Check for earlier records, from elementary school and maybe even pre-school for your mother. You may be able to recover some of them.
5) Try to get copies of your mother's medical records from the time she was a kid. She certainly had a pediatrician and then a family doctor and most likely would have been a regular patient at some clinic. If you find out what those were, your mother may request copies of her medical records from that time.
6) Check if there are still some bank records available from that period. Your mother probably had a bank account then and if that bank still exists, they may be able to provide her some records to that effect.
7) Your mother can try requesting photocopies of her federal income tax returns from IRS for period prior to 1975 (although I am not sure how far back IRS keeps such records). There is a special IRS form for requesting such copies:
http://www.irs.gov/pub/irs-pdf/f4506.pdf
Once you do have enough supporting documents, I do think that scheduling an appointment at a passport agency to try to apply for a passport is a good idea. This way you will get a decision very quickly, within a couple of days, and you would get to talk to someone face to face there. In order to do this you'll need to show them a plane ticket for a foreign trip within 14 days from the date of appointment: passport agencies only handle passport applications for people who are planning to travel abroad in less than 14 days. (Other applicants have to apply at a post office and it takes a few weeks to get a decision.) Even if you have to spend some $400 on a nonrefundable plane ticket, that would be small price to pay for straightening your citizenship dcumentation problem.
.. Dont worry, things will just be fine.