Not true. Check this out:
http://www.cbsa-asfc.gc.ca/publications/pub/bsf5023-eng.html#s2x3
"Permanent residents of the United States may travel to Canada from the United States or Saint-Pierre and Miquelon without passports, travel documents or visas provided they produce satisfactory evidence of their identify and status. However, if these persons travel to Canada from any other part of the world they require passports (or travel documents) and are visa-exempt (provided they can substantiate their status as a U.S. permanent resident).
The following documents are proof of permanent residence:
-U.S. Permanent Resident Card (see Appendix II, item 11).
-Form I-688, U.S. Temporary Resident Card Ó annotated to indicate permanent residence in the United States (see Appendix II, item 13).
-A temporary I-551 machine-readable immigrant visa (MRIV) bearing the statement: "UPON ENDORSEMENT SERVES AS TEMPORARY I-551 EVIDENCING PERMANENT RESIDENCE FOR 1 YEAR" directly above the machine-readable zone, when contained in an unexpired passport and endorsed with an admission stamp, constitutes a temporary I-551, valid for one year from the date of endorsement on the admission stamp (see Appendix III, item 12)."
But, if you do have a passport or travel document, I do agree that you should take it with you just in case.