Guidlines from the USCIS
For a while there was some confusion on the USCIS Web site about whether you must carry your GC at all times, or not. There was some wording that said you must know where it is at all times, which implied that if you were asked for it, you could be allowed to go and get it (like if you forgot your driving license at home and were stopped by the police).
However, the latest M-618 Guide for New Immigrants published by the USCIS states on page 8:
As a permanent resident, it is your responsibility to:
• Obey all federal, state, and local laws.
• Pay federal, state, and local income taxes.
• Register with the Selective Service (U.S. Armed
Forces), if you are a male between ages 18 and 26.
See page 11 for instructions.
• Maintain your immigration status.
• Carry proof of your permanent resident status at all
times.
• Give your new address in writing to the Department
of Homeland Security (DHS) within 10 days of each
time you move. See page 12 for instructions.
And on page 9:
Permanent residents are issued a valid Permanent Resident
Card (Form I-551) as proof of their legal status in
the United States. Some people call this a “Green Card.”
If you are a permanent resident who is 18 years or
older, you must carry proof of your immigration status.
You must show it to an immigration officer if asked for
it.
I guess it doesn't get much clearer than that, although some people still do not carry their cards at all time for risk of having them lost or stolen. Getting them replaced is a pain.
Practically speaking, if there is ANY chance you might be stopped and asked for it (such as in a high-risk area, airport, etc.) it would appear wise to carry it with you.