I thought CBP or IO have the authority to turn away the GC holders if they need to.
No - they find you inadmissible and then place you in removal proceedings. You always have the right to contest. Now they may place you under arrest and keep you locked up - but you can call someone and still have to go to court. For example, someone I know committed a crime in the US and then left and came back - GC was taken away and individual was placed under arrest - but was given a phone call. Family members got a lawyer - went to court - got it resolved and PR status was reinstated.
I've heard of instances when CBP has determined someone never lived in the US when they crossed from the CDN border and cut their GC in half. But they were still given the option to contest it. Some don't and that is when you are turned away. The only way you will be turned away is if you officially surrender your permanent resident status. So never do that and you will be sent to an Immigration Judge for final decision.