Honestly there is no such requirement to bring your passport, if you check your form N-445.
Documents You Need to Bring
Form N-445 lists a number of documents to bring to the ceremony.
Permanent Resident Card
USCIS collects your green card because you will not need it once you are
a citizen. If you lost your green card, USCIS may want an explanation. If it
was stolen, USCIS may also ask to see a police report.
Re-Entry Permit
A Re-Entry Permit is a document used by a permanent resident to re-enter
the country after the person has been outside of the United States more than
a year. The Re-Entry Permit is evidence that the person did not abandon
his or her permanent residence. It shows the reason that the person had to
be outside the country for a long period of time. USCIS wants to see this
document at the ceremony, even if it has expired, to know that the person
is still a permanent resident eligible to naturalize.
Refugee Travel Document
A Refugee Travel Document is used by refugees and asylees to travel outside
the United States without a passport. Once they are citizens, refugees and
asylees should use a United States passport.
All USCIS Documents You Have
All immigration documents issued by USCIS will be collected at the
ceremony because you will no longer need them as a citizen.
-----------------------------------------------------------------------------