I think it really depends on the local setup. When I took my oath in the Baltimore USCIS federal building last week, there were no facilities for applying for a passport at the same time.
The problem is easy to solve however - take a check book and two regular passport photos to a post office in a major town or city with your and you can submit your application the same day. Note: some smaller post offices accept passport applications but require you to have an appointment, so your choice of location may impact what you can do.
The application process requires you to submit your original certificate of naturalization and show a second form of id (e.g. drivers license). There are spaces on the form to specify separate mailing address from residence address.