First off, you are posting in the wrong forum. This one is for adjustment of status (I-485)
Anyway, if you had significant past travel listed in the N-400 and any new travel since, then you would need the passport and the decision could be delayed because of it's unavailability.
If you had no new travel since filing the N-400 and included copies of all the passport pages, the officer might be satisfied with that. Even if he says he is taking your word for it, he may check the accuracy of your answers later in his computer (USCIS, CBP, ICE = DHS + Immigration Court records as INS and EOIR used to be DOJ). That's how a USCIS N-400 interviewer in 2010 was able to confront a guy about remarks made to an Immigration Inspector at an airport in 1996 that cast doubt on his quickie greencard marriage that resulted in a greencard back in 1995, that's from another post.
Short answer, there could be a skight delay in making a decision if he wants copies of the passport in order to decide the N-400.
On another note, how are you going to board an international flight without a passport? A greencard holder needs a passport to enter another country, hence the airline won't let you board. If you naturalize, you have to get a U.S. passport to travel as a USC. Does the country you are visiting require a visa from a USC?
Maybe, I am misunderstanding what you said in your post?
