I would advise you to wait for your card
I would advise you to wait for your card.If you come in to US with only passport stamp, there is secondary verification about your credentials and they ask a few questions in a separate room.
Since it is a employment based petition, you may have problems although you did not leave the company on your own.