No, you are not. Because technically, you are not refused a visa.
You have the right not to pursue your GC visa. You are not obliged to attend an interview and pay $330 per person.
Affidavit is not mandatory. If CO decides that you can find a job in US and will not become public charge, you can...