You don't pay for the first year and a half or so. After that, you have to pay like anyone else (including illegals).
If you return after the end of your J1 to never return, the money is indeed gone. If you return later on (for good), your payments as J1 are credited toward your eligibility for medicare and social security.
Also, the US has treaties with some countries. If your home country has a goverment run retirement system AND has a treaty with the US, you might be able to get payments credited towards your pension back home.