It's not too difficult to get the logic here, even asylee and refugee (those who hasn't got GC) can go back to the persecution country if there's reasonal execuse(it sure will be examined, but you will be fine as far as you have reasonal execuse). LPR should not even be considered here, you absoultely have the right go back visit your family.