DO NOT DO IT if you only have asylum. If you have a green card, it's a "bit less risky" depending on what you asylum claim is based. But there is always the possibility this could make your citizenship interview tricky. There are already several threads/posts about this.
Again, if you only have asylum just don't do it. Get an RTD and visa if necessary or wait until you have a green card.