you can visit japan using refugee travel document or re entry travel documents, its really easy , it usually takes 7 days to get a visa, i used to travel to japan a lot using travel document
my advice yo you, don not mention you have a passport at the consulate general of japan, mostly likely they will ask you to renew your passport,
when you show them your travel document tell them you have no passport, you are asylee, do not show them your passport and you will be ok
what city are you now?