If you haven't filed tax, it is a mark against you claiming you didn't abandon residency during your lengthy period outside and yes it may be a problem. It is requirement of being a green card holder that you file taxes.(@Bob22 , may i respectfully suggest you read this too.)
My one is super weird. This is actually the second time I won the DV lottery (how many can say that???). I won way back in DV2004 and was in the US in the middle of 2004 but had to go home for personal reasons and never did get back stateside. For the life of me I cannot remember if I filed a return that year or if I even needed to (I barely earned any money that year). I definitely didn't file in 2005 although I didn't earn much that year again. After that my LPR expired so I don't think I was required to file after that but trying to get a clear answer from the IRs is like trying to find a mythical unicorn. Two different people over the phone couldn't help me. Do you guys think I should get a lawyer or just file next year and see what the IRS does?