If I may give my opinion, I dont think it will be a problem if its only 4-6 weeks/year. heck, i used to travel that much outside the USA for pleasure when I was a GC holder.
Also, in the application for citizenship, they make you list down all the dates you have been out of the US, e.g. when...