The US should be your primary residence. While there is no hard and fast rule because they look at the totality of your circumstances, a good rule of thumb is to spend at least 6 months per year in the US. Otherwise, it's hard to claim that your primary residence is the US if you're spending more than half the time in Canada.
If you want to keep up the 4-months-per-year in the long term, you should aim to get US citizenship. Then once you have citizenship you can spend as little or as much time in the US. As the spouse of a US citizen, obtaining citizenship would require you to reside in the US for 3 years, of which at least 18 months you must be physically present in the US, and you should not spend more than 6 consecutive months* outside the US at any time in that 3-year span.
So you could make a plan to spend 7 or 8 months** per year in the US for 3 years, get US citizenship, then you can go back to spending 4 months a year in the US.
If I filed from Canada, would that affect travelling to the USA for family visits while the paperwork in being processed?
It could cause a problem. To reduce your risk of trouble at the border, have your spouse file the I-130 when you're visiting the US, with a consulate in Canada specified for Q22 of the I-130. Then on your next trip to the US, if hassled you can point out that you were previously in the US when the I-130 was filed, but you left the US and you'll leave again.
*the rule on 6 consecutive months is not absolute; but you'd have to present evidence to "justify" your 6+ month trip and hope they'll interpret it favorably, so don't let that happen if you don't have to.
**6 months per year is technically enough, but being on the borderline increases the risk of denial or complications so if you want a strong chance of a smooth and easy case you should give yourself some breathing room by spending 7-8 months per year in the US.