As long as your spouse satisfies the requirements of his status you should be fine. There is no requirement to spend a minimum amount of time in the US, unlike a greencard. I have been outside of the US for long periods of time and never had problems coming back.
If possible, please double check with an attorney.