Unfortunately, in a way , YES. H4 has an expiration date and at the time he renews his non-immigrant VISA, he may not include you because he is the primary sponsor. If you will get divorce , then H4 will be more doubtful to renew because H4 means dependent's visa.