That is a potential risk. I don't think USCIS will pursue the OP on this ground unless the ex-spouse can add more dirt. Previous marriage is not that really material fact because it does not affect citizehnship eligibility no matter what. But still one should tell truth.
So we see the following cases
(1) citizenship processs is very sound. Then by current law, one can not be denaturalzied no matte what the person does after citizenship (with eception of treason etc but that applies to native born citizn too)
(2) citizenship process is very flawed, then USCIS will pursue to de-natualize the person even after citzenship, the person is a model percet citizen
(3) citizenship process is not very sound but does have some minor issue such as the OP's not disclosing
previous marriage and divorce, then the USCIS may pursue to denaturalize the person if the person commit a serious crime. But I think this crime would be much more serious than threshhold set for deportation of permanent resident.
So the OP is fine but don't commit any serious crime. I am not saying he can commite a minor crime
but we are talking about citizenhsip here for that purpose, a minor crime may be OK to keep citizenhip but not a serious crime.
and for this purpose, I suggest the OP avoid looking for certain government job and anything
that requires a more strict backgroudn check. Not long ago, a lady from China was found having lied
about immigration benefit when doing a polygraph test as a trainee for FBI. USCIS decided to de-natualize her.