Unless he has been so horribly abusive to the child that the government permanently terminates all of his parental rights, you won't be able to legally remove your child from the US if he doesn't want you to. And that permanent termination is not easy to get -- the courts will often allow the abuser to have supervised visits, or force the mother to periodically bring the child to prison to visit the father.
You haven't mentioned anything about him abusing the child, so getting the court to terminate his parental rights isn't feasible. However, you said he told "us" to get out of the house. I presume you are referring to yourself and your daughter? If he wants you both out of the house, maybe you can get him to agree to voluntarily terminate his parental rights (don't try to negotiate this on your own -- make sure this is done with a lawyer).
To convince of him that, show him the benefits from his perspective if he agrees: (1) He won't have to pay child support and (2) if you agree to leave the US he won't be liable for paying to support you according to the Affidavit of Support (I-864) he signed when sponsoring you for the green card (assuming you haven't already terminated the I-864 obligation by accumulating 40 quarters of work for Social Security).