Don't throw the baby out with the bathwater

Don't throw the baby out with the bathwater is an idiomatic expression and a concept used to suggest an avoidable error in which something good is eliminated when trying to get rid of something bad, or in other words, rejecting the essential along with the inessential.