What Constitutes Harm to Others?

Mill says that we each have an obligation to not injure the interest of one another. But as soon as a person's conduct negatively affects another person, society must become involved. Self-regarding acts are to be left the individual - which begs the question of whether any act is entirely self-regarding. Mill also contradicts himself when discussing what he means by definite damage or a definite risk of damage. He tolerates private gambling, for example, but wouldn't allow public gambling houses. And it's worth mentioning that Mill's liberty also didn't extend tall societies. Despotism, he said, is a legitimate mode of government in dealing with barbarians leading to contemporary critics labelling Mill