The Problem With Bans on Books and Libraries

In the US currently it's more hinging on ideological conflict. The people who want to ban books and libraries, they're not worried about somebody learning how to make a bomb. They're worried that ideology and conflict with their own will be transmitted to someone else. So like if you're removing these opportunities to encounter other beliefs that you may not agree with, then are we not inherently removing an individual's right to choose what they believe?