Isnt It an Adhominum Fallacy to Think Otherwise?

The idea that our thoughts are determined by brain structure is an adhominum fallacy. The fact that there are so many brilliant people that believe such varying things says something about how the facts under determine the interpretation of it. And we just shouldn't really be doing philosophy because we can't know this stuff, and bababa. But still, it seems more often we'll say, well, i have to live with some philosophy or whatever. That's where a whole tradition of philosophical quietism comes in.