How to Judge Expertise in Philosophy

At a talk he gave at TAM 8, Massimo argued that non-experts in a field aren't qualified to reject an expert consensus, such as that on anthropogenic climate change. Most recently, he has taken Jerry Coyne to task for making a philosophical argument without having the necessary expertise. This raises a number of questions: Are there fields that have no experts, or that have pretend experts? If there is a lot of disagreement among experts on a topic, should we take any individual expert's opinion less seriously? How much consensus is required before a non-expert should say, "OK, looks like this question really is settled"?

Perhaps noted expert George Carlin had it right when he said: "I have as much authority as the pope, I just don't have as many people who believe it."