3min chapter

Data Skeptic cover image

StrategyQA and Big Bench

Data Skeptic

CHAPTER

Are the Instructions Really Confusional?

Many crowd workers don't really think a lot about the instructions and what the task creators want to achieve. They just quickly look at the examples and do the most similar thing they can come up with. So if the instructions include all kinds of biases, this will just naturally propagate to the behavior of the annotator. But for, you know, tense disambiguation or some of these NLP tasks, surely there's enough quality in the workers to get something out of it. Or am I too optimistic? No, no. It all depends on the nature of like the task.

00:00

Get the Snipd
podcast app

Unlock the knowledge in podcasts with the podcast player of the future.
App store bannerPlay store banner

AI-powered
podcast player

Listen to all your favourite podcasts with AI-powered features

Discover
highlights

Listen to the best highlights from the podcasts you love and dive into the full episode

Save any
moment

Hear something you like? Tap your headphones to save it with AI-generated key takeaways

Share
& Export

Send highlights to Twitter, WhatsApp or export them to Notion, Readwise & more

AI-powered
podcast player

Listen to all your favourite podcasts with AI-powered features

Discover
highlights

Listen to the best highlights from the podcasts you love and dive into the full episode