How to Distinguish Between Difficulty and Success in an Exercise

I've now been working as a data scientist in educational technology for over four years. In that time I've thought a lot about various educational concepts within edtech, and I want to share some of what I've learnt.

In the first part of this two-part episode, I want to talk about what I call the Fundamental Duality of Educational Materials. The Fundamental Duality is that we use our content to measure our students / users (e.g. what they know), but we also use our users to measure our content (e.g. how difficult it is). This leads to a sort of chicken-and-egg problem, where all we see is the interaction of the users with the content, but from that single fact we have to somehow extract information about the two different interacting entities.

For example, suppose that a user gets a question wrong. This could mean one of a number of things:

1. Is the question difficult?
2. Does the user not know this area very well?

There is also a third possibility: Is this question faulty? i.e. did the user actually answer the question correctly, but it was marked as incorrect due to a bug in the system, or in the way the content was created?

Answering these questions is difficult because they are apparently all possible in this situation. This is an illustration of the Fundamental Duality.

In the episode, I make some mention of Item Response Theory (IRT), which is a method used in computerised adaptive testing (CAT) to handle this very issue. But IRT is quite difficult to explain to a lay audience, especially without the use of images, so I will focus on Elo and Glicko rating systems as examples of handling this duality.

Enjoy the episode.

***

You can support Education Bookcast and join the community forum by visiting https://www.buymeacoffee.com/edubookcast.