How to Decouple Reasoning Processes in Strategy QA?

We tried to break down model performance by different reasoning types. And it's really hard to identify very specific reasoning types in strategy QA that are more challenging than others. Even for, you know, we just prompted with a few strategy questions and we try to evaluate it if it can come up with the right reasoning steps or to be more specific like the entity relation pairs. We were pretty shocked by this. It might also suggest that the reason for failing as strategy is not because these models cannot, like, they don't know how to answer these questions. They just fail to do that. Yeah. But I mean, we had a follow up work where we tried to decouple these two