Measuring Skills with Kian Katanforoosh

How do we measure skills in an age of AI? That question has an effect on everything from hiring to productive teamwork. Join Kian Katanforoosh, founder and CEO of Workera, and Ben Lorica for a discussion of how we can use AI to assess skills more effectively. How do we get beyond pass/fail exams to true measures of a person’s ability?

Points of Interest

• 0:28: Can you give a sense of how big the market for skills verification is?
• 0:42: It’s extremely large. Anything that touches skills data is on the rise. When you extrapolate university admissions to someone’s career, you realize that there are many times when they need to validate their skills.
• 1:59: Roughly what’s the breakdown between B2B and B2C?
• 2:04: Workera is exclusively B2B and federal. However, there are also assessments focused on B2C. Workera has free assessments for consumers.
• 3:00: Five years ago, there were tech companies working on skill assessment. What were prior solutions before the rise of generative AI?
• 3:27: Historically, assessments have been used for summative purposes. Pass/fail, high stakes, the goal is to admit or reject you. We provided the use of assessments for people to know where they stand, compare themselves to the market, and decide what to study next. That takes different technology.
• 4:50: Generative AI became much more prominent with the rise of ChatGPT. What changed?
• 5:09: Skills change faster than ever. You need to update skills much more frequently. The half-life of skills used to be over 10 years. Today, it’s estimated to be around 2.5 years in the digital area. Writing a quiz is easy. Writing a good assessment is extremely hard. Validity is a concept showing that what you intend to measure is what you are measuring. AI can help.
• 6:39: AI can help with modeling the competencies you want to measure.
• 6:57: AI can help streamline the creation of an assessment.
• 7:22: AI can help test the assessment with synthetic users.
• 7:42: AI can help with monitoring postassessment. There are a lot of things that can go wrong.
• 8:25: Five years ago in program, people used tests to filter people out. That has changed; people will use coding assistants on the job. Why shouldn’t I be able to use a coding assistant when I’m doing an assessment?
• 9:16: You should be able to use it. The assessment has to change. The previous generation of assessments focused on syntax. Do you care if you forgot a semicolon? Assessments should focus on other cognitive levels, such as analyzing and synthesizing information.
• 10:06: Because of generative models, it’s become easier to build an impressive prototype. Evaluation is the hard point. Assessment is all about evaluation, so the bar is much higher for you.
• 10:48: Absolutely. We have a study that calculates the number of skills needed to prototype versus deploy AI. You need about 1,000 skills to prototype AI. You need about 10,000 skills for production AI.
• 12:39: If I want to do skills assessment on an unfamiliar workflow, say full stack web development, what’s your process for onboarding?
• 13:17: We have one agent that’s responsible for competency modeling. You can have a subject-matter expert (SME) share a job description or task analysis or job architecture. We take that information and granularize the tasks worth measuring. At that point, there’s a human in the loop.
• 14:27: Where does AI help? What does the AI need? What would you like to see from people using your tool?
• 15:04: Language models have been trained on pretty much everything online. You can get a pretty good answer from AI. The SME takes that from 80% to 100%. Now, there are issues with that process. We separate the core catalog of skills from the custom catalog, where customers create custom assessments. A standardized assessment lets you benchmark against other people or companies.
• 16:32: If you take a custom assessment, it’s highly relevant to your needs, even though comparisons aren’t possible.
• 16:41: It’s obviously anonymized, right?