After more than a year of the covid-19 pandemic, millions of people have looking for employment in the United States. Artificial intelligence-based interview software claims to help employers search for applications and find the best people for the job. Companies specializing in this technology reported a sharp rise in business during a pandemic.
But as the demand for these technologies increases, so does it questions about their accuracy and reliability. In the latest episode of the MIT Technology Review podcast “In the machines we trust,” we tested the software of two companies specializing in AI job interviews, MyInterview i Curious thing. We found variations in predictions and results that matched the work that raised concerns about what exactly these algorithms were estimating.
MyInterview measures the traits discussed in the Big Five Personality Test, a psychometric assessment often used in the hiring process. These traits include openness, conscientiousness, extroversion, agreement, and emotional stability. A curious thing also measures personality traits, but instead of a Big Five, candidates are judged according to other metrics, such as humility and resilience.
Algorithms analyze the candidate’s responses to determine personality traits. MyInterview also compiles results that show how well the candidate matches the characteristics that are recognized as ideal for the job by hiring a manager.
To complete our tests, we first set up the software. We uploaded a fake job posting for an office administrator / researcher on both MyInterview and Curious Thing. We then constructed our ideal candidate by selecting personality-related traits when the system requested it.
At MyInterview, we selected features such as attention to detail and ranked them according to degree of importance. We also selected interview questions, which are displayed on the screen as the candidate records video answers. On the curious thing, we chose characteristics like humility, adaptability, and resilience.
One of us, Hilke, applied for the position and completed interviews for a role in MyInterview and Curious Thing.
Our candidate conducted a telephone interview with Curious Thing. She first did a regular job interview and got 8.5 out of 9 for her English language skills. In the second attempt, an automated examiner asked the same questions, and she answered each by reading a Wikipedia article on psychometry in German.
Still, Curious Thing awarded her a 6 out of 9 for her English language skills. She finished the interview again and got the same result.
Our candidate turned to MyInterview and repeated the experiment. She read the same article on Wikipedia aloud in German. The algorithm not only returned the personality assessment, but also predicted that our candidate would answer 73% for a fake job, putting her in the top half of all the candidates we asked to apply for.