| This document provides information regarding the use of Artificial Intelligence (AI) within our employee and candidate evaluation tools. |
| Various localities require disclosure regarding the use of AI in systems that affect employment decisions. However, whether you are located in a region that requires disclosure or not, it's a good idea to become familiar with how AI technology is being applied in the systems you utilize. |
|
|
| What is AI? |
| AI is an abbreviation for Artificial Intelligence. Artificial intelligence is the simulation of human intelligence processes by machines, especially computer systems. Specific applications of AI can include speech recognition, natural language processing and machine vision (Reference Burns et al., 2022). |
| Background |
| For several decades, AI tools have enabled useful capabilities, including voice recognition, facial recognition, and text analysis. Typically, a specialized machine learning model was trained and then applied to a specific task. Each new task required a new model. |
| Assessment companies, like HR Avatar, use these task-specific AI applications to support their products. For instance, to create transcripts of recorded video interviews, or to perform facial recognition in support of remote test proctoring, or to perform automated essay scoring. |
| More recently, however, new models – and new capabilities – have emerged. These new models, referred to as large language models (LLMs), enable the same model to perform many different tasks without being specifically trained for them. For instance, an LLM can be used to parse a resume or CV into a standardized set of components, or to summarize a long document into a few concise paragraphs. |
| While LLMs don’t do everything well, yet, the range of tasks that they do currently perform effectively includes the scoring of information provided by candidates, and the evaluation of candidates or employees against different jobs, or against a company’s values or culture. |
| Our Commitment to Ethical and Responsible use of AI |
| HR Avatar is committed to ensuring AI is applied in an ethical, fair, and responsible manner within our products. Our approach and the controls we have put in place are summarized in the following table. |
| Governance & Accountability |
All product features, including model selection, prompt construction, and results presentation are reviewed by our Science Team and approved by senior leadership prior to release. |
| Fairness & Bias Mitigation |
AI-derived scores are subject to the same periodic evaluations against fairness criteria as traditional measures. |
| Privacy & Data Protection |
AI-derived scores are subject to the same privacy protections as traditional measures. Additionally, no personally-identifiable information (PII) is sent to AI models. For more information, please see our Privacy Policy (Opens in New Tab). |
| Transparency & Explain-ability |
All Test-Takers can view a disclosure of how AI is utilized within their assessment that also provides a means of requesting additional information. All AI-generated output includes a clear rationale describing strengths, weaknesses, and improvement areas. |
| Security & Resilience |
AI-derived scores and the software used to create them are subject to the same security procedures that are applied to traditional measures. HR Avatar administers a robust security program in order to protect client and test taker information. |
| Use Policy & Guardrails |
Employers can disable calculation and use of AI-derived scores for higher order AI scoring and for all custom-built assessments. |
| Quality & Accuracy |
Periodically, when enough data is available, the Science Team will review the performance of AI-derived scores against traditional measures. Recommendations for optimizations or other changes will be reviewed with the President and implemented expeditiously. |
| Accessibility & Inclusion |
Any methods used to obtain inputs to AI-scores are subject to the same accessibility standards applied to traditional input collection. Currently, no special data collection methods are utilized. HR Avatar aggressively promotes accessibility within it's systems. For more information, please refer to our Accessibility Statement (Opens in New Tab). |
| Customer Controls |
For customized assessments and video interviews, the customer has full control over the use of AI and how it is applied. |
|
| How does AI work? |
| In general, AI systems work by ingesting large amounts of labeled training data, analyzing the data for correlations and patterns, and using these patterns to make predictions about future states (Burns et al., 2022). |
| AI programming focuses on acquiring data and creating algorithms for how to turn the data into actionable information. The algorithms provide computing devices with step-by-step instructions for how to complete a specific task (Burns et al., 2022). |
| Where We Use AI within Our Assessments |
| Essay Question Scoring: Essays written by candidates can be scored using AI. Most standard essay questions administered by HR Avatar include AI-powered scoring. Additionally, essay questions in custom tests created by clients can also include AI-powered scoring. When enabled, AI handles the scoring of content against a specific rubric, such as strength of argument, clarity/coherence, and grammar. The AI system also produces a short rationale describing the rationale behind the scores, including relative strengths and weaknesses, and avenues for improvement. |
| Video Interview Question Scoring and Analysis (live and recorded): AI is used to transcribe video interviews and analyze speech. The transcription can be treated as an essay and the content can be scored using AI in the same way as essay questions (see above). Additionally, we utilize BigTinCan, an AI-powered practice and coaching platform, for all voice analyses. VoiceVibes was developed using AI and machine learning algorithms, predicting how your voice is likely to be perceived by others. BigTinCan measures the paraverbal features of speech and assigns a score based on 20 different "vibes" that the speaker conveys. The vibes are derived from proprietary predictive models. Paraverbal components of speech include pitch, volume, pausing, and pace. Machine learning models were developed using 1800 samples, followed by independent validation of the predictions by a third party using 600 samples. The labels of the vibes were linguistically analyzed to ensure they had a common meaning to listeners resulting in an accuracy of prediction from 90 to 100% (Truninger, 2021). |
| Text-based File Upload Question Scoring: When the candidate is asked to upload a text-based file, such as an MS Word document, or a PDF, the text can be parsed and it can be treated as an essay and the content can be scored using AI in the same way as essay questions (see above). |
| Spoken Language Assessment Scoring: Spoken language assessments utilize AI for speech recognition (similar to how Apple's Siri and Amazon's Alexa use speech recognition). The generated speech text is then used to determine if the correct response was spoken. |
| Remote Test Proctoring: An optional feature, Remote Proctoring, utilizes AI for face to face comparisons, monitoring face direction, and scans for suspicious devices (cellphones). Our remote proctoring features come in different degrees (low stakes to high stakes) that can be changed depending on your specific requirements. |
| Automated Reference Checks: AI is used to transcribe recorded responses to reference check questions. The transcripts are presented to employers as part of the results report. Additionally, text-based responses, file upload responses, and audio/video responses which are transcribed can all be scored using AI in the same was as Essay Questions (see above). |
| Resume Parsing: Customers can request that a candidate or employee upload a resume as part of an assessment or reference check. AI is used to parse and summarize the uploaded resumes. |
| Higher-Order AI-Powered Scoring |
| In addition to the question-level AI scoring presented above, HR Avatar also enables higher-order evaluation of all candidate data against pre-specified information to determine how well a candidate or employee aligns with a company culture, the company’s core employee competencies, or a job description. |
| Company Culture Alignment: When requested by a customer, or when sufficient non-cognitive trait scores are available, AI is used to compare a candidate or employee with a pre-specified company culture and values description. AI is also used to parse company-provided information to summarize and establish their culture information. Company Culture Alignment scores are produced as part of an Evaluation Plan. They are also produced when sufficient trait scores and an organizational culture description are both available. |
| Corporate Key Employee Competencies Comparison: When requested by a customer, AI is used to evaluate a candidate or employee against a pre-specified set of core competencies. AI is also used to parse company-provided information to summarize and establish their core competency information. Employee Competencies Comparison scores are produced as part of an Evaluation Plan. |
| Job Description Match: When requested by a customer, AI is used to estimate the degree of match between a candidate or employee and any number of job descriptions that have been pre-specified by the company. AI is also used to parse company-provided information to summarize and establish the individual job descriptions. Job Description Match scores are produced as part of an Evaluation Plan. They are produced automatically when a job description is mapped to a specific assessment or to a specific test key. |
| Note: When performing comparisons with culture, employee competencies, and job descriptions, the AI System also provides the rationale behind the assigned score, including areas of relative strength and weakness and avenues for improvement. |
| Personality Summary: AI will summarize the personality and other non-cognitive traits measured within an assessment and infer areas of strength and challenges with regard to functioning as a member of a team. |
| Knowledge, Skills, and Abilities Summary: AI will summarize the knowledge, skills, and abilities presented by the competency scores within an assessment, and a resume, if available. Areas of high competence and challenges are presented in the context of utilizing these knowledge, skills, and abilities in a work environment. |
| Note: Personality Summaries and Knowledge, Skills, and Ability summaries are produced automatically when sufficient scores are available and no other high-order evaluations are performed. |
| Is AI-Powered Scoring Valid? |
| When it comes to validity, the answer depends on factors specific to a particular organization and a specific job. The way to determine validity is through a local research study that involves collection of evidence regarding performance and retention of candidates as well as their assessment scores. Correlation of scores versus actual performance and retention leads to an estimate of validity. These principles apply whether the assessment is scored using AI or a more traditional method. |
| HR Avatar performs validity studies with our clients at little or no cost. Please contact us for more information. |
| Is AI-Powered Scoring Reliable? |
| Reliability means that scores are consistent. That is, if Person A scores a 60 on a competency today, will they also score a 60 if they complete the same assessment tomorrow? |
| Reliability is important when evaluating the applicaiton of AI because AI results are non-deterministic. That is, they can vary (usually only slightly) between successive requests. |
| Studies of AI-powered essay scoring show that AI scoring is usually at least as consistent as human scoring, especially when a large number of essays need to be scored. Simply put, the computer doesn’t get tired, while human raters do experience fatigue. Other uses of AI scoring, such as evaluating a candidate against a job description, are too new, and studies are needed to evaluate reliability in these cases. In most cases they will work in the same way as essay or text-based AI scoring, by comparing the results of human evaluations with AI results. HR Avatar is committed to performing such studies as soon as sufficient data are available. |
| Want To Learn More? |
| To learn more about our use of AI, please contact us. |
| For more information on how we protect your data, please refer to our Privacy Policy. |
| References |
| Burns, E., Laskowski, N., & Tucci, L. (2022, February 23). What is Artificial Intelligence (AI)? definition, benefits and use cases. SearchEnterpriseAI. Retrieved April 1, 2022, AI-Artificial-Intelligence. |
| Truninger, M., Ruderman, M. N., Clerkin, C., Fernandez, K. C., & Cancro, D. (2021). Sounds like a leader: An ascription–actuality approach to examining leader emergence and effectiveness. The Leadership Quarterly, 32(5), 101420. |