HR Avatar's tests employ principles developed and researched by many psychologists, universities, and government organizations over the decades since the First World War.
In the paragraphs below, we'll provide some background information about employment testing, as well as a non-technical, summary-level description of key principles and how we've incorporated them into our tests.
To be useful, an employment selection test must quantify one or more differences between candidates that are related to future performance and/or tenure on a specific job.
Many different factors contribute to the suitability of a person for a specific job, and most of them depend on the candidate, the company, and the job itself. Identifying the most relevant factors for a given job requires thoughtful data collection and analysis. Unfortunately, both companies and jobs vary with time, creating the problem of a moving target. Additionally, companies tend to adapt jobs to an individual's strengths over time. So, if Candidates A and B are both hired for the same job, but Candidate A is strong in one area and weak in another, while Candidate B is strong in the latter area, while weak in the former, the company often adapts responsibilities such that both Candidates A and B use their strengths to the company's advantage.
Next, there is the issue of the psychological measurement. While we can measure the weight or height of an object directly and precisely, we can only estimate psychological traits indirectly. Additionally, most psychological traits can only be ordinally measured, which means we can say that Candidate A has more of a specific trait than Candidate B, but there exist no units with which to measure the difference between A and B precisely. The numeric scoring systems used in test results are typically an approximation designed to help us compare one candidate to another efficiently.
The accumulated body of knowledge regarding psychological measurement is vast and highly technical. Instead of putting you to sleep, we'll highlight and discuss the core principles behind our approach.
This principle guides virtually all rational employment selection processes in use today. In order to select the right people, you have to understand what you are looking for. Psychologists use the term 'Job Analysis' to describe the activity of studying a job to identify it's most important tasks, as well as breaking down each task to determine the knowledge, skills, and abilities required to perform it. Job Analysis is a painstaking, tedious, and expensive process that is typically conducted for a specific job within a specific company.
Fortunately for us, the U.S. Department of Labor (DOL) recognized the value of more generalized analysis of common job types over 75 years ago. DOL initially published the Dictionary of Occupational Titles (DOT) in 1938. Sixty years later, in 1998, the DOT was replaced by the online Occupational Information Network (O*NET).
Since 1998, O*NET has been continually expanded and enhanced. As of 2014, it was on it's 18th version, with over 900 occupations analyzed to provide numeric data showing the importance of several hundred different knowledge, skills, and abilities to each occupation.
HR Avatar's tests are targeted to specific jobs. We use O*NET job analysis data to determine which traits to measure within each test and to set the relative importance of all measured traits. The test's overall score reflects a weighted average of individual trait scores based on O*NET.
A quick review of the marketplace indicates numerous test providers touting one type of test over another. For instance, those selling personality inventories suggest their tests predict how well an employee will 'Fit' with an organization, which affects tenure and lifetime achievement. Others may promote the importance of finding employees who have the knowledge and skills needed to hit the ground running, or who are intelligent enough to get up to speed quickly. Still others evaluate a candidate's past behavior to cite the potential for counter-productive behaviors.
By advocating one type of test over another, these test providers force the employer to choose between the different types, or mix and match between them. It can be frustrating for employers. But it doesn't have to be this way.
US Department of Labor Guideline
"To help ensure both efficiency and diversity in your workforce, apply the whole-person approach to assessment. Use a variety of assessment tools to obtain a comprehensive picture of the skills and capabilitiesof applicants and employees. This approach to assessment will help you make sure you don't miss out on some very qualified individuals who could enhance your organization's success."
Testing and Assessment: An Employer's Guide to Good Practices, O*Net/ U.S. Department of Labor
HR Avatar assessments combine several assessment types into a single job-related test for a specific job. Each includes measures of cognitive ability, personality, skills, knowledge, and past behaviors (bio-data). As a result employers don't need to worry that they've selected the wrong test or are missing out on valuable information. This approach is well-proven to improve test fairness and reliability (see sidebar).
This principle simply means that a measurement is only as good as it is repeatable. Obviously, a test that produces random results is useless.
The concept of test reliability has been studied extensively by many different psychologists. As a result, well regarded methods exist for evaluating the performance, from a reliability perspective, of both trait-level measurements and the individual items (questions) used to calculate those measurements. Proven techniques exist for dichotomous and non-dichotomous item types.
HR Avatar collects and analyzes a full suite of statistics at both the item and trait level. At the item level, these include response time, mean and standard deviation for all items, as well as point biserial correlation and discrimination index for dichotomous and quasi-dichotomous items. At the trait level, we measure mean total response time, mean, standard deviation, and standard error, as well as Kuder Richardson F20 internal consistency for traits using dichotomous items, and coefficient alpha for traits using non-dichotomous item types.
Professionally prepared and properly used employment tests can improve your employment decisions, but they can never make those decisions for you. You should always keep in mind that no test results are error free.
Compliance with this principle is the employer's responsibility. We only wanted to emphasize it here (and in our results reports) because it's so important. To make it easier for employers, we include targeted interview questions with our competency scores so that you can probe more deeply into issues cited by the test results.
This principle, often referred to as 'face validity' has been recognized for a long time.
In short, a candidate's level of cooperation and his or her reaction to the procedure itself is dependent on his or her judgment of how well the test appears to work, and whether what it seems to be measuring is related to the job he or she is applying for. At best, low face validity can turn off good candidates and at worst can result in legal complaints from candidates who feel the test was unfair or unrelated to the job.
Methods of enhancing face validity include using direct knowledge-based questions when measuring knowledge and skills, requiring work samples, and conducting assessment centers. Additionally, some tests use video-based items to get at various traits. Recently, the ability to conduct branching simulations for certain parts of the testing process has become feasible. While not all traits can be measured with highly face-valid items, many can.
To make the best possible impression on job candidates, HR Avatar's assessments employ simulations in which test-takers interact with animated characters (customers, co-workers and supervisors) to perform job-related tasks. This principle, often referred to as 'face validity' has been recognized for a long time. Typically, this animated simulation format is used in evaluating cognitive ability, knowledge, and skills within the tests.
For a test to be considered 'valid' in an employment selection context, it must satisfy two criteria:
- It must consistently (reliably) measure the traits it purports to measure.
- The traits it measures must be related to job performance in the job it is being used for.
It's important to keep this dual requirement in mind. When it comes to employment testing, it's not the test that is or is not valid, it's the use of the test for a given job that is either valid or not valid.
What's more, the degree to which a test measures a specific trait, and the degree to which that trait applies to a specific job are fairly subjective and difficult to measure. Therefore, there is really no such thing as 'absolutely valid.' Tests are either more or less valid than other tests, or other parts of the hiring evaluation process.
To ensure the test you are using is valid for the job you are hiring for, you must complete two steps. First, you must be sure the test measures the traits it says it measures. This is the responsibility of the test provider. HR Avatar spends considerable time and effort performing activities designed to ensure our tests measure what they say they measure.
Second, you must confirm that the traits measured by the test are related to job performance in the job you are hiring for. In some cases, this can require a thorough job analysis. HR Avatar provides this service to employers looking for this level of assurance (please contact us to learn more).
However, since HR Avatar's tests are based on the results of the extensive job analysis conducted by the U.S. Government via the Occupational Information Network (O*NET) we suggest you review the tasks identified for the relevant O*NET occupation code. If the O*NET tasks are sufficiently representative of the tasks your new employee(s) will perform, the test should be a valid measure for the job.
In our test description pages within HR Avatar we include a link to review the O*NET job description for each test. This enables you to easily and quickly review the tasks associated with the job to perform this important verification step.
Of course, please feel free to contact us anytime if we can help you with this process.