What HR Leaders Should Know About Assessment Validity and Reliability
At this stage, the concept of assessment is no longer new to HR Professionals. Its advantages are too numerous to be ignored in the talent acquisition process. This is because it provides the most objective way to evaluate talent and also reveals which candidate is most suitable for the job. However, for it to be able to do this, assessment needs to possess some properties and qualities before it can be termed dependable, useful and generally acceptable.
The two most important properties of assessment are reliability and validity. These properties tell whether Assessment can do what it was meant to do effectively.
Let’s examine what reliability and validity are in Assessment.
What Is Reliability in Assessment?
The degree to which assessment scores are accurate and free of measurement error is referred to as reliability.
You can think of reliability in terms of score consistency. Will the scores be similar if you test a candidate today and again tomorrow? Are you going to see similar scores on the same assessment if you test one population of candidates and another population of candidates?
These questions address the issue of whether candidates consistently perform on all of the items in a given assessment.
Reliable assessment results will assure you that similar or repeated assessments will produce consistent results. This puts you in a better position to make broad statements about a candidate’s level of capability, especially when using assessment results to determine a candidate’s suitability for a role.
In a situation where the candidate that takes a test does not get similar results, that does not automatically mean that the test is not reliable. Some factors that may be responsible include:
- Environmental Factors: Some environmental factors that may lead to a candidate not getting the same results include lighting of the test room, the ventilation, noise, the test administrator etc.
- Multiple Assessors: When there are multiple assessors, there is bound to be difference in scores, especially if the scoring of the assessment is dependent on the assessor’s judgement, such as essays etc. Assessors will tend to give ratings based on their own background and level of exposure.
- Candidate’s Psychological State: The state that a candidate is in before taking an assessment is very important to the outcome. If during one assessment, the candidate is in the right frame of mind mentally, and during another assessment, he is not, the assessment scores will most likely be different.
What is Validity in Assessment?
An assessment is said to be valid if it measures what it is designed to measure. It assesses the degree to which assessment tests the competencies it claims to test. For example, if a personality assessment is deployed to measure a candidate’s personality traits and decision-making process, the assessment questions should not contain questions that test their cognitive ability.
Validity gives meaning to the assessment results. There is a link between assessment performance and job performance, according to validity evidence. It can tell you what you can conclude or predict about a candidate based on their assessment results. If an assessment has been shown to be a valid predictor of job performance, you can conclude that candidates who got good results after the assessment exercise are more likely to perform well on the job than candidates who got poor results, all things being equal. Validity also reveals how useful or relevant an assessment tool is.
How to Achieve Reliability and Validity in Assessment
To achieve reliability and validity in your assessment, you must follow some carefully laid out processes. These processes are:
1. Conducting a Proper Job Analysis
The first step in achieving a valid and reliable assessment is by creating a proper job analysis. This involves gathering all the necessary information about the job. This includes skills, attributes, responsibilities, roles etc.
It serves as a guide for creating a proper job description which will also guide the development of assessment tools.
You should also include the subject matter experts in your organisation when carrying out your job analysis. They include the business directors and other senior managers who know about the role and the competencies you need to assess. Only when you have done a proper job analysis can you identify the right competencies for the role(s).
2. The Setting of Assessment Methodology
Here, you sit together with your SMEs and determine the assessment methodology. You will also determine the instruments to use for the competencies you identified.
3. Sourcing of Assessment Instruments
These instruments will be dependent upon the competencies you want to assess for the role. The assessment questions should be of a high standard and must be original. This means that you should not source questions from the internet. You must also avoid plagiarism when creating the content of your assessment. Instead, you should engage subject matter experts in your organisation who can develop the right questions. You should provide the SMEs with the guidelines they need when authoring the assessment instruments. An easier method is to partner with testing organisations like Workforce Group who have access to these questions.
4. Testing of Developed Instruments
After developing the tools, you need to review them to ensure that the assessment can effectively test the relevant competencies of the candidates. Here, the assessment instruments are reviewed to ensure there is no form of plagiarism and to also ascertain they align with the assessment methodology. You should also check for clarity and flow. Next, you will check for language and technical correctness. Finally, you should do an audit of these instruments. You could create a sample exercise with candidates in your department and see how they fare. This test will help you determine the test format that fits the bill the most, the appropriate timing to use based on the test questions, the level of difficulty, the medium to administer the test. This can be quite tedious and rigorous. Again, if you feel that you can’t do this effectively in-house, you can outsource it to reputable testing organisations like Workforce Group who can deliver tested assessment instruments.
5. Administration of the Assessment
Assessment administration plays a huge role in ensuring assessment is valid and reliable. It starts from the assessment administrator who must have knowledge of how the assessment will go. It involves scheduling and setting up the assessment environment, taking of the assessment by your candidates, monitoring of the assessment and collection of the results.
As far as the assessment environment is concerned, some of the questions you should ask include:
- How conducive is the assessment venue?
- Is the venue too large or too small for the invited candidates?
- How is the air quality of the environment?
- Are the chairs and tables suitable for the task?
- Are there any features contained in the venue that can distract the candidates?
For virtual assessments, some areas of concern include checking whether the platform can host exercises for multiple candidates, your remote proctoring solutions, selecting the right assessment questions and the proper scheduling of candidates.
Proctoring is another important aspect of administering the test. It also helps to add credibility to your assessment. When your test is well proctored, any form of cheating is avoided, and you are able to create a level-playing field for everyone taking the assessment. This also means your test results will be much closer to what you want to achieve.
Proper administration of the test is important because you do not want to go through the rigours of creating test instruments only to later realise that the level of validity and reliability of your assessment was truncated by poor administration of the test.
Other things to note include ensuring that the test environment is favourable to the candidates, the assessors can easily interpret the assessment scoring system, administrators know the instructions and carry them out in a proper manner. The last factor is dependent on the candidate. They need to be in the right frame of mind to ensure that assessment results are consistent.
For 18 years, Workforce Group has been deploying valid and reliable assessment tools to test for important job-related competencies in candidates. These tools have helped to predict how well a candidate will perform on the job.
Ready to deploy the assessment in your organisation? Schedule a consultation today .
Akindele Afolabi
Director, Workforce Resourcing