What to consider when choosing psychometrics Part 3: Four key questions to ask your suppliers
23/09/2011, Author: Maria Gardner
Following on from ‘Why use them in the first place?’ and ‘Five key questions to ask yourself before buying psychometrics’, this week’s blog is all about what do you need to know from your suppliers before putting your money where your mouth is and purchasing your psychometric tools.
The reality is that once you’ve got past that first hurdle of figuring out what type of measure you want, you then have a veritable minefield of different suppliers to choose from – all of whom will tell you that their psychometrics are better than all the others!!! So to help you navigate through these tricky waters here are some key questions to ask when making contact:
1. How much the products cost?
Different test publishers have different pricing schedules and structures - with some more easy to navigate around that others! My advice here is that when speaking to a potential test publisher have in mind how many respondents you want to assess, both immediately and approximately over the forthcoming 12 months and then ask them to provide you a total price based on these numbers. This will be the simplest way of getting the information you need about costs.
Hidden (or not so hidden extra) costs to watch out for, that will be additional to the actual costs of the materials you want to purchase, are:
- Licence fee (this may be a one-off or annual)
- Set-up costs (with an online system you may expect some reasonable charge for set up but watch out for being charged just for having access to basic functionality)
- User Manuals
- Training (I’ll come to this in more detail shortly)
- Access to different norm groups
- Being expected to purchase large quantities of paper materials at a time
2. How do your psychometrics tools deter and manage cheating/ faking?
This is something customers always want to ask so I decided I might as well cover it here too. When asking this question ideally your potential supplier will be telling you the following things:
Ability tests
- Test items are randomised to reduce the possibility of cheat sheets (if online)
- The assessment includes clear deterrents as part of the introduction to the test, i.e. honesty contracts
- There are alternative materials available if you want to re-test (e.g. parallel forms, paper and pencil/online equivalents) and ‘verify’ functionality if online
Personality questionnaires
- Rigorous validation to eliminate items that appear to be subject to respondents demonstrating high levels of social desirability
- That there are ways of measuring social desirability (i.e. the extent to which an individual appears to respond in a way which presents them in a favourable light), e.g. inclusion of a Social Desirability scale
- The instructions for the instrument include ‘motivators’ which highlight the benefits of answering honestly
3. What training will we need to be able to use your products?
Different tools require different types or amounts of training. Here is a rundown of some of the most common forms:
Starting with the industry recognised ones, there is the Level A and Level B certification from the British Psychological Society (BPS). Level A provides a good foundation knowledge of psychometrics, as well as covering how to administer, score and interpret ability tests. Level B training specifically covers how to use personality questionnaires and is always much more focused on how to use one (or sometimes two) instruments – typically whichever personality questionnaire that test publisher distributes. This is because whilst there tend to be few differences between ability tests there can be significant differences between personality questionnaires with regards to their scoring and interpretation.
You can typically complete each course in about 3 days or a combined in Level A and B training course within 5 days.
(Note: the BPS’s qualifications in psychometrics are currently changing. However, any test publisher you contact should be able to provide you with an update on this)
Conversion training
This typically only applies to personality questionnaires, and when you have already received some training (e.g. Level B) but need to learn how to use that test publisher’s particular instrument. However, this should only really apply in situations where the instrument you are looking to use is quantifiably different to the personality questionnaire you have already received training on as part of your Level B. For example, you are trained to use a trait-based instrument, e.g. CAL, and you want to start using a type-based instrument, e.g. MBTI.
The length of conversion training is usually 1-2 days.
Product-specific training
Due to the time and cost commitment that can be involved in Level A and B training, some test publishers will offer courses that focus on their specific instruments only. Typically, this will involve all of the skills and knowledge required to use that tool, but does not go into the same depth or breadth that a Level A or B qualification would.
Product specific training typically lasts anywhere between half a day and two days.
4. What is the reliability and validity of your products?
We are slightly venturing into psychometric ‘geek land’ here if you are not naturally inclined to contemplate the statistics that underpin the development of an ability test or personality questionnaire. However, this question is really important as this information fundamentally tells you whether or not that instrument actually works!
Reliability: This provides information on whether or not the psychometric can and will measure the constructs you are interested in, i.e. ability, extraversion, conscientiousness, etc in a consistent manner. This ability to be consistent is measured using correlations, and if I’m starting to get too techie and loose you already, the bit you really want to know is can they show you a figure that is 0.7 or above. Ideally, a test publisher shouldn’t even be publishing an instrument that doesn’t meet 0.7 as a minimum, but there are a few who do so it’s always worth asking.
Validity: This is all about does the instrument measure what it’s supposed to measure, i.e. is the tool valid for the job? Now there isn’t a straight forward single number that a test publisher can show you with this one. Unfortunately! However, what you can do is be requesting to have a look at the test or questionnaire you are interested in and seeing whether the content ‘looks right’, i.e. does that numerical ability test consist of the need to carry out calculations and reason with numerical data? Ultimately, your common sense needs to kick in here.
If you fancy delving into validity in a bit more depth then you can ask about construct validity and criterion validity. The first is what work the test publisher did to check that the tool is really measuring the underlying construct – this is done by getting a group of people to complete their instrument and an established one (that measures the same thing) and correlating the results. In contrast, criterion validity looks to correlate performance on the ability test or personality questionnaire with an external measure, such as interview results or job performance. Test publishers have to work with existing clients to do this and it is primarily with the aim of seeing whether performance on the psychometric can predict performance in the job.
So, there you have four fairly hard hitting questions to ask any potential supplier of your ability tests and/or personality questionnaires. These range from the practical (how much does it cost) through to checking that the tool will actually work (reliability and validity).
Look out for Part 4, which is the final blog in this series and will be picking up key things to consider when buying online psychometrics. In particular, what do you need to know before investing in an online assessment system?
In the meantime, if you have any questions about what psychometrics would be right for you, or would like to discuss your concerns please don’t hesitate to contact the Criterion team on 01273 734000.
Consultant Occupational Psychologist
Follow me on Twitter: @mariagardner78
Follow Criterion Partnership on Twitter: @realcriterion
If you find this post offensive please contact us
Comments
No Comments
You must be logged in to leave comments. You can register here».
