Psychometric Properties

Validity

The term validity refers to the extent to which a questionnaire or inventory measures what it claims to evaluate. Any instrument has many different validities, depending on the inventory's specific purpose and varying with the procedure for assessing its validity. Several of the most popular ways for studying the validity of an instrument are: analyzing its content, relating scores on the instrument to scores on a criterion measure, and investigating the particular psychological characteristics or constructs evaluated by the instrument.

The various procedures for assessing validity are useful to the extent that they increase understanding of what an instrument is measuring, so the scores will represent more accurate information on which decisions can be based.

This table shows different types of validity with corresponding definitions. SalesDialogue's products are then reviewed according to each validity measure.

Validity Measures and Definitions

Validity of SalesDialogue's Products

The construction of valid instruments is not a simple matter – it requires hours of pilot testing, data analysis and instrument refinement. Much time and effort has been expended in the development of our products.

i) Criterion Validity - In the research we have conducted, our products have demonstrated good Criterion Validity such that scores are related to demonstrated competencies and observable behaviors. The relationship is solid and positive, and it has been shown that higher scores are affiliated with superior sales performance. Specifically, there is a statistically significant difference between groups of superior sales performers and average sales people on all six of the sales components.

ii) Face Validity - Our products have appropriate Face Validity. An instrument is said to be face valid if it appears to be measuring what it claims to evaluate. The inventory items appear to be measuring relevant concepts, without of course, giving away how the items align with the scales.

iii) Content Validity - Content Validity refers to the degree that the instrument covers the important aspects of what is being measured. For example, to demonstrate the Content Validity of a sales instrument, the instrument would be given to a number of sales people to complete and they would subsequently be asked whether the report, as well as the individual inventory items, covered important aspects of sales and that the feedback report offered tips and suggestions to help them improve their performance. Based on feedback from three independent pilot studies conducted in 1999, 2000 and 2001, we are satisfied with the Content Validity of our products. We have also refined the instruments and the feedback reports to reflect participant input.

iv) Item Validity - Item Validity examines the appropriateness of each individual inventory item. The items are considered independently of the content to ensure they are culture-free and denote the same meaning for different individuals. As you know, inventory items should be clear and straightforward so respondents are able to accurately and meaningfully describe themselves on the instrument.

v) Predictive Validity - Since our products are primarily developmental, as opposed to evaluative, Predictive Validity is not of primary importance, provided that the instruments meet other validity criteria. To establish the Predictive Validity of our instruments, they would need to be administered to a random group of individuals whose behavior would be subsequently monitored over time. One would then look for correlations between their original thinking styles, as measured by SalesBOOSTER, and observable behaviors in the environment that were linked to these cognitive scales.

vi) Congruent Validity - Congruent Validity refers to the relationship of the scores on an instrument to another instrument thought to be in the same measurement arena. So, Congruent Validity relates to inter-test correlation. We have established the Congruent Validity of our instruments by comparing scores with other cognitive instruments. The observed inter-test correlation has ranged from 0.45 to 0.74. This is a good, solid inter-test correlation coefficient, given that the comparative instruments measure general thinking styles rather than cognitive sales competencies.

vii) Construct Validity - An estimation of Construct Validity is derived from the theory, reasoning, or logic on which the instrument is based. The designer of any instrument has a theoretical view that is expressed by the instrument and the measurement scales. In addition to reviewing these scales and the theoretical assumptions associated with them, it is useful to assess the practicality of the theory by looking at its ability to statistically discriminate between groups known to be more competent versus groups known to be less competent on the measure. At a very basic level, the score range of these two groups will serve to validate the power of the instrument and its Construct Validity. Our products have consistently demonstrated appropriate Construct Validity between groups of differing sales ability, as determined by candidates’ sales performance. The assumptions behind our scales are reasonable and consistent with cognitive behavioral models and effective thinking practices.

viii) Concurrent Validity - Concurrent Validity is the correlation between what the instrument measures and a criterion variable within the same time frame. Whereas Predictive Validity examines the instruments' track record over time, Concurrent Validity uses an immediate external measure in the here and now. Concurrent Validity is not always easy to measure, since immediate, objective feedback can be elusive. It is worthwhile, however, to identify concurrent criterion measures, as their immediacy can help identify instrument strengths and weaknesses. In reviewing our products, we compared individual component scale scores with those obtained on a multi-rater competency instrument where both instruments were completed in approximately the same timeframe. The Concurrent Validity of scores ranged from 0.55 to 0.69.

Reliability

While validity refers to the instrument measuring what it claims to evaluate, reliability refers to consistency over time. By definition, an instrument can be reliable without being valid, but it cannot be valid without being reliable. This is a very important distinction that the reader may want to think about for a couple of minutes.

It is important because on occasion, test publishers present the reliability of their instrument, leaving people to assume it is also valid. Some of the instruments we have seen in our psychology practice are indeed reliable (consistent over time) but they are not valid because they do not measure what they claim to evaluate. For example, suppose we administered a test of numerical ability to a group of sales applicants today and a week from today. We would see very similar results across the time span, indicating that the test was, indeed, very reliable. However, even though it has high reliability, it would not be a valid measure of sales ability because math proficiency is not likely related to sales skills.

Our products have demonstrated validity and are reliable measures over time. There is a second type of reliability related to internal consistency called split-half reliability that compares sub-sets of items within scales. Our products' split-half reliability coefficients are comparable to other well-designed instruments.


Printer friendly version