Validity
Definition: Validity can be defined as the agreement
between a test score or measure and the quality it is believed to measure.
-
standards for educational and psychological testing (1985)
A. Identified
three types of validity evidence:
1.
Construct-related: (convergent/discriminant)
2.
Criterion-related: (concurrent and predictive)
3.
Content Related:
B. History
of Test Validity:
1) Content Validity.
2)
Predictive Validity.
3)
Construct Validity.
Constructs are broad
categories, derived from the common features shared by directly observable
behavioral variables.
II. Types of Validity:
A. Face Validity: Refers to what the test appears
superficially to measure, not to what the test actually measures.
B. Content-Description Procedures: Refers to the
extent that a test or measure provides an adequate representation of the
conceptual domain it is designed to cover.
1.
Procedures for investigating content validity:
Does the
test cover a representative sample of the specified skills and knowledge?
Is test
performance reasonably free from extraneous influences?
C. Criterion-Prediction Procedures: Refers to the
extent that a test performance corresponds to an accurate measure of interest.
Is the test effective in predicting an individual’s performance in specified
activities?
a. Predictive
Validity: Is the test effective in forecasting future behavior?
b. Concurrent
Validity: Is the test effective in predicting performance on a
simultaneously administered criterion measure.
1.
Procedures for investigating Criterion Validity:
Contrasted
groups refers to the comparison of one group of individuals to another
group of individuals that are assumed to differ in some related trait.
a.
Validity coefficient: correlation coefficient.
b.
Meta-Analysis: integrate findings from different studies.
effect size: magnitude of
difference between groups or magnitude of correlation across a number of
studies.
c. Regression Analysis
d. Criterion Contamination:
the extent to which test scores do not themselves influence any individual’s
criterion status.
D. Construct Identification Procedures: involves
assembling evidence about what the test really means.
1. Developmental
Changes:
a. age differentiation:
refers to differential performance of children with increases in chronological
age.
b. Regression Analysis:
determines the extent to which a best-fit line can be constructed to
demonstrate the relationship between X and Y.
Formula for Regression:
2.
Correlations with other tests:
3. Factor Analysis: a
refined statistical technique for analyzing the interrelationships of behavior.
4. Internal Consistency: Typically
compares the total test performance of two groups of individuals.
5.
Convergent and Discriminant validity:
III. Evaluating Validity Coefficients:
A. Magnitude
of a validity coefficient:
1.standard error of measurement
indicates the margin of error to be expected in the individual’s predicted
criterion score as a result of the imperfect validity of the test:
B. Limitations
of the validity coefficient:
1.
Look for changes in the cause of relationships:
2.
What does the criterion mean?
3.
Review the subject population in the validity sample
4.
Be sure that the sample size was adequate:
5.
Never confuse the criterion with the predictor
6.
Check for restricted range on criterion & predictor
7.
Review evidence for validity generalization
8.
Consider differential prediction
IV. Relationship between reliability and validity:
The validity of a test is
contingent upon its reliability and the psychometric properties of criterion
tests.
V. Test Validity and Decision Theory:
Decision theory allows us to
evaluate the effectiveness or utility of a test in certain circumstances
1.
Utility Estimates:
Allows us to estimate the degree to
which certain cutoff points are effective or valid.
Valid
Acceptance = Utility Estimate of +1.0
False
Acceptance = Utility Estimate of -1.0
Valid
Rejection = Utility Estimate of 0
False
Rejection = Utility Estimate of -.50
EU = (%VA * +1.0) + (%FA * -1.0) + (%VR * 0)+ (%FR * -.50)