Many of the figures that historians write about died a long time ago. To gain insight into their lives, historians investigate sources that have in fact survived the test of time. But what would it be like to ask Isaac Newton a question in person? And what would you ask him? Isaac Newton has passed away a long time ago, but the modern-day Newtons are very much alive. Due to technological advancement, it has become incredibly easy to get in touch with prominent scientists, and simply ask them a question. They form an important source of knowledge and are first-hand witnesses to science in progress. And basically, time is short: if we don’t strike in time, we lose their testimonies forever.
In my project, I interview Presidents of the Psychometric Society, an institution that promotes psychometric research around the world. Psychometrics is the field of science that concerns itself with the measurement and prediction of human behavior through psychological testing instruments. Its most famous inventions are the mental ability tests, such as the SAT’s or the IQ-test. Over time, the psychological test has become an incredibly influential instrument that has decisive power over the course of people’s lives. Whether you get a job, or get admitted into a prestigious school, may just depend on that one test score. The people behind this are certainly no public figures, yet they have changed the structure of society. Through an oral history project, I try to gain insight into (historical) dilemmas psychometricians struggle with and their views on the course psychometrics has taken throughout the years.
Oral history is a branch within historical science that focuses on collecting personal testimonies of a certain event or period. Oral history studies often involve long interviews, in which the researchers focus on the memories and visions each participant individually. Part of the reason for conducting an oral history project is indeed to secure the testimonies of people who will at one point no longer be around and let those speak that would otherwise be forgotten by history. A second motivation for oral history research is to zoom in on the individual or idiosyncratic side of things. Rather than telling the ‘general’ story, oral history interviews provide material to show that there always multiple sides to one (hi)story. Sometimes these perspectives overlap, sometimes they’re diametrically opposed. By investigating the individual testimonies of in my case psychometricians, oral history can contribute to a more thorough, and specifically to a more detailed and nuanced account of psychometrics.
Just as I cannot interview Isaac Newton or Galileo Galilei if I were interested in the history of physics, several of the big names in the history of psychometrics are no longer around. Examples of these are Francis Galton, a famous pioneer of psychometric work with his ‘anthropometric laboratory, and Karl Pearson, famous for his work on the correlation coefficient, both at their intellectual peaks at the end of the 19th century. Or Charles Spearman, who invented ‘general intelligence’ and developed the first psychometric model in 1904. However, since psychometrics can be considered to be a young science, psychometricians that are alive today cover almost half of its history.
So what are some of psychometrics’ (historical) dilemmas that are interesting to uncover? There are many, but in this post I will focus on what psychometricians believe to be the key moments and developments within the field, the relation between psychology and psychometrics, and their visions on what lies ahead.
Defining moments
One of the questions I always ask is what the interviewee believes is the biggest scientific contribution that psychometrics has made. What development was key for the course psychometrics has taken over time? Jos ten Berge, a retired psychometrician residing in Groningen, believes one of those moments was the development of the notion of ‘measurement error’. In psychology, measurement is never straightforward, because psychological attributes are not directly observable. When we assign a test score to someone, we can never be completely sure this score accurately reflects what that test is supposed to measure. Moreover, the score might deviate each time due to unforeseen circumstances, such as whether the person is in a good mood while taking the test, or is in fact distracted. In psychometrics, this measurement error was quantified in the latent variable model, a model that has been the main focus of psychometrics over the last century or so.
Not only did the latent variable incorporate this measurement error, but it also provided a systematic way to measure what psychologists have tried to access for a long time: latent, or unobserved, variables. Different variations of this model were developed over time, and this has enabled psychometricians and psychologists to thoroughly analyze test data. For a long time, calculating the sum score was the only way to score a test, but these models have now given us ways to for example analyze individual items (are items too easy, too difficult, are they fair to both men and women, to people of different cultural backgrounds?) and estimate a person’s skill level of for example cognitive abilities. It has opened up to what David Thissen, psychometrician at the University of North Carolina, calls the ‘biggest contribution of psychometrics to society’: the wide availability of testing and with that the proper way to handle test data. The tools that have been developed, tools such as latent variable models, have contributed to proper and fair testing and measurement that goes beyond calculating a simple sum score.
An identity crisis
At the time when psychometrics originated in the beginning of the 20th century, psychometricians were usually trained as psychologists. They often designed techniques to solve methodological problems that occurred in psychological science (e.g. the latent variable model that was designed to solve a measurement problem). I was interested in how the Presidents have experienced the relation between psychology and psychometrics, and what they believe it should be. Does psychometrics play only a supporting role, aiding psychologists with their data analysis, or is psychometrics a stand-alone scientific discipline? And if it is a stand-alone discipline, what is its ultimate goal?
Over the years, psychometrics and psychology have become less intertwined and it is now common to believe that psychometrics has a different purpose from that of psychology, namely: the prediction (rather than explanation) of human behavior. According to this view, the psychometrician is a data analyst, specialized in item response data, from which predictions can be made concerning the cognitive abilities of students or mental states of patients. For such predictions, it is not necessary to know the actual psychological mechanisms behind this behavior: figuring those out would be up to the psychologist. Others argue that the psychometrician is ultimately a psychologist as well, and could and should be involved with theory construction. Paul De Boeck, a psychometrician currently working at Ohio State University, believes that psychometric tests are indeed themselves experiments: ‘I don’t see the difference between an experiment and a test. A test gives you data and an experiment gives you data, and because in a test one uses repeated measures, you can manipulate factors’. Because a test consists of a set of items, you can manipulate the items to create a certain state in the person taking the test, such as stereotype threat. Stereotype threat is the phenomenon where people fear they might conform to stereotypes about their group, such as girls being afraid they might score lower on mathematics tests than boys and because of that fear, end up scoring lower after all. By incorporating certain types of items that for example make people feel conscious of their membership in their group, stereotype threat can be induced in test takers. In this way, a test is not only used for measurement, but can actually be used to experiment on a psychological attribute. Paul De Boeck can thus be considered as a psychometrician of the second type: the psychometrician that is also still psychologist.
However, many believe there is a distinction between psychometrics and psychology among which Ivo Molenaar, retired psychometrician from Groningen university: ‘For many researchers and PhD students, psychometricians have a supportive role, but we also need people who take the subject further and who develop new tools’. Psychometricians may support psychologists in their analyses, and might even do psychological research on their own, but there is still need for psychometricians who create new tools, and they might do this without being interested in the application of these tools.
The future of psychometrics
An oral history project implies of course that the main content of the interview is in fact historical. However, these interviews are also great opportunities to ask how these prominent psychometricians think the future of psychometrics is going to take shape. We live in a day and age where behavioral data is all around, and psychometricians could play an important role in the big data era. A possible future is that psychometricians open up to different techniques coming from the fields of statistics, artificial intelligence, econometrics or data mining, and apply these techniques in their own work. On the other hand, some stress that the expertise psychometricians have should be preserved and protected from these external influences. Psychometrics’ expertise is item response data, and that’s what it should remain. Brain wave data or the increasing amount of consumption data lie far beyond the borders of psychometrics.
There is a definite tension between those who want to protect what psychometrics has achieved, and those that are reaching out to what the world has to offer them. Jos ten Berge stands on the protective side of the dimension and believes that psychometrics has had its prime. ‘There has been an explosion of psychometric methods and the future will show less new methods and more examining the methods that we have, rather than developing completely new ones.’ Psychometricians will always be needed, as human response data is not yet out of date, but we roughly know how to deal with it at this point, and further developments will be marginal in that sense. Others can imagine a psychometrics not only involved with testing data. Paul Holland, retired statistician, points out that ‘the task of psychometrics is bringing the basic concept of measurement with uncertainty associated with it […] to ever increasing complicated kinds of human responses.’ He wonders whether one day even interactions with robots could fall under that category. We’ll just have to wait and see what the future holds, but psychometrics might just take an interesting turn.
To put it mildly, the interview as a data source is not one of the psychometrician’s favorites. Interviews are not systematic enough, and answers to open questions can often not be quantified or generalized. Psychometricians prefer the safe haven of the multiple-choice question or continuous scales. The value of interview data therefore does not lie in its efficiency, conciseness, or its suitability for drawing conclusions about entire populations (core values of proper psychometric research). With an oral history project, it is not the general, but the idiosyncratic, that makes it interesting. The interviews paint a colorful picture of psychometrics as a diverse but divided field, and give us an exclusive peek at what’s on the mind of the psychometrician.