When technology – and time – overtake research

For the past three decades, I have been a psychological test subject. Not in a creepy NSA-and-tin-foil-hat kind of way, but as part of a long-term study designed to understand how mental abilities change as people age. Yet as time affects cognition, technology is encroaching on the study itself.

Dr. K. Warner Schaie

Long-term researcher Dr. K. Warner Schaie

The Seattle Longitudinal Study is something of a research marvel. Started in 1956 by Dr. K. Warner Schaie with randomly selected volunteers from Group Health Cooperative and now based at the University of Washington, it’s persisted for more than 50 years. Along the way, it has grown to encompass three generations of participants (“Luke … I am your grandfather”) and led to more than 300 pieces in scientific publications.

One of the most widely cited findings outside of academic journals is likely the “use it or lose it” advice when it comes to maintaining mental abilities (thankfully, study results also suggest training can reverse some cognitive decline).

I joined the study in 1985 and have been tested roughly every seven years since (plus several supplemental sessions), including two two-hour periods over the same week, with questionnaire homework in between, as part of this year’s “Wave 10″ testing.

I’m delighted to report that I think I did better on some short-term memory and other tests than I did a half-dozen years ago (and your own perception of how you performed is measured, too).

Yes, a specific phone (but far older than the study)

Yes, a specific phone (but far older than the study)

But this year, I also noticed a problem with some of the questions. They were becoming slightly anachronistic. Not in a pop culture sense, where “zombies” might refer to 1968’s Night of the Living Dead instead of 2010’s The Walking Dead. Instead, the problem occurred when the questions applied technology to everyday life:

  1. Determine which long-distance plan would be best for you on a phone company application form, including options such as Unlimited and Budget from providers MCI, AT&T and others;
  2. Respond to whether you have trouble reading “ordinary newsprint;”
  3. Calculate when a phone call would be cheaper, from a grid for telephone rates, that shows at what times Weekday, Night and Weekend pricing is in effect.

The first is not likely a question anyone who has only owned a mobile phone would likely have encountered as part of “Everyday Problems,” the name of the test (“Isn’t all U.S. calling free?”), but it’s easy enough to calculate even if some of the carriers and options are now more abstract than actual. The second begs the question of how many of the youngest test participants have ever read a daily newspaper.

But the third, that third is a nightmare: If you don’t have the right frame of reference and know that Weekend calling rates were always cheapest, you would have no idea how to answer it. And that question is from a test developed in 1996, less than a generation ago.

Now, this is a handful of questions out of hours of testing (and I may have gotten some of the details wrong in my longer-term remembering). Yet these are cases where it seems technology, especially recently, has outrun the test.

How do I work this thing?

How do I work this thing?

Isn’t that a problem for, uh, younger participants?

“The testers explain that some of the tests were designed in 1956 or earlier and that due to the nature of longitudinal studies, the tests need to remain the same,” responded SLS Project Coordinator Robin Dunlap. While noting that the major obsolete item is the long distance question that caught my eye, Dunlap added that even with situations that are no longer common, “the objective is to look at reasoning and other abilities which are not affected by the ‘datedness’ of the questions.”

It’s not just the wording, she said, that’s foreign to Young People These Days: “The challenges to the study are due to the changing nature of education. Younger participants have grown up using calculators, so they tend not to score as well on numeric tests.”

“Another issue is that cursive writing is no longer taught, so the ‘Test of Behavioral Rigidity’ (where you copy a paragraph as-is, then reverse the upper and lower case letters) can be a challenge.” And, of course, no coaching is allowed.

Time- and tech-related changes to how materials used in a scientific study are perceived might seem to potentially cause confusion of more significant impact than, say, a Lost in Space rerun in which “futuristic” 1997 is depicted with reel-to-reel tape recorders and rotary phones. Dunlap was reassuring. “Some outdating is inevitable, but the tests have actually remained quite relevant over the years.”

Which suits me, as I plan to keep up on the parallel advances of technology while fully participating in the study and reaping the benefits of its ongoing research findings. After all, I need to keep my brain pliable, fresh, and tasty. For the inevitable zombie apocalypse.

Frank Catalano (@FrankCatalano) is a strategist, author and veteran analyst of digital education and consumer technologies whose regular GeekWire columns take a practical nerd’s approach to tech. See the column archive. He began participating when he was a cognitively plastic health and science reporter, and hopes his mental progression is less Flowers for Algernon and more The Matrix.

  • Kevin

    I also have been a subject of the SLS, since about 1989. I would argue, though, that since in this study a subject is not compared directly against other subjects but only against his/her past performance, anachronistic questions have little effect: it may take longer for younger generations to figure them out, but that will be true throughout their entire life so the comparison is still apt.

    • http://www.intrinsicstrategy.com/ FrankCatalano

      Agreed. I was careful not to imply that this invalidates the study. It’s a very few questions out of hundreds. But it potentially could make comparisons across cohorts (groups of earlier, older participants against later, younger participants) a bit more challenging, since these test questions are a product of the time in which they were created. Indeed, the SLS uses a whole bunch of different tests of varying vintages and likely has to allow for a number of factors when analyzing and interpreting the results.

  • Kishari Sing

    Frank, thanks for the post–what an interesting study, similar to the “Up” documentary series out of the UK. I’m looking forward to following this research. Also, you’ve given me my new anti-aging rallying cry:”Less Flowers for Algernon and more The Matrix”! It would also be a great name for a Meetup group. Only because it’s too long a name for a rock band. :)

    • http://www.intrinsicstrategy.com/ FrankCatalano

      I recall that documentary series. And yes, I think there are similarities. I actually enjoy being a part of the Seattle Longitudinal Study and look forward to the testing sessions. Not only does it give me a way to “benchmark” my cognitive abilities, but it’s also a way to donate my brain to science — while I’m still alive and using it.

      • Kishari Sing

        Benchmarking your own brain would be fascinating! Separately, I’ve always wondered at what age, life changes or mental milestone makes some people resistant to learning new things, especially technology. Some people are super resistant at middle age, yet some seniors embrace it.

        More to ponder. Anyhow, we’re both signed up for the next EdTech Meetup in December, hopefully I will be able to meet you in person.