CE Corner

About CE

"CE Corner" is a continuing education article offered by the APA Office of CE in Psychology.

To earn CE credit, after you read this article, purchase the online exam at www.apa.org/ed/ce/resources/ce-corner.aspx.

Upon successful completion of the test — a score of 75 percent or higher — you can immediately print your CE certificate.

The test fee is $25 for members and $35 for nonmembers. The APA Office of CE in Psychology retains responsibility for the program. For more information, call (800) 374-2721.

While traditional neuropsychological tests can reliably spot cognitive deficits related to disorders, virtual environments add an important real-world element to the tests.


CE credits: 1

Learning objectives: After reading this article, CE candidates will be able to assess:

  1. Discuss how psychological tests and assessments are being replaced by virtual reality systems and video games.
  2. Describe the possible benefits of these changes.
  3. Discuss the challenges that must be addressed before they are more widely adopted.

For more information on earning CE credit for this article, go to www.apa.org/ed/ce/resources/ce-corner.aspx.

A child with attention-deficit hyperactivity disorder (ADHD) puts on a virtual reality (VR) headset and finds herself in a simulated classroom. She is asked to perform a task, such as watching the teacher draw a particular sequence of letters on the chalkboard. As she watches, the scenario becomes more distracting: Kids pass notes, a bus goes by outside the window, static crackles on the intercom.

Researchers behind the scenes are controlling these distractions, dialing them up and down, all in an effort to assess the child's performance and pinpoint the type of attentional problems she has.

Psychologists have been studying such virtual classrooms for two decades. Now, thanks to rapid improvements in technology, VR assessments are poised to move out of the lab and into the real world.

But VR isn't the only new technology shaking up the world of assessment. Researchers are beginning to use tools such as games and video-based assessments to measure function in people with dementia, gauge individuals' cravings for drugs and alcohol and evaluate job candidates' skills and abilities.

Many of these technologies have yet to be proven with rigorous research, and there are outstanding issues related to data security and privacy to be addressed. Still, experts are optimistic about the possibilities.

"There's a lot of research needed to validate these assessments, but the technologies are becoming so much easier to use, and the cost has come way down," says psychologist Albert "Skip" Rizzo, PhD, director for medical virtual reality at the University of Southern California Institute for Creative Technologies. In the last few years, for example, technological improvements have dramatically reduced the cost of VR systems from around $75,000 to just a few hundred dollars.

"All the barriers to adoption are slowly being eroded," says Rizzo, "and I think we're going to see this come into common practice soon."

Assessment goes virtual

Researchers and clinicians are overdue for some fresh assessment tools. Many of the neuropsychological tests in use today have been around for nearly a century, says Thomas Parsons, PhD, a professor of psychology and founder of the Computational Neuropsychology and Simulation Lab at the University of North Texas.

While traditional neuropsychological tests can reliably spot cognitive deficits related to disorders such as ADHD or dementia, virtual environments add an important real-world element to the tests. "What people really want to know is if a person is going to be able to return to work or to the classroom, and how well they will do," Parsons says.

Enter the virtual classroom. Researchers have shown that assessments from the virtual system correlate with findings from traditional paper-and-pencil measures of function in people with ADHD (Child Neuropsychology, Vol. 13, No. 4, 2007) as well as people with autism (Journal of Autism and Developmental Disorders, Vol. 46, No. 4, 2016). But the virtual test goes a step further, measuring elements of distraction such as how much a child moves her head or fidgets. If the child fails to notice the teacher writing letters on the board, the virtual test can help explain why.

"We can see whether the child is looking at the teacher but missing the target, or if they are looking out the window and missing it. Are they distracted, or is there a loss of focus? Those are two fundamentally different types of errors," Rizzo says.

Virtual environments can play a role in assessing other clinical conditions as well. Parsons, for example, has created a virtual grocery store simulation that runs on a standard laptop. Using the virtual environment, he can test memory and executive function in people with cognitive deficits, such as older adults with Alzheimer's disease.

In one study, he asked older and younger adults to navigate the virtual store to shop for items on a list, drop off a prescription at the pharmacy counter and remember to visit a coupon machine after a certain amount of time had passed. The participants' scores correlated with traditional paper-and-pencil neuropsychological assessments of memory. And as predicted, older adults performed more poorly than younger adults (Journal of Alzheimer's Disease, Vol. 59, No. 4, 2017).

Meanwhile, Patrick Bordnick, PhD, dean of the Tulane University School of Social Work, studies VR for assessing and treating people with substance use disorders. Much of that work has centered on smoking. For instance, Bordnick has confronted smokers with virtual environments designed to elicit nicotine cravings, such as office building courtyards where people take a smoke break and convenience stores where cigarettes are being sold. In one study of young adult smokers, Bordnick and colleagues found that participants responded strongly to those cues with increased cigarette cravings (The American Journal on Addictions, Vol. 17, No. 5, 2008).

In other work, Bordnick and colleagues have shown that VR environments are similarly useful for assessing a person's reactivity to alcohol cues (Addictive Behaviors, Vol. 33, No. 6, 2008) and cannabis cues (Journal of Psychoactive Drugs, Vol. 41, No. 2, 2009).

Beyond just inducing cravings, VR can help clinicians assess how well different coping skills work to prevent patients from acting on those urges, says Bordnick. "Virtual reality is a way to bring real-world contexts into the lab or clinic so we can understand relapse better and teach people skills that empower them not to use," he says.

Playing games

Other researchers are exploring the role that games can play in cognitive assessment. Among them is Adam Gazzaley, MD, PhD, a professor of neurology, physiology and psychiatry at the University of California, San Francisco, and founder of Neuroscape, a translational neuroscience center at the university.

Gazzaley has developed a variety of games to both assess and improve cognitive abilities. In a large project, his team is developing and testing a mobile cognitive-assessment tool called ACE (Adaptive Cognitive Evaluation). As players complete a series of games, the system assesses aspects of cognitive control including sustained attention, selective attention, working memory and task switching—abilities that serve as the foundation of all higher cognitive functions, Gazzaley says. ACE has the added strength of being able to change the level of difficulty based on people's performance and show the threshold of their abilities.

ACE may eventually be used to assess cognitive abilities as they relate to mental health conditions, Gazzaley adds, since many common psychological disorders—among them depression, anxiety, post-traumatic stress disorder and dementia—are associated with specific cognitive deficits. "There's great clinical applicability," he says.

Before ACE can hit the clinic, Gazzaley is undertaking randomized controlled trials to confirm that the tool works as designed—a critical step, he says. "I sometimes see [technology-based assessments] being used out there in the real world, and it's not always clear how valid they are," he adds. "Just because we now have the ability to collect all of these data doesn't mean they're valid and reliable enough to actually use in a meaningful way."

Indeed, while next-generation assessment tools hold a lot of potential, experts caution that these new tools have to be proven first. To do that, researchers must show that a test actually measures the skill or ability it purports to. One way to do that is by showing that the findings from the new tool mimic findings from more traditional assessments, such as paper-and-pencil tests. New tools must also be shown to give consistently reproducible results over time. If a test is reliable, the person taking it should receive a similar score each time he or she takes it, for example.

"Regardless of whether we're talking about technology-based or traditional assessments, there's always a risk that the scores aren't reliable or valid for the purpose at hand," says Dan Putka, PhD, principal staff scientist at the Human Resources Research Organization and a member of APA's Committee on Psychological Tests and Assessment. While the tools might be new, "the rules of evaluating assessment quality haven't really changed," he says.

Security issues must also be addressed before tech-based assessments become the norm, says Sarena Bhatia, an organizational psychologist and consultant for the London-based professional services firm Aon who has studied game-based assessment. Digital assessment apps and devices that gather and store a test taker's data must be able to protect those data. In addition, those who administer the assessments must find ways to protect the integrity of their tests. "If someone is taking an online test remotely, we have to figure out ways to make sure that the person we think we're testing is actually the one taking the test, and also that they're not sharing the content with other people who might be taking the same assessment," she says.

Despite such challenges, assessment experts believe the benefits of using technology to aid assessment will be worth the growing pains. Ultimately, Gazzaley says, technology might lead to new types of assessments that researchers haven't even dreamed of yet. "I'm not excited about replicating a paper-and-pencil task in an iPad version," he says. "This is about making a test that has substantial benefits beyond what could be accomplished with other media."

"Technologies just now being invented," he adds, "will be used to create entirely new assessment platforms."

Further resources

How to Assess Gaming-Induced Benefits on Attention and Working Memory
Mishra, J., et al. Games for Health Journal, 2012

A Virtual Classroom for Ecologically Valid Assessment of Attention-Deficit/Hyperactivity Disorder
Parsons, T.D., & Rizzo, A.A. "Virtual Reality Technologies for Health and Clinical Applications: Psychological and Neurocognitive Interventions," in press

Hiring for the Win: Game-Based Assessment in Employee Selection
Bhatia, S., & Ryan, A.M. "The Brave New World of eHRM 2.0," Dulebohn, J., & Stone, D. (Eds.) Information of Age Publishing, 2017

A Feasibility Study of Virtual Reality-Based Coping Skills Training for Nicotine Dependence
Bordnick, P.S., et al. Research on Social Work Practice, 2012

Is Clinical Virtual Reality Ready for Primetime?
Rizzo, A.S., & Koenig, S.T., Neuropsychology, 2017