TMCnet News
Defense Automated Neurobehavioral Assessment (DANA)-Psychometric Properties of a New Field-Deployable Neurocognitive Assessment Tool [Military Medicine](Military Medicine Via Acquire Media NewsEdge) ABSTRACT The Defense Automated Neurobehavioral Assessment (DANA) is a new neurocognitive assessment tool that includes a library of standardized cognitive and psychological assessments, with three versions that range from a brief 5-minute screen to a 45-minute complete assessment. DANA is written using the Android open-source operating system and is suitable for multiple mobile platforms. This article presents testing of DANA by 224 active duty U.S. service members in five operationally relevant environments (desert, jungle, mountain, arctic, and shipboard). DANA was found to be a reliable instrument and compared favorably to other computer-based neurocognitive assessments. Implications for using DANA in far-forward military settings are discussed. INTRODUCTION In January 2009, the U.S. Navy Bureau of Medicine and Surgery identified a need to enhance existing battlefield concussion assessment and requested the development of a durable, portable, and field-hardened neurocognitive assessment tool (NCAT) to provide a practical means to conduct neurocognitive and psychological assessment in field deployment settings. The purpose of combining neurocognitive and psychological assessment was to permit more comprehensive evaluation of the broad range of problems that may be encountered during combat deployment. This article describes the resulting NCAT, Defense Automated Neurobehavioral Assessment (DANA); DANA's psychometric properties based on assessment of 224 active duty U.S. service members under challenging field conditions; and presents comparisons to published NCAT data. DANA consists of three test batteries of different durations and compositions designed for increasingly detailed assessment (Table I). The three batteries include (1) DANA Rapid, a 5-minute battery of three basic reaction-time measures; (2) DANA Brief, a 15-minute test that includes DANA Rapid plus additional neurocognitive tests as well as psychological screening tools for post-traumatic stress disorder (PTSD), depression, and insomnia; and (3) DANA Standard, a 45-minute more comprehensive battery of neurocognitive and psychological tests. This hierarchical set of batteries is designed to facilitate health care providers' access to standardized, reliable, and valid objective and subjective measures. DANA's portability, multiple test batteries, and user-friendly interface enable its use by a wide range of health care providers, from frontline medics/corpsmen to licensed health care professionals. Establishing reliability and feasibility of this platform in a military population is necessary before attempting clinical validation and utilization. The eventual goal of DANA is to assist clinicians to (a) make rapid and accurate assessment of cognitive and psychological dysfunction secondary to brain injury and/or the psychological wounds and stressors of war, (b) facilitate referral to treatment for wounded service members, (c) monitor recovery, and (d) aid in return-to-duty determination. Thus, DANA is intended to enhance military capability and better ensure a healthy fighting force. METHODS DANA Platform DANA is a Java-based mobile application that runs on an Android operating system. The primary advantages of Android are that it is open source, open license (Apache 2.0), well supported and based on a Linux kernel, which is nearly ubiquitous. Java has the advantage of being a high-level, class-based, object-oriented language designed as a "write once, run anywhere" solution and thus is portable across a wide range of devices and desktops. DANA, therefore, can run on any Android mobile device and can be used with a stylus or touch screen. Based on the Navy Bureau of Medicine and Surgery, requirements for MIL-SPEC commercial-off-the-shelf hardware, we conducted a comprehensive trade study and selected the Trimble Nomad, the military-grade-hardened handheld computer used in the current study. A Tektronix 100 MHz analog to digital (ADC) oscilloscope was used to test the input variability of device hardware and device software that could contribute to the overall response times. A push action switch was connected to the ADC, which was then used as the input stylus on the Nomad to measure RT. The interval between two inputs as recorded by DANA and by the ADC was compared over 10 trials. The average difference was 6.8 milliseconds with a standard deviation (SD) of 3.7 milliseconds. By comparison, the input variability with a Microsoftwindows personal computer can range from 4-25 milliseconds.1,2 DANA Test Battery Selection of the neurocognitive and psychological tests included in DANA was established by a tri-service, Veterans Administration, and civilian scientific advisory board that included military and civilian neuropsychologists and psychologists, neurologists, and corpsmen. All tests included in DANA's test batteries meet the requirements of the American Psychological Association's standard for tests and measurements and all tests are in the public domain. Eight cognitive tests and seven psychological questionnaires were selected (Table AI) and are divided into three test batteries, as shown in Table I. Tests were selected based upon their potential to address specific deployment-related concerns, such as concussion and combat distress or exhaustion. Although all tests utilized have an extensive literature regarding their reliability and validity, they have not been reported in this specific configuration nor implemented for service members in this manner. The advisory board also contributed to and provided feedback on the user interface design to ensure ease of use by multiple levels of caregivers including the corpsmen, general medical officers, and neuropsychologists. Participants and Procedure To evaluate the deployment feasibility of DANA, we recruited 224 active duty service members comprising 40 or more active duty military personnel in each of 5 diverse operational environments. No subjects were excluded, since all service members were fit for duty, not undergoing any disability evaluation, and thus assumed to be healthy. The purpose of assessing service members across diverse environmental conditions was to show the robustness of the hardware and software administration under different operational tempos and to identify any environmental concerns with the reliability of the instrument. - Arctic (Thule Air Force Base-Greenland in the winter) - Jungle (U.S. Marine Corps Jungle Warfare Training Center-Okinawa, Japan, in the summer) - Altitude (U.S. Marine Corps Mountain Warfare Training Center-Bridgeport, CA, approximately 3,000 m) - Desert (U.S. Marine Corps Desert Warfare Training Center-Twentynine Palms, CA, in the summer) - Shipboard (USS George Washington during high seas in the Western Pacific) Device performance (e.g., battery life, display characteristics) was evaluated under the specific environmental conditions (e.g., humidity, temperature) through a minimum of 12-hour exposure. The only instrumentation reliability issue was a screen refresh rate delay in the Go/No-Go (GNG) test in the arctic environment. Because this screen rendering delay would affect test results, the rendering process software was redesigned, which successfullymitigated the delay. The research protocol was approved by the AnthroTronix Institutional Review Board, theVA Institutional ReviewBoard, and received a Department of the Navy Human Research Protections Program review. A letter was obtained from the commanding officer of each test facility and all subjects signed an informed consent document to participate in testing. On Day 1, each subject was tested on all three batteries, the DANA Rapid, Brief, and Standard. Subjects returned on Day 2 to repeat the sequence of batteries resulting in the following protocol: - Day 1 (approximately 120 minutes)-Consent Process, DANA Rapid, DANA Brief, DANA Standard - Day 2 (approximately 40 minutes)-DANA Rapid, DANA Brief, DANA Standard (cognitive tests only) (The above times include 5-minute breaks between each battery.) A research team of clinical psychologists and technical staffadministered testing. Participants were instructed to hold the stylus about 1-cm above the screen, and to respond as rapidly and accurately as possible. All other instructions were embedded within the tests. To minimize learning and practice effects, test stimuli are generated at random and each test has practice trials before the actual test trials. Parameters of the final version of DANA's individual subtests are described in Table AII including each subtest's stimulus presentation duration, response interval, and interstimulus interval. Data were analyzed in SPSSv20 for descriptive statistics, split-half reliability, test-retest reliability, and cross-test correlations. For internal consistency, we examined split-half correlations of the first and second half set of trials for the first administration of each test on each day. To evaluate test- retest reliability across administrations, we calculated intraclass correlation coefficients (ICCs)3 that have been used to evaluate reliability for other health status instruments.4 Because of multiple analyses, significance levels were set to between p <0.01 and p <0.001, depending upon the number of analyses conducted to correct for type-1 error. Psychological measures were scored using conventional methods. RESULTS DANA performed well in all five field environments with no significant difference across data sets; therefore, the data for all five operational environments were combined. The number of total subjects for each test ranged from n = 75 to n = 224 depending on whether or not the service member was available to participate in all administrations across 2 days. All but one test (Code Substitution Delayed [CDD]) had over 200 subjects for at least two test administrations. Scores on psychological measures revealed an overall psychologically healthy sample. Combat Exposure Scale (CES) was in the light range (3.7, with 17 indicating moderate exposure). PTSD Checklist-Military Version (PCL-m) (26), Patient Health Questionnaire (PHQ)-8 (4.6), Pittsburgh Sleep Quality Index (PSQI) (5.9), and Deployment Stress Inventory (DSI) (9.7) were each far below the score needed to reach clinical criteria. Data exclusions included the elimination of trials with anticipatory responses and test administrations indicating random responses (criterion for exclusion was set at less than 65% correct-since tasks are binary, 50% correct represents random responding). No slow RT responses were eliminated, but to mitigate their undue influence we used medians rather than means to describe the data. Less than 1% of the response trials and less than 2% of the test administrations were eliminated based on these criteria. No subjects were eliminated from analysis based upon a criterion of having more than one test administration eliminated in a battery. Table II shows the descriptive data for each administration of each test, and Table III comparesDANAmedian RT and SD to previously published reports of mean data from the Automated Neuropsychological Assessment Metrics (ANAM, currently used by the U.S. Department of Defense for baseline, predeployment neurocognitive testing) in military personnel. 5-7 DANA uses median correct RT as the relevant metric, whereas ANAM reports means; however, because of the sample size and the normal populations assessed, the mean and median are assumed to be similar for ANAM. Although not significantly different, absolute RT values may be different because of differing instrumentation (stylus versus mouse button) between ANAM and DANA. It is also possible to use the published ANAM data to calculate Coefficients of Variation (CVs) (ratio of RT to SD of RT) to show stability8 and these are compared to analogous DANA subtests in Table III. For reliability within administrations, split-half correlations of the odd-even trials are reported for the first administration of day 1 and day 2 for each test. Correlations were acceptably high ( p < 0.001) and generally above 0.85-Simple Reaction Time (SRT) (0.91, 0.93), Procedural Reaction Time (PRO) (0.87, 0.86), GNG (0.85, 0.85), Code Substitution Simultaneous (CDS) (0.94, 0.93)-except for Spatial Discrimination (SPD) (0.76, 0.76) and CDD (0.76, 0.82). For test reliability across administrations, the ICC was calculated. The ICC for SRT was 0.95 indicating excellent reliability across the 10 administrations over 2 days. For PRO, GNG, SPD, and CDS, similar reliability was achieved with ICC values of 0.91, 0.95, 0.89, and 0.88, respectively. Only the CDD test did not have high reliability (0.54), which is expected for repeat CDD tests within a short time period because of the change in codes with each administration, resulting in proactive memory interference. Correlations were also conducted between all psychological and cognitive measures. With the exception of CDD (difficulties using multiple administrations of this subtest because of memory interference were noted above), all cognitive subtests (p < 0.001) and psychological tests (p < 0.001) were correlated with each other; however, the psychological and cognitive tests did not correlate with each other in this sample of nonimpaired service members. The CES correlated mildly with the PCL-m and DSI (p < 0.01). DISCUSSION The data presented here represent a first empirical examination of the DANA tool, a portable NCAT that includes both cognitive and psychological tests. Feasibility, reliability, and internal validity were assessed through the administration of DANA to 224 active duty service members (officers and enlisted) from the U.S. Air Force, Navy, and Marine Corps across five extreme environments. As can be seen in Table II, scores were stable across administrations. Split-half reliability correlations for DANA subtests are within acceptable ranges, and are comparable to those reported for other similar NCAT subtests.9 ICC measures across administrations were excellent and generally exceeded those reported in similar NCAT subtests although they were comparable to previously reported aggregate NCAT scores.10 We are currently developing more sophisticated scoring and statistical approaches to score each subtest as well as to aggregate subtests into composite index scores. Descriptive analysis of the psychological subtests shows that the sample scored, in general, well below scores indicative of clinical problems. From the range of CES scores, it appears that most of the sample had not been exposed to moderate or greater combat. With regard to mean psychological scores and the relevant scoring for cognitive variables, all psychological variables correlate highly with each other, with the exception of the Insomnia Severity Index (ISI) and Primary Care (PC)-PTSD, which had moderate correlations, likely due to having a low number of items (4 and 7 items, respectively). Similarly, most cognitive tests correlated moderately with each other, with the exception of CDD, which did not correlate with the other cognitive measures, likely tapping into a different construct than the other measures. In addition, in this nonclinical population, no cognitive tests correlated with any psychological tests. This is to be expected given that the great majority of scores on the psychological tests were well within the "normal" range, and the cognitive scores were closely grouped. Finally, CES did not correlate with any measure in this nonclinical population of mostly noncombat deployed service members. As can be seen in Table III, DANA compares favorably to existing NCATs in terms of median/mean RT (or % accuracy for CDD) and SD. The CVs are also consistent with CVs reported from ANAM data collected in 2006, 2008, and 2012 cohorts. This suggests that differences in absolute values for RT are most likely due to the testing instrument (mouse click versus stylus) rather than anything implicit in the test itself. Taken together, DANA appears to have adequate reliability and test validity in a sample of nonclinical service members across services and environments. DANA is currently being assessed in postdeployment and clinical samples. The results reported here suggest that the DANA has promise as a next generation NCAT. Benefits of the DANA include that it (a) includes relevant psychological tests as well as standard cognitive tests; (b) is built on a portable OS with more precise timing than previous NCATs; and (c) is a portable handheld device, which is more versatile than a laptop computer. Future studies of DANA are planned to assess DANA's ability to assist frontline providers to more rapidly and accurately evaluate service members to determine the need for higher level evaluation, treatment, or readiness to return to duty. APPENDIX Table AI describes the eight cognitive tests and seven psychological questionnaires that were selected for the DANA test batteries. Each individual subtest parameters are shown in Table AII including stimulus presentation duration, response interval, and interstimulus interval. ACKNOWLEDGMENTS We thank the following people for their help in data collection and/or analysis: James Drane, Jon Farris, and Lindsay Long. This work was supported by the U.S. Navy Bureau of Medicine and Surgery, Wounded, Ill and Injured Directorate, Contract No. N00189-10-P-Z967 as well as with resources and the use of facilities at the Department of Veterans Affairs, National Center for PTSD, Pacific Islands Health Care System. REFERENCES 1. Cernich AN, Brennan DM, Barker LM, Bleiberg J: Sources of error in computerized neuropsychological assessment. Arch Clin Neuropsychol 2007; 22S: S39-48. 2. Wilson AJ, Mellinger J, Schalk G, Williams JC: A procedure for measuring latencies in brain-computer interfaces. IEEE Trans Biomed Eng 2010; 57(7): 1785-97. 3. Weir JP: Quantifying test-retest reliability using the intraclass correlation coefficient and the SEM. J Strength Cond Res 2005; 19(1): 231-40. 4. Marx RG, Menezes A, Horovitz L, Jones EC, Warren RF: A comparison of two time intervals for test-retest reliability of health status instruments. J Clin Epidemiol 2003; 56: 730-5. 5. Reeves DL, Bleiberg J, Roebuck-Spencer T, et al: Reference values for performance on the Automated Neuropsychological Assessment Metrics V3.0 in an active duty military sample. Mil Med 2006; 171: 982-94. 6. Vincent AS, Bleiberg J, Yan S, et al: Reference data from the automated Neuropsychological Assessment Metrics for use in traumatic brain injury in an active duty military sample. Mil Med 2008; 173: 836-52. 7. Vincent AS, Roebuck-Spencer T, Gilliland K, Schlegel R: Automated Neuropsychological Assessment Metrics (v4) Traumatic Brain Injury Battery: military normative data. Mil Med 2012; 177(3): 256-69. 8. Beharelle AR, Tisserand D, Stuss DT, McIntosh AR, Levine B: Brain activity patterns uniquely supporting visual feature integration after traumatic brain injury. Front Hum Neurosci 2011; 5: 164. 9. Cernich A, Reeves D, SunW, Bleiberg J: Automated Neuropsychological Assessment Metrics sports medicine battery. Arch Clin Neuropsychol 2007; 22S: S101-14. 10. Segalowitz SJ, Mahaney P, Santesso DL, MacGregor L, Dywan J, Willer B: Retest reliability in adolescents of a computerized neuropsychological battery used to assess recovery from concussion. Neuro- Rehabilitation 2007; 22(3): 243-51. 11. Bastien CH, Vallières A, Morin CM: Validation of the Insomnia Severity Index as an outcome measure for insomnia research. Sleep Med 2001; 2(4): 297-307. 12. Prins A, Ouimette P, Kimerling R, et al: The Primary Care PTSD screen (PC-PTSD). Primary Care Psychiatry 2003; 9: 9-14. 13. Prins A, Ouimette P, Kimerling R, et al: The Primary Care PTSD screen (PC-PTSD): Corrigendum. Primary Care Psychiatry 2004; 9: 151. 14. Kroenke K, Strine TW, Spitzer RL, Williams JB, Berry JT, Mokdad AH: The PHQ-8 as a measure of current depression in the general population. J Affect Disord 2009; 114(1-3): 163-73. 15. Buysse DJ, Reynolds CF III, Monk TH, Berman SR, Kupfer DJ: The Pittsburgh Sleep Quality Index: a new instrument for psychiatric practice and research. Psychiatry Res 1989; 28(2): 193-213. 16. Keane T, Fairbank J, Caddell J, Zimering R, Taylor K, Mora C: Clinical evaluation of a measure to assess combat exposure. Psychol Assessment 1989; 1: 53-5. 17. Bliese PD, Wright KM, Adler AB, Cabrera O, Castrol CA, Hoge CW: Validating the primary care posttraumatic stress disorder screen and the posttraumatic stress disorder checklist with soldiers returning from combat. J Consult Clin Psychol 2008; 76: 272-81. 18. Meterko M, Baker E, Stolzmann KL, Hendricks AM, Cicerone KD, Lew HL: Psychometric assessment of the Neurobehavioral Symptom Inventory-22: the structure of persistent postconcussive symptoms following deployment-relatedmild traumatic brain injury among veterans. J Head Trauma Rehabil 2012; 27(1): 55-62. Corinna Lathan, PhD*; James L. Spira, PhD, MPH[dagger]; Joseph Bleiberg, PhD[double dagger]; Jack Vice*; CAPT Jack W. Tsao, USN§ *AnthroTronix, 8737 Colesville Road, Suite L-203, Silver Spring, MD 20910. [dagger]Department of Veterans Affairs, National Center for PTSD, Pacific Islands Health Care System, 3375 Koapaka Street, Honolulu, HI 96819. [double dagger]National Intrepid Center of Excellence, Walter Reed National Military Medical Center, 8901 Wisconsin Avenue, Building 51, Bethesda, MD 20889. §Wounded, Ill and Injured Directorate, US Navy Bureau of Medicine and Surgery, 7700 Arlington Boulevard, Falls Church, VA 22032. The preliminary data of this article, "Defense Automated Neurobehavioral Assessment (DANA): A Field-Deployable Tool for Assessing Concussion and Deployment Stress," was presented as an oral presentation by the authors J.L.S., C.L., J.B., and J.W.T. at the Military Health System Research Symposium (MHSRS) Conference, Fort Lauderdale, FL, August 2012. The opinions or assertions contained herein are the private views of the authors and are not to be construed as official or as reflecting the views of the Department of Veterans Affairs, the Department of the Navy, the Department of Defense, or the U.S. Government. doi: 10.7205/MILMED-D-12-00438 (c) 2013 Association of Military Surgeons of the United States |