Inter-rater reliability of preprocessing EEG data

Impact of subjective artifact removal on associative memory task ERP results

Steven D. Shirk, Donald G. McLaren, Jessica S. Bloomfield, Alex Powers, Alec Duffy, Meghan B. Mitchell, Ali Ezzati, Brandon A. Ally, Alireza Atri

Research output: Contribution to journalArticle

Abstract

The processing of EEG data routinely involves subjective removal of artifacts during a preprocessing stage. Preprocessing inter-rater reliability (IRR) and how differences in preprocessing may affect outcomes of primary event-related potential (ERP) analyses has not been previously assessed. Three raters independently preprocessed EEG data of 16 cognitively healthy adult participants (ages 18-39 years) who performed a memory task. Using intraclass correlations (ICCs), IRR was assessed for Early-frontal, Late-frontal, and Parietal Old/new memory effects contrasts across eight regions of interest (ROIs). IRR was good to excellent for all ROIs; 22 of 26 ICCs were above 0.80. Raters were highly consistent in preprocessing across ROIs, although the frontal pole ROI (ICC range 0.60-0.90) showed less consistency. Old/new parietal effects had highest ICCs with the lowest variability. Rater preprocessing differences did not alter primary ERP results. IRR for EEG preprocessing was good to excellent, and subjective rater-removal of EEG artifacts did not alter primary memory-task ERP results. Findings provide preliminary support for robustness of cognitive/memory task-related ERP results against significant inter-rater preprocessing variability and suggest reliability of EEG to assess cognitive-neurophysiological processes multiple preprocessors are involved.

Original languageEnglish (US)
Article number322
JournalFrontiers in Neuroscience
Volume11
Issue numberJUN
DOIs
StatePublished - Jan 1 2017

Fingerprint

Evoked Potentials
Artifacts
Electroencephalography
Healthy Volunteers

Keywords

  • Artifacts
  • EEG/ERP
  • Inter-rater reliability
  • Memory
  • Preprocessing

ASJC Scopus subject areas

  • Neuroscience(all)

Cite this

Shirk, S. D., McLaren, D. G., Bloomfield, J. S., Powers, A., Duffy, A., Mitchell, M. B., ... Atri, A. (2017). Inter-rater reliability of preprocessing EEG data: Impact of subjective artifact removal on associative memory task ERP results. Frontiers in Neuroscience, 11(JUN), [322]. https://doi.org/10.3389/fnins.2017.00322

Inter-rater reliability of preprocessing EEG data : Impact of subjective artifact removal on associative memory task ERP results. / Shirk, Steven D.; McLaren, Donald G.; Bloomfield, Jessica S.; Powers, Alex; Duffy, Alec; Mitchell, Meghan B.; Ezzati, Ali; Ally, Brandon A.; Atri, Alireza.

In: Frontiers in Neuroscience, Vol. 11, No. JUN, 322, 01.01.2017.

Research output: Contribution to journalArticle

Shirk, Steven D. ; McLaren, Donald G. ; Bloomfield, Jessica S. ; Powers, Alex ; Duffy, Alec ; Mitchell, Meghan B. ; Ezzati, Ali ; Ally, Brandon A. ; Atri, Alireza. / Inter-rater reliability of preprocessing EEG data : Impact of subjective artifact removal on associative memory task ERP results. In: Frontiers in Neuroscience. 2017 ; Vol. 11, No. JUN.
@article{2b5514e56a4042fcaf1674d81857f293,
title = "Inter-rater reliability of preprocessing EEG data: Impact of subjective artifact removal on associative memory task ERP results",
abstract = "The processing of EEG data routinely involves subjective removal of artifacts during a preprocessing stage. Preprocessing inter-rater reliability (IRR) and how differences in preprocessing may affect outcomes of primary event-related potential (ERP) analyses has not been previously assessed. Three raters independently preprocessed EEG data of 16 cognitively healthy adult participants (ages 18-39 years) who performed a memory task. Using intraclass correlations (ICCs), IRR was assessed for Early-frontal, Late-frontal, and Parietal Old/new memory effects contrasts across eight regions of interest (ROIs). IRR was good to excellent for all ROIs; 22 of 26 ICCs were above 0.80. Raters were highly consistent in preprocessing across ROIs, although the frontal pole ROI (ICC range 0.60-0.90) showed less consistency. Old/new parietal effects had highest ICCs with the lowest variability. Rater preprocessing differences did not alter primary ERP results. IRR for EEG preprocessing was good to excellent, and subjective rater-removal of EEG artifacts did not alter primary memory-task ERP results. Findings provide preliminary support for robustness of cognitive/memory task-related ERP results against significant inter-rater preprocessing variability and suggest reliability of EEG to assess cognitive-neurophysiological processes multiple preprocessors are involved.",
keywords = "Artifacts, EEG/ERP, Inter-rater reliability, Memory, Preprocessing",
author = "Shirk, {Steven D.} and McLaren, {Donald G.} and Bloomfield, {Jessica S.} and Alex Powers and Alec Duffy and Mitchell, {Meghan B.} and Ali Ezzati and Ally, {Brandon A.} and Alireza Atri",
year = "2017",
month = "1",
day = "1",
doi = "10.3389/fnins.2017.00322",
language = "English (US)",
volume = "11",
journal = "Frontiers in Neuroscience",
issn = "1662-4548",
publisher = "Frontiers Research Foundation",
number = "JUN",

}

TY - JOUR

T1 - Inter-rater reliability of preprocessing EEG data

T2 - Impact of subjective artifact removal on associative memory task ERP results

AU - Shirk, Steven D.

AU - McLaren, Donald G.

AU - Bloomfield, Jessica S.

AU - Powers, Alex

AU - Duffy, Alec

AU - Mitchell, Meghan B.

AU - Ezzati, Ali

AU - Ally, Brandon A.

AU - Atri, Alireza

PY - 2017/1/1

Y1 - 2017/1/1

N2 - The processing of EEG data routinely involves subjective removal of artifacts during a preprocessing stage. Preprocessing inter-rater reliability (IRR) and how differences in preprocessing may affect outcomes of primary event-related potential (ERP) analyses has not been previously assessed. Three raters independently preprocessed EEG data of 16 cognitively healthy adult participants (ages 18-39 years) who performed a memory task. Using intraclass correlations (ICCs), IRR was assessed for Early-frontal, Late-frontal, and Parietal Old/new memory effects contrasts across eight regions of interest (ROIs). IRR was good to excellent for all ROIs; 22 of 26 ICCs were above 0.80. Raters were highly consistent in preprocessing across ROIs, although the frontal pole ROI (ICC range 0.60-0.90) showed less consistency. Old/new parietal effects had highest ICCs with the lowest variability. Rater preprocessing differences did not alter primary ERP results. IRR for EEG preprocessing was good to excellent, and subjective rater-removal of EEG artifacts did not alter primary memory-task ERP results. Findings provide preliminary support for robustness of cognitive/memory task-related ERP results against significant inter-rater preprocessing variability and suggest reliability of EEG to assess cognitive-neurophysiological processes multiple preprocessors are involved.

AB - The processing of EEG data routinely involves subjective removal of artifacts during a preprocessing stage. Preprocessing inter-rater reliability (IRR) and how differences in preprocessing may affect outcomes of primary event-related potential (ERP) analyses has not been previously assessed. Three raters independently preprocessed EEG data of 16 cognitively healthy adult participants (ages 18-39 years) who performed a memory task. Using intraclass correlations (ICCs), IRR was assessed for Early-frontal, Late-frontal, and Parietal Old/new memory effects contrasts across eight regions of interest (ROIs). IRR was good to excellent for all ROIs; 22 of 26 ICCs were above 0.80. Raters were highly consistent in preprocessing across ROIs, although the frontal pole ROI (ICC range 0.60-0.90) showed less consistency. Old/new parietal effects had highest ICCs with the lowest variability. Rater preprocessing differences did not alter primary ERP results. IRR for EEG preprocessing was good to excellent, and subjective rater-removal of EEG artifacts did not alter primary memory-task ERP results. Findings provide preliminary support for robustness of cognitive/memory task-related ERP results against significant inter-rater preprocessing variability and suggest reliability of EEG to assess cognitive-neurophysiological processes multiple preprocessors are involved.

KW - Artifacts

KW - EEG/ERP

KW - Inter-rater reliability

KW - Memory

KW - Preprocessing

UR - http://www.scopus.com/inward/record.url?scp=85021223682&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=85021223682&partnerID=8YFLogxK

U2 - 10.3389/fnins.2017.00322

DO - 10.3389/fnins.2017.00322

M3 - Article

VL - 11

JO - Frontiers in Neuroscience

JF - Frontiers in Neuroscience

SN - 1662-4548

IS - JUN

M1 - 322

ER -