Evaluation of Automatically Assigned Job-Specific Interview Modules

Melissa C. Friesen, Qing Lan, Calvin Ge, Sarah J. Locke, Howard D. Hosgood, Lin Fritschi, Troy Sadkowsky, Yu Cheng Chen, Hu Wei, Jun Xu, Tai Hing Lam, Yok Lam Kwong, Kexin Chen, Caigang Xu, Yu Chieh Su, Brian C H Chiu, Kai Ming Dennis Ip, Mark P. Purdue, Bryan A. Bassig, Nat Rothman & 1 others Roel Vermeulen

Research output: Contribution to journalArticle

4 Citations (Scopus)

Abstract

Objective: In community-based epidemiological studies, job-and industry-specific 'modules' are often used to systematically obtain details about the subject's work tasks. The module assignment is often made by the interviewer, who may have insufficient occupational hygiene knowledge to assign the correct module. We evaluated, in the context of a case-control study of lymphoid neoplasms in Asia ('AsiaLymph'), the performance of an algorithm that provided automatic, real-Time module assignment during a computer-Assisted personal interview. Methods: AsiaLymph's occupational component began with a lifetime occupational history questionnaire with free-Text responses and three solvent exposure screening questions. To assign each job to one of 23 study-specific modules, an algorithm automatically searched the free-Text responses to the questions 'job title' and 'product made or services provided by employer' using a list of module-specific keywords, comprising over 5800 keywords in English, Traditional and Simplified Chinese. Hierarchical decision rules were used when the keyword match triggered multiple modules. If no keyword match was identified, a generic solvent module was assigned if the subject responded 'yes' to any of the three solvent screening questions. If these question responses were all 'no', a work location module was assigned, which redirected the subject to the farming, teaching, health professional, solvent, or industry solvent modules or ended the questions for that job, depending on the location response. We conducted a reliability assessment that compared the algorithm-Assigned modules to consensus module assignments made by two industrial hygienists for a subset of 1251 (of 11409) jobs selected using a stratified random selection procedure using module-specific strata. Discordant assignments between the algorithm and consensus assignments (483 jobs) were qualitatively reviewed by the hygienists to evaluate the potential information lost from missed questions with using the algorithm-Assigned module (none, low, medium, high). Results: The most frequently assigned modules were the work location (33%), solvent (20%), farming and food industry (19%), and dry cleaning and textile industry (6.4%) modules. In the reliability subset, the algorithm assignment had an exact match to the expert consensus-Assigned module for 722 (57.7%) of the 1251 jobs. Overall, adjusted for the proportion of jobs in each stratum, we estimated that 86% of the algorithm-Assigned modules would result in no information loss, 2% would have low information loss, and 12% would have medium to high information loss. Medium to high information loss occurred for <10% of the jobs assigned the generic solvent module and for 21, 32, and 31% of the jobs assigned the work location module with location responses of 'someplace else', 'factory', and 'don't know', respectively. Other work location responses had ≤8% with medium to high information loss because of redirections to other modules. Medium to high information loss occurred more frequently when a job description matched with multiple keywords pointing to different modules (29-69%, depending on the triggered assignment rule). Conclusions: These evaluations demonstrated that automatically assigned modules can reliably reproduce an expert's module assignment without the direct involvement of an industrial hygienist or interviewer. The feasibility of adapting this framework to other studies will be language-and exposure-specific.

Original languageEnglish (US)
Pages (from-to)885-899
Number of pages15
JournalAnnals of Occupational Hygiene
Volume60
Issue number7
DOIs
StatePublished - Aug 1 2016

Fingerprint

Interviews
Workplace
Agriculture
Industry
Job Description
Textile Industry
Food Industry
Microcomputers
Hygiene
Case-Control Studies
Epidemiologic Studies
Consensus
Teaching
Language
Health
Neoplasms

Keywords

  • case-control studies
  • epidemiologic studies
  • occupational exposure
  • solvents

ASJC Scopus subject areas

  • Public Health, Environmental and Occupational Health

Cite this

Friesen, M. C., Lan, Q., Ge, C., Locke, S. J., Hosgood, H. D., Fritschi, L., ... Vermeulen, R. (2016). Evaluation of Automatically Assigned Job-Specific Interview Modules. Annals of Occupational Hygiene, 60(7), 885-899. https://doi.org/10.1093/annhyg/mew029

Evaluation of Automatically Assigned Job-Specific Interview Modules. / Friesen, Melissa C.; Lan, Qing; Ge, Calvin; Locke, Sarah J.; Hosgood, Howard D.; Fritschi, Lin; Sadkowsky, Troy; Chen, Yu Cheng; Wei, Hu; Xu, Jun; Lam, Tai Hing; Kwong, Yok Lam; Chen, Kexin; Xu, Caigang; Su, Yu Chieh; Chiu, Brian C H; Ip, Kai Ming Dennis; Purdue, Mark P.; Bassig, Bryan A.; Rothman, Nat; Vermeulen, Roel.

In: Annals of Occupational Hygiene, Vol. 60, No. 7, 01.08.2016, p. 885-899.

Research output: Contribution to journalArticle

Friesen, MC, Lan, Q, Ge, C, Locke, SJ, Hosgood, HD, Fritschi, L, Sadkowsky, T, Chen, YC, Wei, H, Xu, J, Lam, TH, Kwong, YL, Chen, K, Xu, C, Su, YC, Chiu, BCH, Ip, KMD, Purdue, MP, Bassig, BA, Rothman, N & Vermeulen, R 2016, 'Evaluation of Automatically Assigned Job-Specific Interview Modules', Annals of Occupational Hygiene, vol. 60, no. 7, pp. 885-899. https://doi.org/10.1093/annhyg/mew029
Friesen, Melissa C. ; Lan, Qing ; Ge, Calvin ; Locke, Sarah J. ; Hosgood, Howard D. ; Fritschi, Lin ; Sadkowsky, Troy ; Chen, Yu Cheng ; Wei, Hu ; Xu, Jun ; Lam, Tai Hing ; Kwong, Yok Lam ; Chen, Kexin ; Xu, Caigang ; Su, Yu Chieh ; Chiu, Brian C H ; Ip, Kai Ming Dennis ; Purdue, Mark P. ; Bassig, Bryan A. ; Rothman, Nat ; Vermeulen, Roel. / Evaluation of Automatically Assigned Job-Specific Interview Modules. In: Annals of Occupational Hygiene. 2016 ; Vol. 60, No. 7. pp. 885-899.
@article{d97581be187e41bfb2a50a1cb8db225e,
title = "Evaluation of Automatically Assigned Job-Specific Interview Modules",
abstract = "Objective: In community-based epidemiological studies, job-and industry-specific 'modules' are often used to systematically obtain details about the subject's work tasks. The module assignment is often made by the interviewer, who may have insufficient occupational hygiene knowledge to assign the correct module. We evaluated, in the context of a case-control study of lymphoid neoplasms in Asia ('AsiaLymph'), the performance of an algorithm that provided automatic, real-Time module assignment during a computer-Assisted personal interview. Methods: AsiaLymph's occupational component began with a lifetime occupational history questionnaire with free-Text responses and three solvent exposure screening questions. To assign each job to one of 23 study-specific modules, an algorithm automatically searched the free-Text responses to the questions 'job title' and 'product made or services provided by employer' using a list of module-specific keywords, comprising over 5800 keywords in English, Traditional and Simplified Chinese. Hierarchical decision rules were used when the keyword match triggered multiple modules. If no keyword match was identified, a generic solvent module was assigned if the subject responded 'yes' to any of the three solvent screening questions. If these question responses were all 'no', a work location module was assigned, which redirected the subject to the farming, teaching, health professional, solvent, or industry solvent modules or ended the questions for that job, depending on the location response. We conducted a reliability assessment that compared the algorithm-Assigned modules to consensus module assignments made by two industrial hygienists for a subset of 1251 (of 11409) jobs selected using a stratified random selection procedure using module-specific strata. Discordant assignments between the algorithm and consensus assignments (483 jobs) were qualitatively reviewed by the hygienists to evaluate the potential information lost from missed questions with using the algorithm-Assigned module (none, low, medium, high). Results: The most frequently assigned modules were the work location (33{\%}), solvent (20{\%}), farming and food industry (19{\%}), and dry cleaning and textile industry (6.4{\%}) modules. In the reliability subset, the algorithm assignment had an exact match to the expert consensus-Assigned module for 722 (57.7{\%}) of the 1251 jobs. Overall, adjusted for the proportion of jobs in each stratum, we estimated that 86{\%} of the algorithm-Assigned modules would result in no information loss, 2{\%} would have low information loss, and 12{\%} would have medium to high information loss. Medium to high information loss occurred for <10{\%} of the jobs assigned the generic solvent module and for 21, 32, and 31{\%} of the jobs assigned the work location module with location responses of 'someplace else', 'factory', and 'don't know', respectively. Other work location responses had ≤8{\%} with medium to high information loss because of redirections to other modules. Medium to high information loss occurred more frequently when a job description matched with multiple keywords pointing to different modules (29-69{\%}, depending on the triggered assignment rule). Conclusions: These evaluations demonstrated that automatically assigned modules can reliably reproduce an expert's module assignment without the direct involvement of an industrial hygienist or interviewer. The feasibility of adapting this framework to other studies will be language-and exposure-specific.",
keywords = "case-control studies, epidemiologic studies, occupational exposure, solvents",
author = "Friesen, {Melissa C.} and Qing Lan and Calvin Ge and Locke, {Sarah J.} and Hosgood, {Howard D.} and Lin Fritschi and Troy Sadkowsky and Chen, {Yu Cheng} and Hu Wei and Jun Xu and Lam, {Tai Hing} and Kwong, {Yok Lam} and Kexin Chen and Caigang Xu and Su, {Yu Chieh} and Chiu, {Brian C H} and Ip, {Kai Ming Dennis} and Purdue, {Mark P.} and Bassig, {Bryan A.} and Nat Rothman and Roel Vermeulen",
year = "2016",
month = "8",
day = "1",
doi = "10.1093/annhyg/mew029",
language = "English (US)",
volume = "60",
pages = "885--899",
journal = "Annals of Work Exposures and Health",
issn = "2398-7308",
publisher = "Oxford University Press",
number = "7",

}

TY - JOUR

T1 - Evaluation of Automatically Assigned Job-Specific Interview Modules

AU - Friesen, Melissa C.

AU - Lan, Qing

AU - Ge, Calvin

AU - Locke, Sarah J.

AU - Hosgood, Howard D.

AU - Fritschi, Lin

AU - Sadkowsky, Troy

AU - Chen, Yu Cheng

AU - Wei, Hu

AU - Xu, Jun

AU - Lam, Tai Hing

AU - Kwong, Yok Lam

AU - Chen, Kexin

AU - Xu, Caigang

AU - Su, Yu Chieh

AU - Chiu, Brian C H

AU - Ip, Kai Ming Dennis

AU - Purdue, Mark P.

AU - Bassig, Bryan A.

AU - Rothman, Nat

AU - Vermeulen, Roel

PY - 2016/8/1

Y1 - 2016/8/1

N2 - Objective: In community-based epidemiological studies, job-and industry-specific 'modules' are often used to systematically obtain details about the subject's work tasks. The module assignment is often made by the interviewer, who may have insufficient occupational hygiene knowledge to assign the correct module. We evaluated, in the context of a case-control study of lymphoid neoplasms in Asia ('AsiaLymph'), the performance of an algorithm that provided automatic, real-Time module assignment during a computer-Assisted personal interview. Methods: AsiaLymph's occupational component began with a lifetime occupational history questionnaire with free-Text responses and three solvent exposure screening questions. To assign each job to one of 23 study-specific modules, an algorithm automatically searched the free-Text responses to the questions 'job title' and 'product made or services provided by employer' using a list of module-specific keywords, comprising over 5800 keywords in English, Traditional and Simplified Chinese. Hierarchical decision rules were used when the keyword match triggered multiple modules. If no keyword match was identified, a generic solvent module was assigned if the subject responded 'yes' to any of the three solvent screening questions. If these question responses were all 'no', a work location module was assigned, which redirected the subject to the farming, teaching, health professional, solvent, or industry solvent modules or ended the questions for that job, depending on the location response. We conducted a reliability assessment that compared the algorithm-Assigned modules to consensus module assignments made by two industrial hygienists for a subset of 1251 (of 11409) jobs selected using a stratified random selection procedure using module-specific strata. Discordant assignments between the algorithm and consensus assignments (483 jobs) were qualitatively reviewed by the hygienists to evaluate the potential information lost from missed questions with using the algorithm-Assigned module (none, low, medium, high). Results: The most frequently assigned modules were the work location (33%), solvent (20%), farming and food industry (19%), and dry cleaning and textile industry (6.4%) modules. In the reliability subset, the algorithm assignment had an exact match to the expert consensus-Assigned module for 722 (57.7%) of the 1251 jobs. Overall, adjusted for the proportion of jobs in each stratum, we estimated that 86% of the algorithm-Assigned modules would result in no information loss, 2% would have low information loss, and 12% would have medium to high information loss. Medium to high information loss occurred for <10% of the jobs assigned the generic solvent module and for 21, 32, and 31% of the jobs assigned the work location module with location responses of 'someplace else', 'factory', and 'don't know', respectively. Other work location responses had ≤8% with medium to high information loss because of redirections to other modules. Medium to high information loss occurred more frequently when a job description matched with multiple keywords pointing to different modules (29-69%, depending on the triggered assignment rule). Conclusions: These evaluations demonstrated that automatically assigned modules can reliably reproduce an expert's module assignment without the direct involvement of an industrial hygienist or interviewer. The feasibility of adapting this framework to other studies will be language-and exposure-specific.

AB - Objective: In community-based epidemiological studies, job-and industry-specific 'modules' are often used to systematically obtain details about the subject's work tasks. The module assignment is often made by the interviewer, who may have insufficient occupational hygiene knowledge to assign the correct module. We evaluated, in the context of a case-control study of lymphoid neoplasms in Asia ('AsiaLymph'), the performance of an algorithm that provided automatic, real-Time module assignment during a computer-Assisted personal interview. Methods: AsiaLymph's occupational component began with a lifetime occupational history questionnaire with free-Text responses and three solvent exposure screening questions. To assign each job to one of 23 study-specific modules, an algorithm automatically searched the free-Text responses to the questions 'job title' and 'product made or services provided by employer' using a list of module-specific keywords, comprising over 5800 keywords in English, Traditional and Simplified Chinese. Hierarchical decision rules were used when the keyword match triggered multiple modules. If no keyword match was identified, a generic solvent module was assigned if the subject responded 'yes' to any of the three solvent screening questions. If these question responses were all 'no', a work location module was assigned, which redirected the subject to the farming, teaching, health professional, solvent, or industry solvent modules or ended the questions for that job, depending on the location response. We conducted a reliability assessment that compared the algorithm-Assigned modules to consensus module assignments made by two industrial hygienists for a subset of 1251 (of 11409) jobs selected using a stratified random selection procedure using module-specific strata. Discordant assignments between the algorithm and consensus assignments (483 jobs) were qualitatively reviewed by the hygienists to evaluate the potential information lost from missed questions with using the algorithm-Assigned module (none, low, medium, high). Results: The most frequently assigned modules were the work location (33%), solvent (20%), farming and food industry (19%), and dry cleaning and textile industry (6.4%) modules. In the reliability subset, the algorithm assignment had an exact match to the expert consensus-Assigned module for 722 (57.7%) of the 1251 jobs. Overall, adjusted for the proportion of jobs in each stratum, we estimated that 86% of the algorithm-Assigned modules would result in no information loss, 2% would have low information loss, and 12% would have medium to high information loss. Medium to high information loss occurred for <10% of the jobs assigned the generic solvent module and for 21, 32, and 31% of the jobs assigned the work location module with location responses of 'someplace else', 'factory', and 'don't know', respectively. Other work location responses had ≤8% with medium to high information loss because of redirections to other modules. Medium to high information loss occurred more frequently when a job description matched with multiple keywords pointing to different modules (29-69%, depending on the triggered assignment rule). Conclusions: These evaluations demonstrated that automatically assigned modules can reliably reproduce an expert's module assignment without the direct involvement of an industrial hygienist or interviewer. The feasibility of adapting this framework to other studies will be language-and exposure-specific.

KW - case-control studies

KW - epidemiologic studies

KW - occupational exposure

KW - solvents

UR - http://www.scopus.com/inward/record.url?scp=84982987649&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=84982987649&partnerID=8YFLogxK

U2 - 10.1093/annhyg/mew029

DO - 10.1093/annhyg/mew029

M3 - Article

VL - 60

SP - 885

EP - 899

JO - Annals of Work Exposures and Health

JF - Annals of Work Exposures and Health

SN - 2398-7308

IS - 7

ER -