Exploring automatic text-to-sign translation in a healthcare setting (2024)

Esselink, L., Roelofsen, F., Dotlacil, J., Mende-Gillings, S., de Meulder, M., Sijm, N., & Smeijers, A. (2024). Exploring automatic text-to-sign translation in a healthcare setting. Universal Access in the Information Society, 23(1), 35-57. https://doi.org/10.1007/s10209-023-01042-6

Esselink, L ; Roelofsen, F ; Dotlacil, J et al. / Exploring automatic text-to-sign translation in a healthcare setting. In: Universal Access in the Information Society. 2024 ; Vol. 23, No. 1. pp. 35-57.

@article{2595adb05d304eb3a28511499da02dac,

title = "Exploring automatic text-to-sign translation in a healthcare setting",

abstract = "Communication between healthcare professionals and deaf patients has been particularly challenging during the COVID-19 pandemic. We have explored the possibility to automatically translate phrases that are frequently used in the diagnosis and treatment of hospital patients, in particular phrases related to COVID-19, from Dutch or English to Dutch Sign Language (NGT). The prototype system we developed displays translations either by means of pre-recorded videos featuring a deaf human signer (for a limited number of sentences) or by means of animations featuring a computer-generated signing avatar (for a larger, though still restricted number of sentences). We evaluated the comprehensibility of the signing avatar, as compared to the human signer. We found that, while individual signs are recognized correctly when signed by the avatar almost as frequently as when signed by a human, sentence comprehension rates and clarity scores for the avatar are substantially lower than for the human signer. We identify a number of concrete limitations of the JASigning avatar engine that underlies our system. Namely, the engine currently does not offer sufficient control over mouth shapes, the relative speed and intensity of signs in a sentence (prosody), and transitions between signs. These limitations need to be overcome in future work for the engine to become usable in practice.",

keywords = "Access to healthcare information, Avatar technology, Sign language, User study",

author = "L Esselink and F Roelofsen and J Dotlacil and S Mende-Gillings and {de Meulder}, M and N Sijm and A Smeijers",

note = "Publisher Copyright: {\textcopyright} The Author(s) 2023.",

year = "2024",

month = mar,

doi = "10.1007/s10209-023-01042-6",

language = "English",

volume = "23",

pages = "35--57",

journal = "Universal Access in the Information Society",

issn = "1615-5289",

publisher = "Springer",

number = "1",

}

Esselink, L, Roelofsen, F, Dotlacil, J, Mende-Gillings, S, de Meulder, M, Sijm, N & Smeijers, A 2024, 'Exploring automatic text-to-sign translation in a healthcare setting', Universal Access in the Information Society, vol. 23, no. 1, pp. 35-57. https://doi.org/10.1007/s10209-023-01042-6

Exploring automatic text-to-sign translation in a healthcare setting. / Esselink, L; Roelofsen, F; Dotlacil, J et al.
In: Universal Access in the Information Society, Vol. 23, No. 1, 03.2024, p. 35-57.

Research output: Contribution to journal › Article › Academic › peer-review

TY - JOUR

T1 - Exploring automatic text-to-sign translation in a healthcare setting

AU - Esselink, L

AU - Roelofsen, F

AU - Dotlacil, J

AU - Mende-Gillings, S

AU - de Meulder, M

AU - Sijm, N

AU - Smeijers, A

N1 - Publisher Copyright:© The Author(s) 2023.

PY - 2024/3

Y1 - 2024/3

N2 - Communication between healthcare professionals and deaf patients has been particularly challenging during the COVID-19 pandemic. We have explored the possibility to automatically translate phrases that are frequently used in the diagnosis and treatment of hospital patients, in particular phrases related to COVID-19, from Dutch or English to Dutch Sign Language (NGT). The prototype system we developed displays translations either by means of pre-recorded videos featuring a deaf human signer (for a limited number of sentences) or by means of animations featuring a computer-generated signing avatar (for a larger, though still restricted number of sentences). We evaluated the comprehensibility of the signing avatar, as compared to the human signer. We found that, while individual signs are recognized correctly when signed by the avatar almost as frequently as when signed by a human, sentence comprehension rates and clarity scores for the avatar are substantially lower than for the human signer. We identify a number of concrete limitations of the JASigning avatar engine that underlies our system. Namely, the engine currently does not offer sufficient control over mouth shapes, the relative speed and intensity of signs in a sentence (prosody), and transitions between signs. These limitations need to be overcome in future work for the engine to become usable in practice.

AB - Communication between healthcare professionals and deaf patients has been particularly challenging during the COVID-19 pandemic. We have explored the possibility to automatically translate phrases that are frequently used in the diagnosis and treatment of hospital patients, in particular phrases related to COVID-19, from Dutch or English to Dutch Sign Language (NGT). The prototype system we developed displays translations either by means of pre-recorded videos featuring a deaf human signer (for a limited number of sentences) or by means of animations featuring a computer-generated signing avatar (for a larger, though still restricted number of sentences). We evaluated the comprehensibility of the signing avatar, as compared to the human signer. We found that, while individual signs are recognized correctly when signed by the avatar almost as frequently as when signed by a human, sentence comprehension rates and clarity scores for the avatar are substantially lower than for the human signer. We identify a number of concrete limitations of the JASigning avatar engine that underlies our system. Namely, the engine currently does not offer sufficient control over mouth shapes, the relative speed and intensity of signs in a sentence (prosody), and transitions between signs. These limitations need to be overcome in future work for the engine to become usable in practice.

KW - Access to healthcare information

KW - Avatar technology

KW - Sign language

KW - User study

UR - http://www.scopus.com/inward/record.url?scp=85172871214&partnerID=8YFLogxK

U2 - 10.1007/s10209-023-01042-6

DO - 10.1007/s10209-023-01042-6

M3 - Article

SN - 1615-5289

VL - 23

SP - 35

EP - 57

JO - Universal Access in the Information Society

JF - Universal Access in the Information Society

IS - 1

ER -

Esselink L, Roelofsen F, Dotlacil J, Mende-Gillings S, de Meulder M, Sijm N et al. Exploring automatic text-to-sign translation in a healthcare setting. Universal Access in the Information Society. 2024 Mar;23(1):35-57. Epub 2023 Sept 29. doi: 10.1007/s10209-023-01042-6