UX · RES · 04 · Case study
LLMs in HCI data work
First-author paper on whether GPT-3.5 and Llama 2 can extract experimental data from HCI studies, and where they go wrong.
- Where
- CHI 2024 workshop
- When
- 2024
- Type
- Research
- arXiv paper arxiv.org: the full paper, free to read (opens in a new tab)
Context
Literature reviews in Human-Computer Interaction take a long time, because experimental details are spread across many papers. The paper asks whether large language models can do part of that work, and how much they can be trusted.
Who did what
My part
- First author of the paper.
- Read HCI papers closely to judge the models’ answers and find where they were quietly wrong.
Team part
- Co-authors: Iman Mohammadi, Vittorio Fuccella and Mattia De Rosa.
Decisions
Check the model against the papers
The value of the study is in the errors. Each extracted value was checked against the source paper, which shows where the models can help and where a person still has to look.
The same habit as usability testing: look at what really happens, not what should happen.
Outcome
- Published at a CHI 2024 workshop as “LLMs in HCI Data Work: Bridging the Gap Between Information Retrieval and Responsible Research Practices”.