2024-03-19T02:04:27Z
https://nagoya.repo.nii.ac.jp/oai
oai:nagoya.repo.nii.ac.jp:00007110
2023-01-16T04:50:48Z
309:318:319
An Application of the NTCIR-WEB Raw-data Archive Dataset for User Experiments
TAKAKU, Masao
19221
EGUSA, Yuka
19222
SAITO, Hitomi
19223
TERAI, Hitoshi
19224
寺井, 仁
19225
Evaluation
User experiments
User interface
Rawdata archive
Web information retrieval
This paper presents a simple approach to utilize past test collections as a material for user experiments. We have built a Web-based user interface for NTCIR-5 WEB run results, and conducted a user experiment with 29 subjects to investigate whether performance evaluation metrics of information retrieval systems used in test collections such as TREC and NTCIR comparable to user performance. In this experiment, we selected three types of systems from among systems that participated in NTCIR-5 WEB, and then selected three topics with roughly the same values from among several search topics. The results of the experiment showed no significant differences among these systems and topics in the time for search. While, in general, the user experiment itself have been successfully conducted and shown similar trends with prior study, the approach seems to have some limitations mainly on interactivity and cached page display.
journal article
National Institute of Informatics (NII)
2007
application/pdf
Proceedings of the First International Workshop on Evaluating Information Access (EVIA 2007) (NTCIR-6 Pre-Meeting Workshop)
78
81
http://hdl.handle.net/2237/8815
https://nagoya.repo.nii.ac.jp/record/7110/files/EVIA.pdf
eng
http://research.nii.ac.jp/ntcir/workshop/OnlineProceedings6/EVIA/14.pdf