Abstract
This paper describes our proposal for an evaluation metric for XML retrieval that is solely based on the highlighted text. We support our decision of ignoring the exhaustivity dimension by undertaking a critical investigation of the two INEX 2005 relevance dimensions. We present a fine grained empirical analysis of the level of assessor agreement of the five topics double-judged at INEX 2005, and show that the agreement is higher for specificity than for exhaustivity. We use the proposed metric to evaluate the INEX 2005 runs for each retrieval strategy of the CO and CAS retrieval tasks. A correlation analysis of the rank orderings obtained by the new metric and two XCG metrics shows that the orderings are strongly correlated, which demonstrates the usefulness of the proposed metric for evaluation of XML retrieval performance.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Preview
Unable to display preview. Download preview PDF.
Similar content being viewed by others
References
Hiemstra, D., Mihajlovic, V.: The Simplest Evaluation Measures for XML Information Retrieval that Could Possibly Work. In: Fuhr, N., Lalmas, M., Malik, S., Kazai, G. (eds.) INEX 2005. LNCS, vol. 3977, pp. 6–13. Springer, Heidelberg (2006)
Jarvelin, K., Kekalainen, J.: Cumulated gain-based evaluation of IR techniques. ACM Transactions on Information Systems (TOIS) 20, 422–446 (2002)
Kazai, G., Lalmas, M.: INEX 2005 evaluation metrics. In: Fuhr, N., Lalmas, M., Malik, S., Kazai, G. (eds.) INEX 2005. LNCS, vol. 3977, pp. 401–406. Springer, Heidelberg (2006)
Kazai, G., Lalmas, M.: Notes on what to measure in INEX. In: Fuhr, N., Lalmas, M., Malik, S., Kazai, G. (eds.) INEX 2005. LNCS, vol. 3977, pp. 22–38. Springer, Heidelberg (2006)
Kazai, G., Lalmas, M., de Vries, A.P.: Reliability tests for the XCG and inex-2002 metrics. In: Fuhr, N., Lalmas, M., Malik, S., Szlávik, Z. (eds.) INEX 2004. LNCS, vol. 3493, pp. 60–72. Springer, Heidelberg (2005)
Lalmas, M.: INEX 2005 retrieval task and result submission specification. In: Fuhr, N., Lalmas, M., Malik, S., Kazai, G. (eds.) INEX 2005. LNCS, vol. 3977, pp. 385–390. Springer, Heidelberg (2006)
Lalmas, M., Piwowarski, B.: Inex 2005 relevance assessment guide. In: Fuhr, N., Lalmas, M., Malik, S., Kazai, G. (eds.) INEX 2005. LNCS, vol. 3977, pp. 391–400. Springer, Heidelberg (2006)
Pehcevski, J., Thom, J.A., Vercoustre, A.-M.: Users and Assessors in the Context of INEX: Are Relevance Dimensions Relevant? In: Fuhr, N., Lalmas, M., Malik, S., Kazai, G. (eds.) INEX 2005. LNCS, vol. 3977, pp. 47–62. Springer, Heidelberg (2006)
Sanderson, M., Zobel, J.: Information retrieval system evaluation: effort, sensitivity, and reliability. In: Proceedings of the ACM-SIGIR International Conference on Research and Development in Information Retrieval, Salvador, Brazil, pp. 162–169 (2005)
Trotman, A.: Wanted: Element Retrieval Users. In: Fuhr, N., Lalmas, M., Malik, S., Kazai, G. (eds.) INEX 2005. LNCS, vol. 3977, pp. 63–69. Springer, Heidelberg (2006)
Voorhees, E.M.: Variations in relevance judgements and the measurment of retrieval effectiveness. Information Processing & Management 36(5), 697–716 (2000)
Author information
Authors and Affiliations
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2006 Springer-Verlag Berlin Heidelberg
About this paper
Cite this paper
Pehcevski, J., Thom, J.A. (2006). HiXEval: Highlighting XML Retrieval Evaluation. In: Fuhr, N., Lalmas, M., Malik, S., Kazai, G. (eds) Advances in XML Information Retrieval and Evaluation. INEX 2005. Lecture Notes in Computer Science, vol 3977. Springer, Berlin, Heidelberg. https://doi.org/10.1007/978-3-540-34963-1_4
Download citation
DOI: https://doi.org/10.1007/978-3-540-34963-1_4
Publisher Name: Springer, Berlin, Heidelberg
Print ISBN: 978-3-540-34962-4
Online ISBN: 978-3-540-34963-1
eBook Packages: Computer ScienceComputer Science (R0)