Hostname: page-component-6587cd75c8-vj8bv Total loading time: 0 Render date: 2025-04-23T18:04:56.481Z Has data issue: false hasContentIssue false
  • English
  • Français

Machine learning for query formulation in question answering

Published online by Cambridge University Press:  05 January 2011

CHRISTOF MONZ
CHRISTOF MONZ*
Affiliation:
Informatics Institute, University of Amsterdam, Science Park 107, 1098 XG Amsterdam, The Netherlands e-mail: c.monz@uva.nl
Get access Rights & Permissions [Opens in a new window]

Abstract

Research on question answering dates back to the 1960s but has more recently been revisited as part of TREC's evaluation campaigns, where question answering is addressed as a subarea of information retrieval that focuses on specific answers to a user's information need. Whereas document retrieval systems aim to return the documents that are most relevant to a user's query, question answering systems aim to return actual answers to a users question. Despite this difference, question answering systems rely on information retrieval components to identify documents that contain an answer to a user's question. The computationally more expensive answer extraction methods are then applied only to this subset of documents that are likely to contain an answer. As information retrieval methods are used to filter the documents in the collection, the performance of this component is critical as documents that are not retrieved are not analyzed by the answer extraction component. The formulation of queries that are used for retrieving those documents has a strong impact on the effectiveness of the retrieval component. In this paper, we focus on predicting the importance of terms from the original question. We use model tree machine learning techniques in order to assign weights to query terms according to their usefulness for identifying documents that contain an answer. Term weights are learned by inspecting a large number of query formulation variations and their respective accuracy in identifying documents containing an answer. Several linguistic features are used for building the models, including part-of-speech tags, degree of connectivity in the dependency parse tree of the question, and ontological information. All of these features are extracted automatically by using several natural language processing tools. Incorporating the learned weights into a state-of-the-art retrieval system results in statistically significant improvements in identifying answer-bearing documents.

Type
Articles
Information
Natural Language Engineering , Volume 17 , Issue 4 , October 2011 , pp. 425 - 454
Copyright
Copyright © Cambridge University Press 2010

Access options

Get access to the full version of this content by using one of the access options below. (Log in options will check for institutional or personal access. Content may require purchase if you do not have access.)

Article purchase

Temporarily unavailable

References

Agichtein, E., Lawrence, S. and Gravano, L. 2004. Learning to find answers to questions on the web. ACM Transactions on Internet Technology 4 (2): 129–162.CrossRefGoogle Scholar
Bendersky, M. and Croft, B. 2008. Discovering key concepts in verbose queries. In Proceedings of the 31st Annual International ACM SIGIR Conference on Research and Development in Information Retrieval, Singapore, pp. 491–498.Google Scholar
Bilotti, M., Ogilvie, P., Callan, J. and Nyberg, E. 2007. Structured retrieval for question answering. In Proceedings of the 30th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval, Amsterdam, The Netherlands, pp. 351–358.CrossRefGoogle Scholar
Breimann, L., Friedman, J., Ohlsen, R. and Stone, C. 1984. Classification and Regression Trees. Wadsworth and Brooks.Google Scholar
Brill, E., Dumais, S. and Banko, M. 2002. An analysis of the AskMSR question-answering system. In Proceedings of Empirical Methods in Natural Language Processing (EMNLP 2002), Morristown, N.J., USA, pp. 257–264.Google Scholar
Brown, P. F., Della Pietra, S., Della Pietra, V., and Mercer, R. L. 1991. Word sense disambiguation using statistical methods. In Proceedings of the 29th Annual Meeting of the Association for Computational Linguistics (ACL), Morristown, N.J., USA, pp. 264–270.CrossRefGoogle Scholar
Buckley, C., Singhal, A. and Mitra, M. 1995. New retrieval approaches using SMART: TREC 4. In Proceedings of the Fourth Text REtrieval Conference (TREC-4), Gaithersburg, M.D., USA, pp. 25–48.Google Scholar
Chen, H., Shankaranarayanan, G., She, L. and Iyer, A. 1998. A machine learning approach to inductive query by examples: an experiment using relevance feedback, ID3, genetic algorithms, and simulated annealing. Journal of the American Society for Information Science 49 (8): 693–705.3.0.CO;2-O>CrossRefGoogle Scholar
Cooper, W., Chen, A. and Gey, F. 1993. Full text retrieval based on probabilistic equations with coefficients fitted by logistic regression. In Proceedings of the 2nd Text REtrieval Conference, Gaithersburg, M.D., USA, pp. 57–66.Google Scholar
Domingos, P. and Pazzani, M. (1997) On the optimality of the simple bayesian classifier under zero-one loss. Machine Learning 29 (2–3): 103–130.CrossRefGoogle Scholar
Efron, B. 1979. Bootstrap methods: another look at the jackknife. Annals of Statistics 7 (1): 1–26.Google Scholar
Frank, E., Trigg, L., Holmes, G. and Witten, I. H. 2000. Naive bayes for regression. Machine Learning 41 (1): 5–25.Google Scholar
Harabagiu, S., Paşca, M., and Maiorano, S. 2000. Experiments with open-domain textual question answering. In Proceedings of the 18th International Conference on Computational Linguistics (COLING 2000), Saarbrücken, Germany, pp. 292–298.Google Scholar
Harabagiu, S., Moldovan, D., Paşca, M., Mihalcea, R., Surdeanu, M., Bunescu, R., Girju, R., Vasile, R., and Morarescu, P. 2001. The role of lexico-semantic feedback in open-domain textual question-answering. In Proceedings of the 39th Annual Meeting of the Association for Computational Linguistics (ACL-2001), Morristown, N.J., USA, pp. 274–281.Google Scholar
Haruno, M., Shirai, S. and Ooyama, Y. 1998. Using decision trees to construct a practical parser. In Proceedings of the joint 17th International Conference on Computational Linguistics and 36th Annual Meeting of the Association for Computational Linguistics (COLING-ACL), Montreal, Canada, pp. 1136–1142.Google Scholar
Hermjakob, U. 2001. Parsing and question classification for question answering. In Proceedings of the ACL 2001 Workshop on Question Answering, Morristown, N.J., USA, pp. 17–22.Google Scholar
Ittycheriah, A., Franz, M., Zhu, W., Ratnaparkhi, A., and Mammone, R. 2001. Question answering using maximum entropy components. In Proceedings of the 2nd Meeting of the North American Chapter of the Association for Computational Linguistics (NAACL-2001), Morristown, N.J., USA, pp. 33–39.Google Scholar
Keenan, S., Smeaton, A. F. and Keogh, G. 2001. The effect of pool depth on system evaluation in TREC. Journal of the American Society for Information Science and Technology 52 (7): 570–574.Google Scholar
Lee, C., Chen, R., Kao, S. and Cheng, P. 2009. A term dependency-based approach for query terms ranking. In Proceedings of the 18th ACM Conference on Information and Knowledge Management (CIKM), Hong Kong, China, pp. 1267–1276.Google Scholar
Li, X. and Roth, D. 2006. Learning question classifiers: the role of semantic information. Natural Language Engineering 12 (3): 229–249.CrossRefGoogle Scholar
Lin, D. 1998. Dependency-based evaluation of Minipar. In Proceedings of the Workshop on the Evaluation of Parsing Systems, Granada, Spain, pp. 317–330.Google Scholar
Lin, D. and Pantel, P. 2001. Discovery of inference rules for question-answering. Natural Language Engineering 7 (4): 343–360.Google Scholar
Lita, L. V. and Carbonell, J. 2004. Unsupervised question answering data acquisition from local corpora. In Proceedings of the Thirteenth Conference on Information and Knowledge Management (CIKM 2004), Washington, D.C., USA, pp. 607–614.CrossRefGoogle Scholar
Mayfield, J. and McNamee, P. 2005. JHU/APL at TREC 2005: QA retrieval and robust tracks. In Voorhees, E. M., and Buckland, L. P. (eds.), Proceedings of the Fourteenth Text REtrieval Conference (TREC 2005). NIST Special Publication: SP 500-266, Gaithersburg, M.D., USA.Google Scholar
Mitra, M., Singhal, A. and Buckley, C. 1998. Improving automatic query expansion. In Proceedings of the 21st Annual International ACM SIGIR Conference on Research and Development in Information Retrieval, Melbourne, Australia, pp. 206–214.CrossRefGoogle Scholar
Monz, C. 2003. Document retrieval in the context of question answering. In Sebastiani, F. (ed.), Proceedings of the 25th European Conference on Information Retrieval Research (ECIR-03), LNCS 2633, pp. 571–579. Springer.Google Scholar
Paşca, M. 2001. High-Performance Open-Domain Question Answering from Large Text Collections. PhD thesis, Southern Methodist University.Google Scholar
Quinlan, J. R. 1992. Learning with continuous classes. In Proceedings of the 5th Australian Joint Conference on Artificial Intelligence, Singapore, pp. 343–348.Google Scholar
Quinlan, J. R. 1993. C4.5: Programs for Machine Learning. Morgan Kaufmann.Google Scholar
Radev, D., Qi, H., Zheng, Z., Blair-Goldensohn, S., Zhang, Z., Fan, W., and Prager, J. 2001. Mining the web for answers to natural language questions. In Proceedings of the tenth International Conference on Information and Knowledge Management (CIKM), Atlanta, Georgia, USA, pp. 143–150.Google Scholar
Ravichandran, D. and Hovy, E. 2002. Learning surface text patterns for a question answering system. In Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics, Morristown, N.J., USA, pp. 41–47.Google Scholar
Robertson, S., Walker, S. and Beaulieu, M. 1998. Okapi at TREC-7: automatic ad hoc, filtering, VLC and interactive. In The 7th Text REtrieval Conference (TREC 7), Gaithersburg, M.D., USA: NIST Special Publication 500-242, pp. 253–264.Google Scholar
Robnik-Šikonja, M., and Kononenko, I. 1997. An adaptation of relief for attribute estimation on regression. In Fisher, D. (ed.), Proceedings of 14th International Conference on Machine Learning ICML'97, pp. 296–304, San Francisco, C.A.Google Scholar
Robnik-Šikonja, M., and Kononenko, I. 2003. Theoretical and empirical analysis of ReliefF and RReliefF. Machine Learning 54 (1–2): 23–69.Google Scholar
Roberts, I. and Gaizauskas, R. 2004. Evaluating passage retrieval approaches for question answering. In Proceedings of the 26th European Conference on Information Retrieval (ECIR'04), Sunderland, UK, pp. 72–84.Google Scholar
Sampson, G. 1995. English for the Computer—The SUSANNE Corpus and Analytic Scheme. Clarendon Press.Google Scholar
Santorini, B. 1990. Part-of-speech tagging guidelines for the Penn Treebank, 3rd rev., 2nd ed. Department of Computer Science, University of Pennsylvania.Google Scholar
Schmid, H. 1994. Probabilistic part-of-speech tagging using decision trees. In Proceedings of International Conference on New Methods in Language Processing, Manchester, UK.Google Scholar
Singhal, A., Salton, G., Mitra, M. and Buckley, C. 1996. Document length normalization. Information Processing & Management 32 (5): 619–633.Google Scholar
Collins-Thompson, K., Callan, J., Terra, E., and Clarke, C. 2004. The effect of document retrieval quality on factoid question answering performance. In Proceedings of the 27th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval, Sheffield, UK, pp. 574–574.Google Scholar
Voorhees, E. 1994. Query expansion using lexical-semantic relations. In Proceedings of the 37th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval, Dublin, Ireland, pp. 61–69.Google Scholar
Wang, Y. and Witten, I. H. 1997. Induction of model trees for predicting continuous classes. In Proceedings of the Poster Papers of the European Conference on Machine Learning (ECML), Prague, Czech Republic, pp. 128–137.Google Scholar
Wilbur, W. J. 1994. Nonparametric significance tests of retrieval performance comparisons. Journal of Information Science 20 (4): 270–284.Google Scholar
Williams, E. 1981. On the notions ‘lexically related’ and ‘head of a word’. Linguistic Inquiry 12: 245–274.Google Scholar
Witten, I. H. and Frank, E. 2005. Data Mining: Practical Machine Learning Tools and Techniques with Java Implementations, 2nd ed.Morgan Kaufmann.Google Scholar
Yousefi, J. and Kosseim, L. 2006. Automatic acquisition of semantic-based question reformulations for question answering. In Proceedings of the Seventh International Conference on Intelligent Text Processing and Computational Linguistics (CICLing), Mexico City, Mexico, pp. 441–452.Google Scholar