A lot of models have been proposed for using deep-learning for comparing text, but they don’t tend to work for the specific task of retrieval. The paper makes the argument that this is because in relevance, we aren’t as interested in the very “soft” match between terms that typical NLP models would be. Java and C++ are programming languages, but when I search for a “Java Tutorial”, I don’t want C++ tutorials instead (my example).
In particular, in relevance, we’re very interested in specific (query-term, document-term) interactions, as opposed to building some compressed model of the query and document, and computing a correlation. While there are previous models (DeepMatch, MatchPyramid) that capture these local interactions, the authors argue that they don’t do enough to separate out the effect of exact matches vs. soft matches.
The main contribution of the paper appears to be the authors use of histogram features to capture a rich set of query-term, document-term interactions. For each query term, they build a histogram of counts of distance(query-term, doc-term) term. A separate bucket is maintained for exact matches. (Terms are represented by their word2vec embeddings, trained on the input datasets).
I want to like this paper. The histogram bucketing appears to be a clever technique to preserve local interactions without exploding the number of parameters a network has to work with. Unfortunately, the evaluation seems weak, despite it’s apparent thoroughness: they didn’t evaluate one of the original claims of the paper (the importance of exact matches). There also isn’t any evaluation of n-grams, which tend to be an extremely powerful ranking signal. This makes it harder to compare against models which do try to handle n-grams (e.g. CNNs). The small size of the training set would almost certainly require some pre-training step, but that could be interesting on it’s own.
Overall, I think the approach is interesting, and hopefully gets pushed on further; using NNs for relevance matching is still in it’s infancy; any reasonable ideas are worth investigating further.