Term embeddings for IR

Unsupervised term embeddings can be utilized in 2 main ways:

Query-document matching directly in the latent space

• Embedding based models often perform poorly when the retrieval is performed over the full document collection. For example, the model may know that a "passage" in text is about fruit, but it fails to realize that the passage is about banana

Telescoping

• IDEA: Chain different IR models where each successive model re-ranks a smaller number of candidate documents
• PRACTICE: Use the embedding based model to re-rank only a subset of the documents retrieved by a different IR model.

Query expansion

• Use embeddings to find good expansion candidates from a global vocabulary, and then retrieving documents using the expanded query.

Learning to rank: a supervised tale

LTR for IR uses relevance information, such as labels and click data as training data to improve the pertinence of the ranking

• Rankable item

  Query-document pairs, represented by feature vectors \( \bar{x} \)

  Traditional models use represented hand-crafted features like:

  • - Query-independent or static features (e.g., query length/document length)
  • - Query-dependent or dynamic features (e.g., BM25)
  
  

  Modern models use deep architecture for feature learning, starting from a simple vector representations of the input.

  Additional parameters, such as the document popularity, could be used in the process

• Ranking model

  Trained to map the vector to a real-valued score s.t. for a given query the more relevant documents are scored higher and some chosen rank-based metric is maximized

  Depending on the input several algorithm can be used: NN, SVM, Trees etc.

• A LOSS function

  Depending on the type of relevance (how it is encoded) can be calculated as:

  • - Regression loss when the relevance information is numerical; a standard loss function, such as the square loss, can be employed.

    $$ L_{squared}=||True - Predicted||^2$$

  • - Classification loss when the relevance information are labels: the probability of the correct label for the query/document pair \( \bar{x} \) can be obtained by the softmax function which normalizes the score of the correct label against the set of all possible labels.

About the INPUT...

Neural models that learn representations of text take raw text as input:

• Character level:

  Each character is typically represented by a one-hot vector. In this setting text can be derived by either aggregate or summing the char-level vector

• Term level:

  Each term is represented by either a sparse vector or using pre-trained term embeddings. Usually, term vectors can be derived by aggregating one-hot vectors of its constituting characters (or character n-graphs). Text is then derived from the concatenation/sum.

Character Level

Bag-of-trigraphs per term

Term level with pretrained term embeddings

About the MODEL...

Autoencoders

Unsupervised learning model based on the information bottleneck method, the train consists in feeding in the high-dimensional vector inputs and trying to reconstruct the same representation at the output layer

In the process the parameters that encode & decode the input are adjusted to minimize the squared loss (usually) between the reconstructed output and the input

Variational autoencoders

The encoder generates two separate vectors: means \( \vec{\mu} \) and deviations \( \vec{\sigma} \) (latent space is 2*z).

The latent representation is generated by sampling a gaussian along each of the k latent dimensions.

By sampling the latent representation, we expose the decoder to a certain degree of local variations in its input that should force the model to learn a smoother continuous latent space.

An important application of VAE is for the synthesis of new items or text not observed in the training collection.

Siamese Network

The siamese network's architecture consists of 2 models ( \( m1,m2 \)), that projects 2 inputs ( \( i1,i2 \)) to the same latent space, thus obtaining ( \( v1,v2 \))

The distance(usually cosine similarity) is then computed and the parameters, which are shared, are optimized such that ( \( v1,v2 \)) are closer when when expected to be close & further otherwise.

For example, the features representing all the documents about “banana bread” should be very close to each other, but far away from feature clusters of all other documents.

The LOSS is usually the triplet loss: where a baseline vector \( \vec q \) (query) is compared against a positive(relevant) vector \( \vec p \) (e.g. document clicked by the user) and a negative vector \( \vec n \) (falsy document sampled with uniform probability from the full collection).

Conclusions

• Desiderata of a model

  Of short and long text

  Retrieval of long text: a model must deal with variable length documents where the relevant sections (to the query) may be surrounded by (a lot of) irrelevant text.

  Retrieval of short text: a model must deal with query-document vocabulary mismatch problem, by learning how patterns of query terms and (different) document terms can indicate relevance.

  In either cases, a model should also consider lexical matches when the query contains rare terms (not seen during training) to avoid retrieving semantically related but irrelevant results.

  context

  Ideal IR models should be able to discriminate or rank between documents inferring the meaning of a query from context.

  For example, if one searches for "soccer world cup winner", it is highly probable that what he/she wants is the last edition's winner. And that should be understood by the model via the context or the user’s short or long-term history.

  The need of labelled data

  IR is “a little behind” wrt CV & NLP mostly because it suers heavily (and for good reasons) for lack of annotated (labelled) document collections, for privacy reasons.

• Present problems and future goals

  
  

  "Should the ideal IR model behave like a library that knows about everything in the Universe, or like a librarian who can effectively retrieve without memorizing the corpus"

  Mitra (2018)

  A brute force approach