Mutual Implication Score EU AI Act Compliance Profile

Model overview

Mutual Implication Score is a symmetric measure of text semantic similarity based on a RoBERTA model pretrained for natural language inference and fine-tuned on a paraphrase detection dataset.

The code for inference and evaluation of the model is available here.

This measure is particularly useful for paraphrase detection, but can also be applied to other semantic similarity tasks, such as content similarity scoring in text style transfer.

How to use

The following snippet illustrates code usage:

!pip install mutual-implication-score

from mutual_implication_score import MIS
mis = MIS(device='cpu')#cuda:0 for using cuda with certain index
source_texts = ['I want to leave this room',
                'Hello world, my name is Nick']
paraphrases = ['I want to go out of this room',
               'Hello world, my surname is Petrov']
scores = mis.compute(source_texts, paraphrases)
print(scores)
# expected output: [0.9748, 0.0545]

Model details

We slightly modify the RoBERTa-Large NLI model architecture (see the scheme below) and fine-tune it with QQP paraphrase dataset.

Performance on Text Style Transfer and Paraphrase Detection tasks

This measure was developed in terms of large scale comparison of different measures on text style transfer and paraphrase datasets.

The scheme above shows the correlations of measures of different classes with human judgments on paraphrase and text style transfer datasets. The text above each dataset indicates the best-performing measure. The rightmost columns show the mean perform

Your obligations as a Mutual Implication Score deployer

Track your progress

Sign up to track obligations, receive deadline reminders, and generate audit packages.

Tags

Languages

Library