MeaningBERT EU AI Act Compliance Profile

Here is MeaningBERT

MeaningBERT is an automatic and trainable metric for assessing meaning preservation between sentences. MeaningBERT was proposed in our article MeaningBERT: assessing meaning preservation between sentences. Its goal is to assess meaning preservation between two sentences that correlate highly with human judgments and sanity checks. For more details, refer to our publicly available article.

This public version of our model uses the best model trained (where in our article, we present the performance results of an average of 10 models) for a more extended period (500 epochs instead of 250). We have observed later that the model can further reduce dev loss and increase performance. Also, we have changed the data augmentation technique used in the article for a more robust one, that also includes the commutative property of the meaning function. Namely, Meaning(Sent_a, Sent_b) = Meaning(Sent_b, Sent_a).

• HuggingFace Model Card
• HuggingFace Metric Card

Sanity Check

Correlation to human judgment is one way to evaluate the quality of a meaning preservation metric. However, it is inherently subjective, since it uses human judgment as a gold standard, and expensive since it requires a large dataset annotated by several humans. As an alternative, we designed two automated tests: evaluating meaning preservation between identical sentences (which should be 100% preserving) and between unrelated sentences (which should be 0% preserving). In these tests, the meaning preservation target value is not subjective and does not require human annotation to be measured. They represent a trivial and minimal threshold a good automatic meaning preservation metric should be able to achieve. Namely, a metric should be minimally able to return a perfect score (i.e., 100%) if two i

Your obligations as a MeaningBERT deployer

Track your progress

Sign up to track obligations, receive deadline reminders, and generate audit packages.

Tags

Library