Response Accuracy Retention Index (RARI) — Evaluating Impact of Data Masking on LLM Response

As language models (LLMs) in enterprise applications continue to grow, ensuring data privacy while maintaining response accuracy becomes crucial. One of the primary methods for protecting sensitive information is data masking. However, this process can lead to significant information loss, potentially rendering responses from LLMs less accurate. How can this loss be measured? Here, the Response Accuracy Retention Index (RARI) is a metric designed to measure how well an LLM retains its accuracy after data masking.

RARI — Response Accuracy Retention Index

What is RARI?

RARI — Response Accuracy Retention Index — is a metric that quantifies how closely the responses generated by a language model on masked data align with those generated on unmasked, clear-text data. In simple terms, RARI measures the retention of response accuracy after sensitive information has been obscured through masking.

The Problem with Data Masking

Data masking involves altering sensitive information, such as Personally Identifiable Information (PII), to hide the original details, enhance security, and protect privacy. This process is essential when data is used for AI model training, Retrieval Augmented Generation (RAG), or prompts/responses.

However, a major downside of masking is that it can render the data incoherent, leading to a significant loss of context. For instance, replacing names with random strings or redacting key details can strip the data of the essential information that a language model relies on to generate accurate responses. This loss of context can result in inaccurate, misleading, or irrelevant outputs from the model.

That said, not all data masking hurts data utility and LLM responses. Some emerging technologies, like Protecto.ai, have developed unique masking techniques that can preserve up to 99% of LLM response accuracy, ensuring that the masking process does not come at the cost of model performance.

How RARI Works

RARI aims to quantify the impact of data masking on LLM accuracy by following these steps:

1. Original Response Generation: Start with a prompt and the original, unmasked context text. This prompt and context are sent to the LLM, which generates a response based on the clear-text data.
2. Masked Response Generation: The same prompt and context are then subjected to data masking, obscuring sensitive details. This masked version is fed into the LLM to generate a second response. After receiving the response with the masked data, any placeholders or masked elements in the output are “unmasked”.
3. Similarity Search: Finally, the original (step 1) and masked responses (step 2) are compared using similarity metrics, such as cosine similarity. The degree to which these responses match gives us the RARI score.

Interpreting the RARI Score

The RARI score is expressed as a percentage with a corresponding similarity value:

• 100% (Cosine Similarity = 1): The original and masked data responses are identical in meaning, which indicates that the data masking process has not caused any loss in response accuracy.
• Lower RARI Scores: As the RARI score decreases, it signifies greater divergence between the original and masked responses. A low RARI score indicates that the data masking process has significantly degraded the accuracy of the LLM’s response, highlighting potential issues with the masking technique used.

Example 1: High RARI Score

Prompt: “Who reported a problem? Who worked on it? “

Original

• Context: “Customer support agent Sarah and her manager Ava provided a resolution to a problem reported by John Smith.”
• Response 1: “John Smith reported a problem. Sarah and Ava worked on it.”

Masked

• Context: “Customer support agent A123 and her manager A345 provided a resolution to a problem reported by K678.”
• Response 2 (Masked): “K678 reported a problem that A123 and A345 helped resolve.”
• Unmasked Response 2 for Comparison: “John Smith reported a problem that Sarah and Ava helped resolve.”

High RARI Score: 98%+ The responses are identical in meaning after unmasking, indicating no loss in response accuracy due to the masking.

Example 2: Low RARI Score

Prompt: “Who resolved the customer’s problem? “

Original

• Context: “John Smith wasn’t happy about customer support agent Sarah. Later, she and her manager Ava resolved a problem reported by John.”
• Response 1: “Ava had to help Sarah to resolve the problem.”

Masked

• Context: “J123 wasn’t happy about customer support agent S234. Later, she and her manager A234 resolved a problem reported by S123.”
• Response 2(Masked): “J123 reported a problem that wasn’t resolved.”
• Unmasked Response 2 for Comparison: “John Smith reported a problem that wasn’t resolved.”

Low RARI Score: 30% — The masked response failed to retain the key details about who attended the meeting, resulting in a significant loss of accuracy.

Why RARI Matters

The accuracy of responses generated by language models (LLMs) is critical. Inaccurate answers could lead to significant consequences, such as financial loss, low feature adoption, and customer churn. Achieving high accuracy in LLM responses requires substantial effort, including fine-tuning models and optimizing data inputs.

Data masking, while essential for privacy and security, should not undermine this hard-won accuracy. If the masking process introduces errors or obscures critical context, it can derail an entire project, rendering the LLM’s outputs unreliable.

In the AI-first world, RARI has become invaluable. RARI offers the best way to compare various data masking tools and techniques, enabling you to choose a solution that protects sensitive information and preserves the accuracy of your AI systems. With RARI, you ensure that you’re not compromising accuracy to gain privacy and security, thereby maintaining the integrity and effectiveness of your AI-driven initiatives.