How much private information do text em- beddings reveal about the original text? Researchers investigate the problem of embedding inversion, reconstructing the full text represented in dense text embeddings. They frame the problem as controlled generation: generating text that, when reembedded, is close to a fixed point in latent space. They find that although a naïve model conditioned on the embedding performs poorly, a multi-step method that iteratively cor- rects and re-embeds text is able to recover 92% of 32-token text inputs exactly. They train their model to decode text embeddings from two state-of-the-art embedding models, and also show that their model can recover important personal information (full names) from a dataset of clinical notes.
Feedback Figure 3: Recovery performance across multiple rounds of self-correction comparing models with access to vs text-only (32 tokens per sequence). 7 Analysis How much does the model rely on feedback from ? Figure 3 shows an ablation study of the importance of feedback, i.e. performing corrections with and without embedding the most recent hypothesis. The model trained with feedback (i.e. additional conditioning on (x(t)) is able to make a more accurate first correction and gets better BLEU score with more rounds. The model trained with no feedback can still edit the text but does not receive more information about the geometry of the embedding space, and quickly plateaus. The most startling comparison is in terms of the number of exact matches: after 50 rounds of greedy selfcorrection, our model with feedback gets 52.0% of examples correct (after only 1.5% initially); the model trained without feedback only perfectly matches 4.2% of examples after 50 rounds.
id: be021d7a3381a07b53af5ec3fb5b9b34 - page: 7
During training, the model only learns to correct a single hypothesis to the ground-truth sample. Given new text at test time, our model is able to correct the same text multiple times, pushing the text from 0.9 embedding similarity to 1.0. We plot the closeness of the first hypothesis to the groundtruth in the training data for the length-32 model in Figure 4. We see that during training the model learns to correct hypotheses across a wide range of closenesses, implying that corrections should not go out-of-distribution as they approach the ground-truth. How informative are embeddings for textual recovery? We graph BLEU score vs. cosine similarity from a selection of of reconstructed text inputs in Figure 5. We observe a strong correlation between the two metrics. Notably, there are very 0.700.750.800.850.900.951.00Cosine distance from hypothesis to true embedding050001000015000200002500030000Count
id: 368a5c59bd35c3fe1b739b35dbefa198 - page: 7
Figure 4: Distribution of cos(e, (x(0))) over training data. The mean training output from the GTR base model has a cosine similarity of 0.924 with the true embedding. few generated samples with high cosine similarity but low BLEU score. This implies that better following embedding geometry will further improves systems. Theoretically some embeddings might be impossible to recover. Prior work (Song et al., 2020; Morris, 2020) has shown that two different sequences can collide in text embedding space, having similar embeddings even without any word overlap. However, our experiments found no evidence that collisions are a problem; they either do not exist or our model learns during training to avoid outputting them. Improved systems should be able to recover longer text.
id: f332b04fb0da1961a6922be71295e8cb - page: 7
Does having a strong base model matter? We ablate the impact of initialization by evaluating our 32-token Wikipedia model at different initializations of x(0), as shown in Section 7. After running for 20 steps of correction, our model is able to recover from an unhelpful initialization, even when the initialization is a random sequence of tokens. This suggests that the model is able to ignore bad hypotheses and focus on the true embedding when the hypothesis is not helpful.
id: bd23017d61bf6385de095409d4e619f8 - page: 7