While I was trying to figure out which paper to read this weekend, I got the answer to one question I have had for ages - how do cloud storage providers detect and delete duplicate files even when they are encrypted?
The worst they could do is to decrypt and compare, but that would be a huge lapse in security. Digging deeper I found they use an encryption scheme called Convergent Encryption, also called content hash keying, which is both simple and elegant.
The core idea is to produce identical encrypted output from identical plain data; and this is achieved by deriving encryption keys from the content instead of using a separate, randomly generated key. So, the flow is
- apply a hash function to plain data and generate the hash value
- use this hash value to derive a specific encryption key deterministically
- encrypt the pain data with this encryption key
- keep the hash or the key secured with some party, if required
With this approach, the identical files will produce the same encryption key, resulting in identical encrypted data. Now that we can de-duplicate redundant copies, we can leverage this to make storage and transmission efficient, without compromising on security.
Given that the encryption key is derived from the data itself, so there is no need to have a central provider and maintainer of original encryption keys, making the entire design simple, scalable, and much more secure.
I stumbled upon Convergent Encryption while I was going through a super interesting paper. a detailed write-up coming soon on my website.
ps: admissions for my Oct cohort are open, if you are looking for a no-fluff and highly practical system design course, checkout
arpitbhayani.me/course