nip4e: decoupling encryption from identity

[fiatjaf]

this is inspired by MLS, but much simpler, and definitely not trying to be a group communication system, but only a way for users to encrypt things to themselves.

https://github.com/nostr-protocol/nips/blob/per-device-keys/4e.md

[vitorpamplona]

+1 for decoupling signing and encryption. The security requirements for signing events are different—not necessarily better or worse, just different—from those for encrypting and decrypting content. In my opinion, Nostr Signers shouldn't even bother with or control access to encryption/decryption.

I’m not sure if it makes sense to create "device keys" for lists and other kinds that are used between devices (private follow lists for instance). On top of that, the concept of attaching keys to "devices" is fundamentally flawed on Nostr, as multiple clients can run on the same device. In this PR, would they share the same key or would each app create their own?

Many clients now support multiple accounts, which complicates the user’s mental model. It becomes challenging to explain which clients can access specific sub keys for each account. Maintaining a clear understanding of which apps have authorization to perform specific actions is difficult even before introducing rotating keys.

If we are to operate with multiple encryption keys, we MUST include the key ID in each encrypted payload. This allows the receiving client to identify the correct key or prompt the user to provide it.

The challenge of determining which keys can encrypt/decrypt a given payload (or versions in a replaceable) was a key issue in our decision to place the keys directly in the event in 1228. That design separates signing and encryption without requiring the maintenance or resynchronization of key lists, neither on Nostr nor externally. In the same way, users don't even know the event is using a separate key to encrypt and don't need to keep track on which apps have which keys.

I don’t know which approach is better. But I am confident that signing and encryption should use separate keys—at the least one per kind / app usecase type. In more extreme cases, such as spreadsheets, a new key could be generated for each group collaborating on an event.

[fiatjaf]

Sorry, I wrote "device key" just because I thought that would be clearer, but actually a "device" here means a "client".

The challenge of determining which keys can encrypt/decrypt a given payload (or versions in a replaceable) was a key issue in our decision to place the keys directly in the event in 1228. That design separates signing and encryption without requiring the maintenance or resynchronization of key lists, neither on Nostr nor externally. In the same way, users don't even know the event is using a separate key to encrypt and don't need to keep track on which apps have which keys.

I don't get this. Granted I didn't read that NIP with my full attention, but how can you decrypt a thing without having a key? Or how else could we apply that technique to solve the current problem?

I don’t know which approach is better. But I am confident that signing and encryption should use separate keys—at the least one per kind / app usecase type. In more extreme cases, such as spreadsheets, a new key could be generated for each group collaborating on an event.

This is all fine but the problem remains that somehow different clients ("devices") have to learn what is the relevant encryption key -- how to do that (without relying on user identity keys for encryption) is the real question. This NIP is a proposal to address that.

[mikedilger]

I'm not a cryptographer but this looks good to me.

[fiatjaf]

I made this amazing artistically inspired diagram that shows that the process of sharing keys between devices can happen seamlessly without much user hassle:

remember that "device" actually means "client" or "app", but I think "device" sounds better and makes everything look more cryptographically sound.

[reyamir]

Hi @fiatjaf , I'm starting try implement this NIP for Coop, one part confuse me is

3. device A also creates the user-level keypair to be used for encryption and publishes its public part

Is this user's keys (created elsewhere, example: start.njump.me)? or device must created both keys, one for internal, one for user?

In the NIP file, I only see mention about random encryption key in this part:

The first device to come into the world will generate a random encryption key.

[fiatjaf]

@reyamir first: no one has implemented this, and this NIP is very much in proposal stage, so whatever you do here will probably be better than my text, feel free to tweak things according to what works better.

---

The idea is that there will be a key just for nip44 encryption. That key will probably not be created by start.njump.me, odds are that it will be created by Coop, or Flotilla, or Amethyst, or 0xchat (some client that deals with nip44 encryption), and then shared around.

Then that key must be announced so other users know they should encrypt using that in order to message such user (instead of encrypting to their "identity" pubkey).

A more concrete example

(Forgive me if this is unnecessarily verbose, but it may help others understand my point in the future.)

1. Alice creates a keypair (a, A) (a is the secret key, A is the public key) on start.njump.me.
2. A is Alice's main identity on Nostr, her npub will be, say, npub1A;
3. Alice installs Coop, Coop somehow realizes Alice can't use her a secret key for encryption because it's behind a FROST bunker, so Coop creates an encryption keypair (e, E). This doesn't change Alice's identity, it will only be used for encryption.
4. Coop publishes an event (let's say it's kind:10044) to announce this to the world:

{
  "kind": 10044,
  "pubkey": "<A>",
  "tags": [
    ["n", "<E>"] // `n` is for "encryption", doesn't matter
  ]
}

5. Now Bob (keypairs (b, B)) will send a DM to Alice. Because Bob's client fetched Alice's kind:10044 event, instead of computing the conversation key with ecdh(b, A) he does ecdh(b, E) = S
6. Because Alice knows e Alice can decrypt Bob's message doing ecdh(e, B) = S and all is good
7. Now the fun part starts: Alice has decided to use Flotilla to chat on her phone, and Flotilla wants to do encryption stuff.
8. Flotilla sees that Alice has a kind:10044 published, which means Flotilla won't create a new key, Flotilla will have to ask for Coop to share that key securely. So Flotilla generates a local keypair (f, F) that won't be shown or leave the device ever, and Flotilla publishes an announcement (let's say it's kind:4454) for that local key (signed by Alice):

{
  "kind": 4454,
  "pubkey": "<A>",
  "tags": [
    ["client", "Flotilla on Android"],
    ["pubkey", "<F>"]
  ]
}

9. Flotilla cannot proceed without known the secret key e, so it has to tell the user to turn Coop on.
10. Alice opens up Coop and Coop immediately looks for all kind:4454 events from Alice, and sees that there is this app called "Flotilla on Android" signed by Alice herself, so Coop publishes the secret key e nip44-encrypted to ecdh(c, F) -- in which c is the secret key of a keypair that Coop has just generated locally. Coop does that using a new event kind, say kind:4455:

{
  "kind": 4455,
  "pubkey": "<A>"
  "tags": [
    ["P", "<C>"]
    ["p", "<F>"]
  ],
  "content": nip44(e)
}

12. Immediately Flotilla wakes up and sees the kind:4455 that had just been published by Coop, decrypts the content using ecdh(f, C) and now Flotilla also knows the secret key e. Flotilla can now decrypt and encrypt the same things Coop could before.

---

I'll change the NIP text to this, I think it's better.

[pablof7z]

This is very useful.

I think the 10044 should include the client name(s) that have access to the private key so client B can tell the user "open client A".

Also, perhaps clients that have access to the private key could register a schema in the device for deeplinking-communication.

nek+[user-pubkey]:// -- that way client B can query the mobile asking "is there an app that handles nek (nostr-encryption-key) for pubkey X? if so, open it passing in a signed kind:4454 and a callback URI on the query string and client A sends back the kind:4455 to client B's callback URI.

Effectively that would mean

• user has client A in their phone
• they download client B
• client B sees that some other app has the key locally
• they hit one button
• client A shows up and immediately goes back to client B
• now client B has the key.

(this is basically the same idea as https://github.com/nostr-protocol/nips/pull/1777)

[fiatjaf]

Another reason why this is useful even if you're not doing the bunker stuff: https://njump.me/nevent1qqsx880slhhsg0u53te2u8mkgq28y7dke9z2u2mxa5w629fx9pv893qppemhxue69uhkummn9ekx7mp0qywhwumn8ghj7mn0wd68ytnzd96xxmmfdejhytnnda3kjctv9uq32amnwvaz7tmwdaehgu3wdau8gu3wv3jhvtcjqk59m

[vitorpamplona]

Reading this again... Few questions:

1. Do we want to do just one key for all encryptions? I can see two different things: (i) encryptions for my own, like NIP-51 lists, drafts, etc that I need to have decryption power in every client; and (ii) DM encryptions where I only want in a few apps/devices and where that key needs to be advertised to others. I don't need my DM key in every client. That could enhance security.

2. Do we want DMs per device or not? We could have just one active DM key for all DM clients that I use. Otherwise, if I use 3 devices, should I get 3 separate keys that never leave that device? And if so, should everybody else add my 3 keys to the NIP-17 dms when messaging me? In that way, I can see the conversation in every DM client. Or if we don't want that, how do I tell which client I'd like to receive DMs at this moment to my friends?

3. How can we make it easier for people to back up their accounts since now the nsec alone is not enough anymore?

[fiatjaf]

1. In the beginning I was thinking that this method could be used for sharing various types of keys indeed, but now I kinda lost my hope that that will happen because the complexity is too large, it's probably easier to just share one global key and then it gets used wherever NIP-44 was going to be invoked, the code impact should be very minimal in this case. However once we get this scheme working we can see if we want to reuse it for other types of keys.
2. No, you share one encryption keypair among your devices, all your devices know these keys and other people only have to encrypt to that key and not learn about how many devices you have.
3. For people that have an nsec it shouldn't be hard to use it to encrypt the encryption key and save it to relays once (or once every time the encryption key is changed).

[reyamir]

Hi @vitorpamplona , I wonder are you interested in implement this NIP for Amethyst?, I've done it for Coop, but I cannot test much because I'm only one implement this NIP. If Amethyst do it, we can test together and push this further.

[mikedilger]

Will this require a new DM standard with signalling that it uses the encryption key derived this way? If one client does this and encrypts e.g. bookmarks under it, does that lock out other clients that don't follow this NIP yet? How do we migrate to avoid this?

[reyamir]

I have the same concern. Despite having implemented this NIP for Coop, I was able to do so because Coop is a brand new app, allowing me to implement anything I want. For existing clients, the work might be too much.

Maybe we can split it into 2 stages?

1. Encourage all clients which use NIP44 to implement this NIP, but not involve the encryption/decryption aspects yet.
2. After all major clients have implemented it (maybe 3 or 4?), then we start doing encryption/decryption with device keys.

[fiatjaf]

Will this require a new DM standard with signalling that it uses the encryption key derived this way? If one client does this and encrypts e.g. bookmarks under it, does that lock out other clients that don't follow this NIP yet? How do we migrate to avoid this?

Clients have to check if the receiver has announced an encryption key and encrypt to that if they do, otherwise encrypt to their identity key as always. That should be a very simple change and doesn't actually break anything as the receivers wouldn't have received the message otherwise anyway.

[dentropy]

Reading this makes Nostr feel a lot more like Matrix Protocol, just missing the one time pads

[vitorpamplona]

I am concerned about race conditions, offline relays, and other client bugs that might delete keys and make the encrypted information irrecoverable. In a similar way, when the key has been rotated multiple times, it will be virtually impossible to know for which key was each encryption encrypted to.

What if instead of generating new random keys and having to store them separately and running the risk of losing them, we generate random 32-byte nonces that are supposed to be applied to the nsec and returned as a new key? Our signers could expose that method and in that way apps could always recover keys.

Then on 10044, we log the nonce.

{
  "kind": 10044,
  "pubkey": "<A>",
  "tags": [
    ["n", "<E>", "<nonce to produce the new key from nsec>"] 
  ]
}

To generate any key, the signer just uses this method from NIP-44 and return the new key to the app.

sha256(hkdf(private_key, salt: 'nip4e' + '<nonce>'))

On each encryption, we encrypt to E but log the nonce itself in the message. In that way, we always know which nonce each encryption used. We could do that as an extension to NIP-44, as v3, or as an extra tag on the event. I would prefer a v3 on NIP-44.

Then there is no need to send keys between devices (ie.. 4455, 4454 are not needed). The signer decides which app gets which keys. If they need to share a single key, so be it.

In this way, there is no way to accidentally lose the information on an encrypted blob. Even if 10044 gets completely wiped, all the individual encryptions can still be recovered.

[fiatjaf]

This kills the primary use case I had in mind: multisig signers.

Also I don't understand this fear of losing information, that kind of thing never happens, in fact it's much more likely that you'll keep information you don't want around than it is that you'll lose information you want. Also it's much more likely that you'll lose the encrypted messages themselves than that you'll lose the encryption key.

Also, shit happens, You cannot prevent it absolutely. I just lost a giant chat history I had on Telegram for reasons yet unknown, I also lost all my Signal chats and my Simplex chats because I didn't take 5 minutes to correctly migrate them.

And keys aren't meant to be rotated all the time, that should be more of a possibility in case of catastrophes than a recommended common behavior. If your use case requires rotating keys all the time then need the ratchet spec or the MLS spec instead.

[vitorpamplona]

This kills the primary use case I had in mind: multisig signers.

Why? The multisig needs to give permission to each device to decrypt. You can create one nonce, use the main nsec of the multisig to derive a new key and transfer that key to each application to decrypt. It's the same procedure, but without random keys.

[vitorpamplona]

During my draft of #1838 I realized that you need to document how the NIP-44's conversation key was assembled: which key of the sender was used as private key and which key of the receiver was used as pubkey. Otherwise, you can't rebuild the conversation key. You would have to try all possible pairs of the receiver's and sender's kind 10044 + their main keys

[fiatjaf]

It's not, some multisig setups may never have access to the nsec because it's generated in a distributed way by a pool of signers none of which ever get to know the main nsec.

Also in the current frost bunker implementation the nsec is generated once by the client and sharded, but afterwards it's meant to be kept cold by the user basically forever after, with signers cooperating to generate signatures afterwards but never reassembling the nsec.

[vitorpamplona]

Also in the current frost bunker implementation the nsec is generated once by the client and sharded

That's what I am thinking. At the same time, a nonce should be created and an encryption key generated. That encryption key is then shared with all participants.

The group can also create another random nsec to serve as the key-derivation master key for encryption.

[fiatjaf]

Oh! Indeed it doesn't have to be the nsec, it can be anything!

Very interesting. This makes the process much easier for everybody.

[mikedilger]

Clients have to check if the receiver has announced an encryption key and encrypt to that if they do, otherwise encrypt to their identity key as always. That should be a very simple change and doesn't actually break anything as the receivers wouldn't have received the message otherwise anyway.

That answers the first question. But the second one remains:

If one client does this and encrypts e.g. bookmarks under it, does that lock out other clients that don't follow this NIP yet?

If new encryption rolls out and my bookmarks get encrypted by Amethyst, and let's say gossip didn't do it yet, then gossip can't read my bookmarks. So I'm thinking there might be a way to store the bookmarks encrypted both ways somehow.

[mikedilger]

Oh! Indeed it doesn't have to be the nsec, it can be anything!

Anything that is secret and available across clients. A "shared secret".

In the case of #1837, devices don't have the same signing keypairs. And the signing keypairs are not derived (because current nostr keypairs need to be included). So that also makes it difficult to derive from an nsec. But I suppose I could come up with a way to share a secret amongst them, that would be different from the multisig signer case.

[fiatjaf]

If new encryption rolls out and my bookmarks get encrypted by Amethyst, and let's say gossip didn't do it yet, then gossip can't read my bookmarks. So I'm thinking there might be a way to store the bookmarks encrypted both ways somehow.

I think this is a very narrow use case that is only likely to affect power users who will know how to deal with it.