Add Onion Transport

[umgefahren]

Description

Adding a libp2p transport built on top of the newly released Arti. Currently only dialing is supported, since Onion Services are not supported by Arti yet and it will likely take some time until it's supported, since it's not funded yet.

The implementation should be considered as a first step towards a stable Tor based transport.

Links to any relevant issues

This implementation follows the discussion in #708 and should close this issue.

Open Questions

I added libp2p-onion to the default features. However this might be a bad idea, since arti has a lot of dependencies and therefore increases the build time of libp2p greatly.

Since there is no documentation yet and it's very untested (I might need help designing good tests) libp2p-onion maybe shouldn't be integrated in libp2p at all now.

Change checklist

• [x] I have performed a self-review of my own code
• [ ] I have made corresponding changes to the documentation
• [ ] I have added tests that prove my fix is effective or that my feature works
• [ ] A changelog entry has been made in the appropriate crates

[mxinden]

Very happy to see this happening! Thanks @umgefahren for the work and thanks @thomaseizinger for the reviews. Please ping me in case you need my input anywhere.

[mxinden]

Cross-referencing related effort on the nim-libp2p side: https://github.com/status-im/nim-libp2p/pull/765

//CC @Menduist and @diegomrsantos

[thomaseizinger]

It seems we have a CI failure from cargo deny because it can't interpret some license properly :/

[thomaseizinger]

I added libp2p-onion to the default features. However this might be a bad idea, since arti has a lot of dependencies and therefore increases the build time of libp2p greatly.

There is consensus to remove all the default features and I've finally gotten around to implementing that in https://github.com/libp2p/rust-libp2p/pull/2918 so this shouldn't be a concern really :)

[umgefahren]

It seems we have a CI failure from cargo deny because it can't interpret some license properly :/

It seems like the license of bounded-vec-deque and the license of tinystr seem to have problems. The latter is licensed under Unicode-DFS-2016 which seems like a pretty permisse MIT-like license, but I'm not a lawyer. Although I'm no particular expert, I think we could just add this license to the allowed ones. bounded-vec-deque on the other hand causes me a bit of headache. Not only because the repository linked to, seems to be dead. I could file an issue with Arti describing the issues... although this would probably require writing an alternative.

[thomaseizinger]

It seems we have a CI failure from cargo deny because it can't interpret some license properly :/

It seems like the license of bounded-vec-deque and the license of tinystr seem to have problems. The latter is licensed under Unicode-DFS-2016 which seems like a pretty permisse MIT-like license, but I'm not a lawyer. Although I'm no particular expert, I think we could just add this license to the allowed ones. bounded-vec-deque on the other hand causes me a bit of headache. Not only because the repository linked to, seems to be dead. I could file an issue with Arti describing the issues... although this would probably require writing an alternative.

One can inspect the actual source code here: https://docs.rs/crate/bounded-vec-deque/0.1.1/source/

It is dual-licensed under GPL3 and BSD. You should be able to somehow tell cargo deny, what license it actually is. I think we had to do that for ring too but I am no expert in this tool, sorry :(

Regarding tinystr, I'd like to have @mxinden's thoughts.

In both cases, it may be worth reaching out to the arti developers and letting them know that these two crates are being flagged here. Not necessarily with the goal of having them replaced but it is likely good to know. They don't seem to use cargo deny so they probably just don't know about these crates :)

[mxinden]

I could file an issue with Arti describing the issues... although this would probably require writing an alternative.

In both cases, it may be worth reaching out to the arti developers and letting them know that these two crates are being flagged here. Not necessarily with the goal of having them replaced but it is likely good to know. They don't seem to use cargo deny so they probably just don't know about these crates :)

That would be my preference. @umgefahren would you mind reaching out to them?

In case they decide to stick with the two crates, which of course is very much a valid option, I can try to organize some legal advice to learn more about the special licenses. I myself don't feel knowledgeable enough to make a decision here.

[umgefahren]

How about reaching out to the Arti maintainers, asking whether they would be so kind and give this a review as well? @umgefahren are you in touch with them already? I have been in touch with them in the past, thus happy to do the reach out.

I tried to get in touch with them, however I wasn't successful or didn't try hard enough. So it would be nice, if you could reach out to them for now.

[umgefahren]

Is there anything left to do for me?

[thomaseizinger]

Is there anything left to do for me?

Yes, sorry for the delay!

• Please fix the merge conflicts
• Please adopt our new feature handling (no more default features). See #2918.
• Please add an override of some sort for cargo deny so that it is happy :)
• Please add a changelog file and the necessary entries to the root CHANGELOG.md

[umgefahren]

I implemented all changes requested. I hope you are pleased with my resolution of the issues with cargo deny.

[thomaseizinger]

I implemented all changes requested. I hope you are pleased with my resolution of the issues with cargo deny.

Yep, looks all good! I just pushed a few commits with minor nits, hope you don't mind :) This looks good to be merged!

I did notice that libp2p-onion does not compile without default features because of an issue in arti_client. It references PreferredRuntime without a feature gate but the symbol itself is feature-gated.

This shouldn't be an issue for us though as we only allow users to depend on the crate by selecting one of the two runtimes.

[umgefahren]

Are there any issues left for me to resolve? @mxinden

[mxinden]

:wave: @nmathewson and ijackson,

I hope you don't mind me pinging you here. First off, thank you for your work on https://gitlab.torproject.org/tpo/core/arti. You might remember a past call with me during the early days of Arti.

As you can tell from the pull request description, @umgefahren is integrating Arti with rust-libp2p, thus enabling a rust-libp2p node to dial a remote endpoint through the Tor network.

I am tagging you here for the following reasons:

1. Might be cool for you to see where Arti is being used.
2. In case you have time, your input on this pull request would be very valuable. No worries if you don't have time.
3. Getting your opinion on the license issues we are facing with the import of Arti. More specifically the crates bounded-vec-deque and tinystr. See https://github.com/libp2p/rust-libp2p/pull/2899#issuecomment-1254760056. Again, no worries if you don't have time.

[thomaseizinger]

Can you fix the clippy error please? See https://github.com/libp2p/rust-libp2p/pull/2859 for reference.

[umgefahren]

Done

[umgefahren]

Why did the CI fail?

[thomaseizinger]

Why did the CI fail?

Some download issue during setup of the jobs. I re-triggered them!

[mxinden]

I did not forget about this. I am sorry for the delay.

[umgefahren]

@thomaseizinger could you add the hacktoberfest-accepted label :)

[umgefahren]

Is it ok that the interoperability check always fails? And since the semver-checks were introduced in #2635 the CI doesn't get green anymore sadly.

[mxinden]

I will reach out to our legal team as well, just in case. I don't think we need to block on this.

No objections from our legal team.

[mxinden]

Is it ok that the interoperability check always fails? And since the semver-checks were introduced in #2635 the CI doesn't get green anymore sadly.

Interoperability tests are back to green with https://github.com/libp2p/test-plans/pull/60/.

[thomaseizinger]

I will reach out to our legal team as well, just in case. I don't think we need to block on this.

No objections from our legal team.

On what in particular? Removing the licence headers in each file?

[mxinden]

I will reach out to our legal team as well, just in case. I don't think we need to block on this.

No objections from our legal team.

On what in particular? Removing the licence headers in each file?

No. Regarding the two crates with licensing concerns.

[marten-seemann]

I'd be hesitant to merge this code into the main repository, for multiple reasons:

1. False sense of security: libp2p, and applications built on top of libp2p, broadcast their IDs and multiaddresses at many different places. It's really easy to accidentally reveal your identity, probably even if you turn off the obvious candidates like Identify, AutoNAT and DCuTR. libp2p has not been designed with privacy goals as strict as Tor's in mind, and making sure that the stack has the privacy properties that people expect when they hear that "libp2p over Tor is possible" would be a massive engineering effort. There's probably a lot more ways to compromise yourself other than AutoNAT just publishing your non-Tor addresses. In all likelihood, adding a Tor transport would not yield the privacy properties that people are looking for. Keep in mind that Tor is used by dissidents all around the world, who are not experts in network engineering. It is our responsibility to not endanger their safety by overpromising on the privacy properties of the software we publish.
2. Usability: DHT lookups are already slow. Adding the Tor hops to the equation will make things even more painful. There might be optimizations one could achieve from a deeper integration between libp2p and Tor (after all, Tor is running a DHT as well), but that would be a larger effort and require a really in-depth understanding of both IPFS and Tor.
3. Ethical: If the libp2p Tor transport gained a lot of adoption, this might pose a problem for the Tor network. It's a fairly small network with less than 10k relays, handling important traffic of people who actually rely on the privacy properties of the network.

Merging a Tor transport into a PL-maintained libp2p implementation sends a signal that this transport is properly tested and vetted, no matter how many disclaimers we include. As a modular stack, it should be possible to have this code live outside of rust-libp2p (and outside the libp2p org). This would make it possible for people to play around and experiment with this code, without giving it the blessing it would receive from being a part of rust-libp2p.

[marten-seemann]

Adding more context here. The ask for a Tor transport is as old as IPFS / libp2p itself. See for example this discussion from 2016 (!): https://github.com/libp2p/go-libp2p/pull/79

[Menduist]

I'll reply to your points from nim-libp2p point of view, since we'll soon merge our Tor transport. As always, you are free to do want you want, I'm just giving another perspective :)

1. We are going to recommend (or even force) application developers to create a second switch for the Tor transport only. Trying to isolate the Tor transport from other transports properly in a single switch is very tricky and will probably fail, since a lot of state is shared (conn manager, identity, etc)
2. The application developer will then have to balance, and think about which operations are safe to put on the regular switch, and which one are privacy risk and should go on the Tor switch (at the cost of latency & bandwidth lost etc). For instance, participating in a gossipsub mesh is probably fine on the regular switch, but publishing on it should go on the Tor switch
3. I mean, the Tor network exists to be used. Of course people can abuse it and use a lot of resources from it without actually needing to. But they don't need our help for that. For me, Tor transport is a first step in privacy protection in libp2p at the network level. If we start to see a lot of adoption, it may be the sign that we need to build onion routing (or equivalent) in libp2p networks themselves, instead of leaching another network. But today we can't know if that appetite for network level privacy exist

EDIT: and just to clarify: the goal of the Tor transport is not to say "you just enable this flag and your application becomes magically private", it's here to say "if you are building an application with privacy in mind, you can use this flag as one of the tool used to keep it secure". But it's so easy to leak informations at every layer, the whole application has to be built with privacy in mind

[diegomrsantos]

We are considering:

• Making the TorTransport private
• Creating a TorSwitch where it's not possible to add other transports and name resolving is disabled (see Tor design paper section 4.3 Opening and closing streams). It will force the use of a new dedicated switch with a new peer id.

[umgefahren]

First, I want to thank for all the thoughts and comments on this PR. I see pretty much every point made here, but I have some additional things to say, and I want to offer a way to go forward.

Regarding @marten-seemann comments:

• Of course, "libp2p over Tor" may imply more privacy than this offers. But I don’t think it’s that big of an issue. If we are honest about the caveats that lay here, we can somewhat trust developers to handle the transport with the care it requires. After all, libp2p is not faced towards direct users, so the responsibility is somewhat with the libp2p consumers and thus not directly with the users of Tor. If we include some notices in the documentation and continue our effort here, I consider it a hand able risk.
• Usability is an issue though. I can’t really argue against that. Although there might some usage profiles where the drawbacks are worth the benefits. Maybe the Tor transport is only used to retrieve data directly from a peer via Request-Response. Something like this would require the "switch" described later.
• While I understand your ethical considerations, I also agree with @Menduist. While we don’t make abuse easier, we make it a lot easier to write privacy focused software and possibly help the groups under attack we want to protect.

Regarding @Menduist:

I pretty much agree with all you said. I think we in this repository should look at methods on how we can steer behaviors towards using a given transport and also take a look at nim-libp2p here. But I think this is beyond the scope of this PR. I will open an issue about it, as soon as this is merged.

Finally, I want to leave some comments of my own here.

The reason why I wrote this transport was because I’m really interested in bringing transport level privacy to libp2p. The new iteration of the Dione project I wrote would benefit heavily from it and I could think of a lot of other applications where transport level privacy would be good. I recognize it will be quite hard to properly integrate that into libp2p, but we should do it anyway. I would also recommend looking into writing a generic (in customizable) but tested and capable onion router as libp2p behavior and transport. I would say it’s well in the scope of the project and very doable. In fact, there are Onion Routers written with libp2p (i.e. hoprnet).

Recommendations

If it’s fine by @mxinden and @thomaseizinger, I’m still in favor of merging here, although I would like to take some steps previously.

1. Include a privacy notice in the documentation.
2. Rename everything onion to Tor (we could add other onion routers later and libp2p-onion would fit better if we were ever to develop our own onion router).

We should monitor how this transport develops and is being used. Throwing it out is of course always an option. However, I would do that, once we have better alternatives, transport level privacy seems to be a concern to some people.