ipmjs
Core Goals/Building an Alpha
Quick intro: Hi, I'm Josh - I've recently joined the IPMJS project to contribute towards the eventual goal of having a viable decentralized npm alt.
In order to jumpstart the development of ipmjs, we're going be forking over yarn and building on their foundations with IPFS as the P2P protocol. The core goals of the project are:
- Maintain backwards compatibility with npm and npm based scripts
- Secure users against malicious scripts (allowing users to do code reviews before executing anything untrusted)
- Create a type of truststore for trusted package maintainers/certain package versions
- Offer users the ability to use 3rd party truststores
- Provide decentralized package search functionality
Currently, the architecture looks like this:
- a CLI client which interacts with the underlying IPFS service
- a helper service in conjunction with the IPFS service to provide search services
For the search service, we're gonna be using Orbit to create a pubsub channel where clients can send a search request. From there, other clients grab the query and search their own internal search index and return anything they find back.
This is exciting!
I'm not an expert with IPFS, but for the record there is also a growing IPLD standard (by the same folks) that is a more generic way to link data across distributed systems, for not just files. This may prove useful.
More familiar with dat and ssb personally, but I'm happy to try and learn IPFS and friends, so let me know if I can help out in any way. I'll be keeping tabs. :smile:
For IPLD, as I understand it, we would still need an IPFS service running on the client machine as IPLD just acts as an abstraction layer between the app and any distributed services underneath. I think for simplicity's sake, we're gonna stick to IPFS right now while we build the foundations then later consider if IPLD is viable.
After a bit more reading up, I've changed my tune about IPLD - it'd allows us to be protocol-agnostic and future-proof the project for any new protocols that come up.
First, @joshgarde, welcome to the family. Always nice with some injection of momentum and drive into a project that has gone a little...stale. :)
I completely agree with all core goals, here is some implementation-specific thoughts.
Maintain backwards compatibility with npm and npm based scripts 👍 Since we are forking yarn, we'll almost get this for free, yeay!. The one issue with forking is that we will have to merge in all their changes which can be very time consuming. And we might end up with a fork that is several versions behind but I guess that is okay for now.
Another solution could be to wrap the cli instead and call it when needed and do our own magic when doing IPM-specific actions such as searching/fetching IPFS-distributed packages.
Secure users against malicious scripts (allowing users to do code reviews before executing anything untrusted)
One idea is to always use --ignore-scripts on install and then let the user know if there was a pre/post hook. If the package isn't on the trusted list, let's warn them again before they can run the scripts.
Create a type of truststore for trusted package maintainers/certain package versions This is an important feature but also one of the reasons why the project got stalled in the first place. Who will control this list, who decides what is trusted or not? We need to really think this through so that we won't get into similar problems npm get's into when people are suing because of ™-issues etc. I guess if the packages are trully distributed, there isn't anyone to sue really, but that wouldn't mean that they would try to go after... people like us. :)
Offer users the ability to use 3rd party truststores This might solve the issue I am talking about above. 🤔
Provide decentralized package search functionality I think this is one of the most important core features. The issue, at least one year ago, was that IPFS was painfully slow. I think we need to find a solution that get it close to the speed of npm/yarn's homepage.
Secondly, I agree with @jamen that IPDL definitely looks promising and something i'd would bet on for now.
These are such exciting times, it's like being a part of web 2.0. :)
Also, if you haven't read it already, please do to get some details what I am talking about: https://github.com/ipmjs/ipmjs/issues/9
Maintain backwards compatibility with npm and npm based scripts Wrapping the npm CLI seems like a good idea. Like you said, we wouldn't need to worry about backporting all of our changes (which is easier when it comes time to update) and we could focus efforts on code for IPM.
Another interesting thought from #9 - running a custom registry server. For simply installing IPMJS packages, it seems like a good solution, but it would immediately break npm-only installs (because how is npm supposed to download an IPM package without IPM and how would we establish a distinction between the two in the package.json without breaking vanilla npm?)
Secure users against malicious scripts (allowing users to do code reviews before executing anything untrusted) Exactly what I was going for, except more strict - placing possible malicious scripts into a temp directory without execute permissions. Your implementation might be more reasonable here since once it's downloaded, a malicious script can't really do anything on it's own (except through the install hooks).
Create a type of truststore for trusted package maintainers/certain package versions Truststores are completely managed by the user and none are distributed by default with the client (not even ours). Users must choose to trust package managers/packages/package versions by adding them to the truststore. The goal of distributed systems (at least to me) is to create a system that can operate independently of any party and without the risk of being completely taken down. Even if they sue us, there'd be nothing to go after - we don't host any search index (like Bittorrent sites) and we don't host any unlicensed packages (seeding only what's needed to install our package).
Offer users the ability to use 3rd party truststores Yep, 3rd parties can offer their own truststores for users who don't want to manually approve every package they try to install. We should think about a revocation system too - whether that just means removing the cert from a truststore or if there's an actual revocation list. Another thing we should think about: whether or not we should offer our own truststore for packages other than the ones needed to install IPM. Would that be a conflict of interest?
Provide decentralized package search functionality I don't think speed is the core goal with this one. If it was, we'd just allow users to download a static 3rd party search index and do their searching against that, but that index might be huge (GBs) filled with useless data for the end-user. I think the core goal on anything in this project should be decentralization and distribution (as-in only distributing what the end-user wants/is currently using). Here's my proposal: We have internal search indexes on each client which contain only the packages cached on that client. When we want to search for a package, we send out a global message to all clients asking for packages that match a query string. Each client can then choose to search their own index and return back anything they have relating to the query string. This means that any packages that aren't being seeded, won't clog up the search index and we can ensure only the freshest/most used packages are returned. We'd probably end up using Orbit (another project from Protocol Labs/IPFS) which allows for communication between clients through IPFS.
Maintain backwards compatibility with npm and npm based scripts Just one last thing - we were forking yarn mainly for it's dependency resolution logic (walking down the dependency tree/executing the script hooks). We could just use the logic code from yarn and forward any legacy npm requests to the npm cli instead of maintaining (and backporting) an entire npm implementation.
Well, if no one has any further comments, I'll start building the alpha with these design ideas in mind.