public inbox for nncp-devel@lists.cypherpunks.ru Atom feed
* Thoughts on flexible mesh-routing of NNCP packets @ 2021-01-31 7:03 John Goerzen 2021-01-31 11:19 ` Sergey Matveev 2021-01-31 15:16 ` Sergey Matveev 0 siblings, 2 replies; 5+ messages in thread From: John Goerzen @ 2021-01-31 7:03 UTC (permalink / raw) To: nncp-devel Hi, So I've been thinking about the problem of getting NNCP packets delivered in a dynamic topology. As they are source-routed, that can be a bit of a challenge. I've got some ideas. First I want to make sure I understand how the onion routing in NNCP works, because I think it's somewhat different than Tor (not trying to preserve the anonymity of the sender). Is this description correct? (I originally posted this in comp.mail.uucp and will correct it there if not) In this example, node1 is wanting to send a packet to node5 via intermediaries. So what happens here is that node1 first generates an encrypted packet to node5, using node5's public key, and that packet is signed by node1. Next, it takes that whole thing, encryption and signature and all, and uses it as input for an encrypted packet to node4 -- that packet itself also signed by node1. It repeats this process until it generates the packet to node2. Think of it like this: node1$ gpg --sign -e -r node5 < datafile > node5.gpg node1$ gpg --sign -e -r node4 < node5.gpg > node4.gpg node1$ gpg --sign -e -r node3 < node4.gpg > node3.gpg node1$ gpg --sign -e -r node2 < node3.gpg > outbound-to-node2.gpg node1$ rm node[345.jpg] Node2 receives the packet - encrypted only to it, and signed by node1 - and decrypts the payload. The payload is a command (send this on to node2) and an encrypted payload for THAT command - encrypted to node2 and signed by node1. This process also continues until you get to node5. So, at node4, node4 receives its packet from node3 -- signed by node1, encrypted to node4. Node4 decrypts it and finds a payload for node5 -- again, signed by node1, encrypted to node5. It places this in the queue for node5. (Just like with node2) Node5 receives the packet. It verifies the signature from node1, and it is the only node that can decrypt it. nncp-toss finds the packet in the rx queue from node4, but it doesn't really pass this information on anywhere; it knows from the signature that the origin was node1 (doesn't much care how it arrived) and that single verified signature is what is used for all processing (looking up whether node1 is authorized to run a command, exposed as an environment variable to nncp-exec runners, used to determine the incoming directory for nncp-file, etc). Node5 could sort of know that the packet arrived from node4, but there is no cryptographic guarantee of that (the ONLY signatures in this whole chain come from node1) so that information is basically discarded. This is different from the UUCP model, in which node5 only really knows about its neighbor node4 and may have no idea whatever about node1's existence. Permissions in UUCP are based on what the neighbor can do, not what the source can do, if I remember correctly. The other thing that follows from this is that if node3 were compromised, the worst it can do is fail to relay packets. It cannot modify packets (signature validation from node1 would fail), it cannot spoof node1 (lacking its private key), it can't inject fake packets claiming to have followed the node1!node2!node3 path (again, lacking node1's private key), it can't even see what the ultimate destination is (it just knows the next hop is node4 but since it doesn't have node4's private key, it doesn't even know if node4 is the ultimate hop or not). Now then.... If that is correct, that would imply that it would be difficult to, say, bypass node2 on the way to node5 -- though it would be (theoretically) easy enough to send from node1 to node1.5 and then on to node2. So I've been thinking about this more expansively. https://verbnetworks.com/research/store-forward-networks/ planted the idea in my head: why not use Syncthing for this? It would seem to make perfect sense, given nncp-xfer. There could be a shared Syncthing folder which leaf sites could dump packets into using nncp-xfer, and read other packets out (also using nncp-xfer). Since Syncthing implements a Bittorrent-like protocol, a leaf site can just dump packets into its local Syncthing folder, letting Syncthing get them out via whatever path it can. Once the files are properly received, the receiver deletes them and this propagates out over the swarm as well. Slick! I had initially been thinking up some fancy scheme to send back a "I've got this packet, delete your copy" back to the originator, but then realized Syncthing basically already solved that. - John ^ permalink raw reply [flat|nested] 5+ messages in thread
* Re: Thoughts on flexible mesh-routing of NNCP packets 2021-01-31 7:03 Thoughts on flexible mesh-routing of NNCP packets John Goerzen @ 2021-01-31 11:19 ` Sergey Matveev 2021-02-01 6:21 ` John Goerzen 2021-01-31 15:16 ` Sergey Matveev 1 sibling, 1 reply; 5+ messages in thread From: Sergey Matveev @ 2021-01-31 11:19 UTC (permalink / raw) To: nncp-devel [-- Attachment #1: Type: text/plain, Size: 9171 bytes --] Greetings! *** John Goerzen [2021-01-31 01:03]: >First I want to make sure I understand how the onion routing in NNCP works, >because I think it's somewhat different than Tor (not trying to preserve the >anonymity of the sender). Is this description correct? Yes, it is not intended to preserve identity of the sender. But it tries to preserve the whole transmission path knowledge for intermediate and target points. >In this example, node1 is wanting to send a packet to node5 via >intermediaries. [...] Everything in your example is correct! >Permissions in UUCP are based on what the neighbor can do, not what the >source can do, if I remember correctly. I do not remember too, but seems you are right. NNCP is different there. >The other thing that follows from this is that if node3 were compromised, the >worst it can do is fail to relay packets. [...] Yep, also everything is true. >If that is correct, that would imply that it would be difficult to, say, >bypass node2 on the way to node5 -- though it would be (theoretically) easy >enough to send from node1 to node1.5 and then on to node2. Yes, it complicates some things. However actual path can be extended in reality: node2 can have "node3: via:node2.5" option that will send transitient packet additionally through node2.5, without node1's knowledge. That was just a remark, that "full" path is not so "hardcoded" in the packets :-) >https://verbnetworks.com/research/store-forward-networks/ planted the idea in >my head: why not use Syncthing for this? First of all: do not forget that initially NNCP, with all that transition/via packets were developed only for offline data transmission, without online/network protocols in mind. So idea of transition packets/nodes was not about relying between network nodes, for example because of NAT between them. It is really better to use some kind of network shared directories in that case: FTP, NFS, 9p, rsync and all that kind of things. I have never used Syncthing and only heard of its name. Seems it is really some kind of very good tool for nncp-xfer! If you can "share" only some part of directory hierarchy (directories you are "interested" in), if file renames (renaming of temporary file to "normal" fully written packet) are atomic/fast and even deletions will propagate, even you can setup relay nodes as you wish, then I see nothing against Syncthing. It is even written on Go, that is good. Initially I though just about using rsync to share nncp-xfer-compatible directory with some remote servers. And I planned to create Noise-encrypted link to run rsync-protocol inside. But I discovered that rsync protocol just has no specification at all -- it is just coded in the software. So I created "sync protocol" for its "replacement", trying to make as few roundtrips as possible (however it is still uses TCP). https://verbnetworks.com/research/store-forward-networks/ -- previously I have read documents about DTNs, but never known that software like ion-dtn exists! Some time I thought about creating fully DTN-protocol (like in RFC) capable implementation. Must look at it closer. Syncthing is another thing I should also try :-). However personally have no tasks for it. That article also mentions CJDNS (that has a much better working Yggdrasil continuation remake), DAT, IPFS, ZeroNet, TahoeLAFS. Below is completely my opinion and experience with all of them: * DAT -- JavaScript, npm... no, thanks. However I like the idea * ZeroNet -- Python, that is not fast and from year to year is becoming more burdensome to use it with all that PyPI infrastructure permanent regression (rsync ability to mirror it was removed and replaced with constantly badly working bandersnatch and constantly unavailable CDN; then they removed pip search; then they removed Py27; then many packages removed Py35 support; and still nearly none of packages uses checksums and strict version for their dependencies, basically leading to completely undeterministic builds that hardly ever can be done without manual interventions). Also ZeroNet, as I remember, requires to use the browser with JavaScript. However idea is good * TahoeLAFS -- just for simple file sharing it is just too heavy, too slow and hard to manage. It is for other kind of tasks, that I personally have ever met. But I very like idea of that project * IPFS -- like the idea, but... literally for several years I tried to setup and use it once per half-year/year and it worked ONLY once. I tried it only on FreeBSD. Some attempts just can not build it, because of hard-coded-Linux specific stuff. Some attempts *without* any questions and precautions downloaded some binaries from the Internet, silently trying to run them. That is *completely* unacceptable behaviour, so all next times I tried IPFS -- I deeply check every step of building/installation. And when I got it build... it just either segfaults or consumes *all* available RAM, leading system to swap. So literally I have seen it working (able to run at least for a day, not segfault and being able to transfer files) many years ago when the project was very young * CJDNS -- I very like the idea and its simple protocol, doing exactly what is should do, without any kind of additional and sometimes harmful and unneeded stuff like anonymization, built-in cryptocurrencies, DNS alternative and so on. Anonymization requirement greatly complicates all the things and basically it hardly can be achieved without efficiency regression. I have nothing against anonymity, privacy, but not everyone and not every tasks want them. There are many tasks requiring *efficient* overlays, *efficient* "mesh" networking. And CJDNS was just that kind of needed creation! It worked pretty good when using with own nodes. But when I connected to Hyperboria, it segfaults randomly possibly once per day, possibly after ten minutes. So it was not workable when connected to Hyperboria, possibly because of various used software versions in it. Do not know. Yggdrasil project, that is written for replacement and written on Go -- was working very stable (however only with "own" nodes, no any kind of global Hyperboria-network)! And it is one of few projects I can recommend to use. However seems it could be interesting to NNCP only as an overlay transport, probably when both nodes behind the NAT for example * GNUnet -- coolest thing from academic point of view (the ideal creation for cypherpunk-world!), but never actually usable in practice (tried many years ago last time) * Freenet, I2P, YaCy -- I have *never* seen really workable software written on Java. No offence to any kind of developers, but I literally see, that after several days/weeks of working, all of that software just consumes more and more RAM and become "completely" unresponsive. I assume that when there are many "objects" in their databases (Freenet, YaCy), they starts to "lag" so heavily, that completely unresponsive to nearly all commands. I tried I2P C++ implementation -- it worked much better, but... without any known reasons, it just becomes unresponsive to the locally running eepsites: traffic seems relaying, but locally "hosted" eepsites just stops answering. i2pd was usable only when restarted possibly once per half and hour I know that all of that seems pretty strange. Many people uses those projects. But, most of them do not use FreeBSD and most of people uses proprietary Java JVMs. So possibly most of that projects just do not run good on FreeBSD, because of small (or lack) of testing on it, and Java sucks probably because of OpenJDK (I heard about that many times). But summary is: there are many projects with interesting and good ideas, but only Tor works really perfectly good and probably Yggdrasil (I hope mainly because of Go). But I assume that Tor has several times more founding that all of other projects together. Many years ago I was very interested in all that stuff: anonymity, privacy, and so on, so on, so on. I am still interested, but VERY disappointed with the fact that hardly any of most of that kind of software works :-). So, being the software developer, I just must stop whimpering and complaining and do the job myself. But... lack of time, damned lack of time. That is why people just have to used proprietary and far from ideal solutions because other ones just do not work in practice, or just much more expensive to run and use. Software development is very expensive thing. And portable software, as I can see with my FreeBSD experience, is hard to do right. But I will look closer at Syncthing, ION(-DTN). Thanks for sharing that information and thoughts!. And I still remember about asynchronous checksumming, checksumming during packets receiving and other issues, that are pretty high-priority tasks for NNCP! -- Sergey Matveev (http://www.stargrave.org/) OpenPGP: CF60 E89A 5923 1E76 E263 6422 AE1A 8109 E498 57EF [-- Attachment #2: signature.asc --] [-- Type: application/pgp-signature, Size: 833 bytes --] ^ permalink raw reply [flat|nested] 5+ messages in thread
* Re: Thoughts on flexible mesh-routing of NNCP packets 2021-01-31 11:19 ` Sergey Matveev @ 2021-02-01 6:21 ` John Goerzen 2021-02-01 8:39 ` Sergey Matveev 0 siblings, 1 reply; 5+ messages in thread From: John Goerzen @ 2021-02-01 6:21 UTC (permalink / raw) To: nncp-devel On Sun, Jan 31 2021, Sergey Matveev wrote: > Greetings! > > *** John Goerzen [2021-01-31 01:03]: >>First I want to make sure I understand how the onion routing in >>NNCP works, >>because I think it's somewhat different than Tor (not trying to >>preserve the >>anonymity of the sender). Is this description correct? > > Yes, it is not intended to preserve identity of the sender. But > it tries > to preserve the whole transmission path knowledge for > intermediate and > target points. That makes perfect sense (and is fine by me). Thanks for the confirmation. >>Permissions in UUCP are based on what the neighbor can do, not >>what the >>source can do, if I remember correctly. > > I do not remember too, but seems you are right. NNCP is > different there. Yes, and that's good! >>If that is correct, that would imply that it would be difficult >>to, say, >>bypass node2 on the way to node5 -- though it would be >>(theoretically) easy >>enough to send from node1 to node1.5 and then on to node2. > > Yes, it complicates some things. However actual path can be > extended in > reality: node2 can have "node3: via:node2.5" option that will > send > transitient packet additionally through node2.5, without node1's > knowledge. That was just a remark, that "full" path is not so > "hardcoded" in the packets :-) That is very interesting. I hadn't realized that! Which reminds me - when you get back a file from nncp-freq, or any other event that sends something back, I'm assuming it uses the via path (if any) defined on the system that's sending back the response? >>https://verbnetworks.com/research/store-forward-networks/ >>planted the idea in >>my head: why not use Syncthing for this? > > First of all: do not forget that initially NNCP, with all that > transition/via packets were developed only for offline data > transmission, without online/network protocols in mind. So idea > of > transition packets/nodes was not about relying between network > nodes, > for example because of NAT between them. It is really better to > use some > kind of network shared directories in that case: FTP, NFS, 9p, > rsync and > all that kind of things. Interesting. This seems like a perfect fit for Syncthing. It is sort of like a distributed FTP or NFS. > I have never used Syncthing and only heard of its name. Seems it > is > really some kind of very good tool for nncp-xfer! If you can > "share" Yes, I have been using it for some years now and am quite pleased with it. > only some part of directory hierarchy (directories you are > "interested" > in), if file renames (renaming of temporary file to "normal" > fully > written packet) are atomic/fast and even deletions will > propagate, even > you can setup relay nodes as you wish, then I see nothing > against > Syncthing. It is even written on Go, that is good. Yes to all of these. Only part of the directory hierarchy: https://docs.syncthing.net/users/ignoring.html#ignoring-files Renames are fast: https://docs.syncthing.net/users/faq.html#is-synchronization-fast Can set up relay nodes as you wish: https://docs.syncthing.net/users/introducer.html So I wouldn't suggest anybody just rip out nncp-call(er) and go to Syncthing entirely; nncp-call and the daemon are quite useful and more efficient than Syncthing would be in fixed topology cases, or situations with a non-TCP transport. But the nncp-xfer file format is *perfect* for this. A "leaf" node can use the .stignore file (which also can have "include" lines) to register its interest in just certain packets (perhaps anything going to or from it). Relay nodes could carry the whole thing. And when nncp-xfer imports packets, the deletion out of the Syncthing tree would propagate. I hope to have the time to set up and test this all tomorrow. One particularly interesting use for this - Syncthing can to local LAN discovery. Syncthing also runs on Android. So it is entirely possible to use Syncthing as a NNCP "bridge" between two disconnected networks, syncing up work whenever it connects to a local LAN. Of course, one could probably make nncp work in Termux on Android also. > https://verbnetworks.com/research/store-forward-networks/ -- > previously > I have read documents about DTNs, but never known that software > like > ion-dtn exists! Some time I thought about creating fully > DTN-protocol > (like in RFC) capable implementation. Must look at it closer. Same here. > Syncthing is another thing I should also try :-). However > personally > have no tasks for it. I saw your other message about issues with your patched Go cryptography layer. I have been using Syncthing's debs for a long time without issue, and I'm not a Go programmer so probably can't help there, but Syncthing makes a very nice alternative to things like Dropbox, or even Nextcloud, since it can be run entirely serverless and disconnected. > That article also mentions CJDNS (that has a much better working > Yggdrasil continuation remake), DAT, IPFS, ZeroNet, TahoeLAFS. > Below is > completely my opinion and experience with all of them: Thanks for this! > > * DAT -- JavaScript, npm... no, thanks. However I like the idea Being an append-only log has me somewhat concerned. It seems to be something like a "public Syncthing, but unidirectional." IPFS seems to be catching on more than DAT at the moment. > Many years ago I was very interested in all that stuff: > anonymity, > privacy, and so on, so on, so on. I am still interested, but > VERY > disappointed with the fact that hardly any of most of that kind > of > software works :-). I keep trying to do my part around the edges, but yes, lack of time is always an issue for me as well. > But I will look closer at Syncthing, ION(-DTN). Thanks for > sharing that > information and thoughts!. And I still remember about > asynchronous > checksumming, checksumming during packets receiving and other > issues, > that are pretty high-priority tasks for NNCP! And thank you, again, for NNCP. Unlike many of these other things, it's fairly simple, well-documented, and secure and private by default. Heck, a person could even use Google Drive for the packets, if one wanted. - John ^ permalink raw reply [flat|nested] 5+ messages in thread
* Re: Thoughts on flexible mesh-routing of NNCP packets 2021-02-01 6:21 ` John Goerzen @ 2021-02-01 8:39 ` Sergey Matveev 0 siblings, 0 replies; 5+ messages in thread From: Sergey Matveev @ 2021-02-01 8:39 UTC (permalink / raw) To: nncp-devel [-- Attachment #1: Type: text/plain, Size: 1847 bytes --] *** John Goerzen [2021-02-01 00:21]: >Which reminds me - when you get back a file from nncp-freq, or any other >event that sends something back, I'm assuming it uses the via path (if any) >defined on the system that's sending back the response? Yes, correct. >So I wouldn't suggest anybody just rip out nncp-call(er) and go to Syncthing >entirely; nncp-call and the daemon are quite useful and more efficient than >Syncthing would be in fixed topology cases, or situations with a non-TCP >transport. I understand. There can not be killer solution for all cases. >One particularly interesting use for this - Syncthing can to local LAN >discovery. Once I wanted to create same thing in nncp-daemon/caller -- multicasted notifications about NNCP daemons presence. >very nice alternative to things like Dropbox, or even Nextcloud, since it can >be run entirely serverless and disconnected. Dropbox is not the choice of course, because it is not free software. >Being an append-only log has me somewhat concerned. It seems to be something >like a "public Syncthing, but unidirectional." IPFS seems to be catching on >more than DAT at the moment. I like DAT idea because it is very close to WARC: https://en.wikipedia.org/wiki/Web_ARChive whole website distribution through single file. And DAT (seems) allows to effectively publish changes to it. It seems can be conveniently archived for long term storage. >And thank you, again, for NNCP. Unlike many of these other things, it's >fairly simple, well-documented, and secure and private by default. Hope so! Glad you like it! >Heck, a person could even use Google Drive for the packets, if one wanted. Yes, of course, no barriers. -- Sergey Matveev (http://www.stargrave.org/) OpenPGP: CF60 E89A 5923 1E76 E263 6422 AE1A 8109 E498 57EF [-- Attachment #2: signature.asc --] [-- Type: application/pgp-signature, Size: 833 bytes --] ^ permalink raw reply [flat|nested] 5+ messages in thread
* Re: Thoughts on flexible mesh-routing of NNCP packets 2021-01-31 7:03 Thoughts on flexible mesh-routing of NNCP packets John Goerzen 2021-01-31 11:19 ` Sergey Matveev @ 2021-01-31 15:16 ` Sergey Matveev 1 sibling, 0 replies; 5+ messages in thread From: Sergey Matveev @ 2021-01-31 15:16 UTC (permalink / raw) To: nncp-devel [-- Attachment #1: Type: text/plain, Size: 817 bytes --] *** John Goerzen [2021-01-31 01:03]: >why not use Syncthing for this? Just tried it. Well, I am disappointed again :-) Downloaded tarball, unpacked, "go run build.go build", and then: % ./syncthing -help panic: qtls.ConnectionState doesn't match goroutine 1 [running]: github.com/marten-seemann/qtls-go1-15.init.0() github.com/marten-seemann/qtls-go1-15@v0•1.1/unsafe.go:11 +0x245 % ./syncthing panic: qtls.ConnectionState doesn't match Actually, it is because I use slightly modified Go version with GOST cryptography, but syncthing's dependencies prohibits me to use modified crypto/tls.ConnectionState, so it will take time trying to run it. -- Sergey Matveev (http://www.stargrave.org/) OpenPGP: CF60 E89A 5923 1E76 E263 6422 AE1A 8109 E498 57EF [-- Attachment #2: signature.asc --] [-- Type: application/pgp-signature, Size: 833 bytes --] ^ permalink raw reply [flat|nested] 5+ messages in thread
end of thread, other threads:[~2021-02-01 8:39 UTC | newest] Thread overview: 5+ messages (download: mbox.gz / follow: Atom feed) -- links below jump to the message on this page -- 2021-01-31 7:03 Thoughts on flexible mesh-routing of NNCP packets John Goerzen 2021-01-31 11:19 ` Sergey Matveev 2021-02-01 6:21 ` John Goerzen 2021-02-01 8:39 ` Sergey Matveev 2021-01-31 15:16 ` Sergey Matveev