public inbox for nncp-devel@lists.cypherpunks.ru
Atom feed
* Thoughts on flexible mesh-routing of NNCP packets
@ 2021-01-31 7:03 John Goerzen
2021-01-31 11:19 ` Sergey Matveev
2021-01-31 15:16 ` Sergey Matveev
0 siblings, 2 replies; 5+ messages in thread
From: John Goerzen @ 2021-01-31 7:03 UTC (permalink / raw)
To: nncp-devel
Hi,
So I've been thinking about the problem of getting NNCP packets
delivered in a dynamic topology. As they are source-routed, that
can be a bit of a challenge. I've got some ideas.
First I want to make sure I understand how the onion routing in
NNCP works, because I think it's somewhat different than Tor (not
trying to preserve the anonymity of the sender). Is this
description correct? (I originally posted this in comp.mail.uucp
and will correct it there if not)
In this example, node1 is wanting to send a packet to node5 via
intermediaries.
So what happens here is that node1 first generates an encrypted
packet to node5,
using node5's public key, and that packet is signed by node1.
Next, it takes
that whole thing, encryption and signature and all, and uses it as
input for an
encrypted packet to node4 -- that packet itself also signed by
node1. It
repeats this process until it generates the packet to node2.
Think of it like this:
node1$ gpg --sign -e -r node5 < datafile > node5.gpg
node1$ gpg --sign -e -r node4 < node5.gpg > node4.gpg
node1$ gpg --sign -e -r node3 < node4.gpg > node3.gpg
node1$ gpg --sign -e -r node2 < node3.gpg > outbound-to-node2.gpg
node1$ rm node[345.jpg]
Node2 receives the packet - encrypted only to it, and signed by
node1 -
and decrypts the payload. The payload is a command (send this on
to
node2) and an encrypted payload for THAT command - encrypted to
node2
and signed by node1. This process also continues until you get to
node5.
So, at node4, node4 receives its packet from node3 -- signed by
node1, encrypted
to node4. Node4 decrypts it and finds a payload for node5 --
again, signed by
node1, encrypted to node5. It places this in the queue for node5.
(Just like
with node2)
Node5 receives the packet. It verifies the signature from node1,
and it is the
only node that can decrypt it. nncp-toss finds the packet in the
rx queue from
node4, but it doesn't really pass this information on anywhere; it
knows from
the signature that the origin was node1 (doesn't much care how it
arrived) and
that single verified signature is what is used for all processing
(looking up
whether node1 is authorized to run a command, exposed as an
environment variable
to nncp-exec runners, used to determine the incoming directory for
nncp-file,
etc).
Node5 could sort of know that the packet arrived from node4, but
there is no
cryptographic guarantee of that (the ONLY signatures in this whole
chain come
from node1) so that information is basically discarded.
This is different from the UUCP model, in which node5 only really
knows about
its neighbor node4 and may have no idea whatever about node1's
existence.
Permissions in UUCP are based on what the neighbor can do, not
what the source
can do, if I remember correctly.
The other thing that follows from this is that if node3 were
compromised, the
worst it can do is fail to relay packets. It cannot modify packets
(signature
validation from node1 would fail), it cannot spoof node1 (lacking
its private
key), it can't inject fake packets claiming to have followed the
node1!node2!node3 path (again, lacking node1's private key), it
can't even see
what the ultimate destination is (it just knows the next hop is
node4 but since
it doesn't have node4's private key, it doesn't even know if node4
is the
ultimate hop or not).
Now then....
If that is correct, that would imply that it would be difficult
to, say, bypass node2 on the way to node5 -- though it would be
(theoretically) easy enough to send from node1 to node1.5 and then
on to node2.
So I've been thinking about this more expansively.
https://verbnetworks.com/research/store-forward-networks/ planted
the idea in my head: why not use Syncthing for this?
It would seem to make perfect sense, given nncp-xfer. There could
be a shared Syncthing folder which leaf sites could dump packets
into using nncp-xfer, and read other packets out (also using
nncp-xfer). Since Syncthing implements a Bittorrent-like
protocol, a leaf site can just dump packets into its local
Syncthing folder, letting Syncthing get them out via whatever path
it can. Once the files are properly received, the receiver
deletes them and this propagates out over the swarm as well.
Slick!
I had initially been thinking up some fancy scheme to send back a
"I've got this packet, delete your copy" back to the originator,
but then realized Syncthing basically already solved that.
- John
^ permalink raw reply [flat|nested] 5+ messages in thread
* Re: Thoughts on flexible mesh-routing of NNCP packets
2021-01-31 7:03 Thoughts on flexible mesh-routing of NNCP packets John Goerzen
@ 2021-01-31 11:19 ` Sergey Matveev
2021-02-01 6:21 ` John Goerzen
2021-01-31 15:16 ` Sergey Matveev
1 sibling, 1 reply; 5+ messages in thread
From: Sergey Matveev @ 2021-01-31 11:19 UTC (permalink / raw)
To: nncp-devel
[-- Attachment #1: Type: text/plain, Size: 9171 bytes --]
Greetings!
*** John Goerzen [2021-01-31 01:03]:
>First I want to make sure I understand how the onion routing in NNCP works,
>because I think it's somewhat different than Tor (not trying to preserve the
>anonymity of the sender). Is this description correct?
Yes, it is not intended to preserve identity of the sender. But it tries
to preserve the whole transmission path knowledge for intermediate and
target points.
>In this example, node1 is wanting to send a packet to node5 via
>intermediaries. [...]
Everything in your example is correct!
>Permissions in UUCP are based on what the neighbor can do, not what the
>source can do, if I remember correctly.
I do not remember too, but seems you are right. NNCP is different there.
>The other thing that follows from this is that if node3 were compromised, the
>worst it can do is fail to relay packets. [...]
Yep, also everything is true.
>If that is correct, that would imply that it would be difficult to, say,
>bypass node2 on the way to node5 -- though it would be (theoretically) easy
>enough to send from node1 to node1.5 and then on to node2.
Yes, it complicates some things. However actual path can be extended in
reality: node2 can have "node3: via:node2.5" option that will send
transitient packet additionally through node2.5, without node1's
knowledge. That was just a remark, that "full" path is not so
"hardcoded" in the packets :-)
>https://verbnetworks.com/research/store-forward-networks/ planted the idea in
>my head: why not use Syncthing for this?
First of all: do not forget that initially NNCP, with all that
transition/via packets were developed only for offline data
transmission, without online/network protocols in mind. So idea of
transition packets/nodes was not about relying between network nodes,
for example because of NAT between them. It is really better to use some
kind of network shared directories in that case: FTP, NFS, 9p, rsync and
all that kind of things.
I have never used Syncthing and only heard of its name. Seems it is
really some kind of very good tool for nncp-xfer! If you can "share"
only some part of directory hierarchy (directories you are "interested"
in), if file renames (renaming of temporary file to "normal" fully
written packet) are atomic/fast and even deletions will propagate, even
you can setup relay nodes as you wish, then I see nothing against
Syncthing. It is even written on Go, that is good.
Initially I though just about using rsync to share nncp-xfer-compatible
directory with some remote servers. And I planned to create
Noise-encrypted link to run rsync-protocol inside. But I discovered that
rsync protocol just has no specification at all -- it is just coded in
the software. So I created "sync protocol" for its "replacement", trying
to make as few roundtrips as possible (however it is still uses TCP).
https://verbnetworks.com/research/store-forward-networks/ -- previously
I have read documents about DTNs, but never known that software like
ion-dtn exists! Some time I thought about creating fully DTN-protocol
(like in RFC) capable implementation. Must look at it closer.
Syncthing is another thing I should also try :-). However personally
have no tasks for it.
That article also mentions CJDNS (that has a much better working
Yggdrasil continuation remake), DAT, IPFS, ZeroNet, TahoeLAFS. Below is
completely my opinion and experience with all of them:
* DAT -- JavaScript, npm... no, thanks. However I like the idea
* ZeroNet -- Python, that is not fast and from year to year is becoming
more burdensome to use it with all that PyPI infrastructure permanent
regression (rsync ability to mirror it was removed and replaced with
constantly badly working bandersnatch and constantly unavailable CDN;
then they removed pip search; then they removed Py27; then many
packages removed Py35 support; and still nearly none of packages uses
checksums and strict version for their dependencies, basically leading
to completely undeterministic builds that hardly ever can be done
without manual interventions). Also ZeroNet, as I remember, requires
to use the browser with JavaScript. However idea is good
* TahoeLAFS -- just for simple file sharing it is just too heavy, too
slow and hard to manage. It is for other kind of tasks, that I
personally have ever met. But I very like idea of that project
* IPFS -- like the idea, but... literally for several years I tried to
setup and use it once per half-year/year and it worked ONLY once. I
tried it only on FreeBSD. Some attempts just can not build it, because
of hard-coded-Linux specific stuff. Some attempts *without* any
questions and precautions downloaded some binaries from the Internet,
silently trying to run them. That is *completely* unacceptable
behaviour, so all next times I tried IPFS -- I deeply check every step
of building/installation. And when I got it build... it just either
segfaults or consumes *all* available RAM, leading system to swap. So
literally I have seen it working (able to run at least for a day, not
segfault and being able to transfer files) many years ago when the
project was very young
* CJDNS -- I very like the idea and its simple protocol, doing exactly
what is should do, without any kind of additional and sometimes
harmful and unneeded stuff like anonymization, built-in
cryptocurrencies, DNS alternative and so on. Anonymization requirement
greatly complicates all the things and basically it hardly can be
achieved without efficiency regression. I have nothing against
anonymity, privacy, but not everyone and not every tasks want them.
There are many tasks requiring *efficient* overlays, *efficient*
"mesh" networking. And CJDNS was just that kind of needed creation! It
worked pretty good when using with own nodes. But when I connected to
Hyperboria, it segfaults randomly possibly once per day, possibly
after ten minutes. So it was not workable when connected to
Hyperboria, possibly because of various used software versions in it.
Do not know. Yggdrasil project, that is written for replacement and
written on Go -- was working very stable (however only with "own"
nodes, no any kind of global Hyperboria-network)! And it is one of few
projects I can recommend to use. However seems it could be interesting
to NNCP only as an overlay transport, probably when both nodes behind
the NAT for example
* GNUnet -- coolest thing from academic point of view (the ideal
creation for cypherpunk-world!), but never actually usable in practice
(tried many years ago last time)
* Freenet, I2P, YaCy -- I have *never* seen really workable software
written on Java. No offence to any kind of developers, but I literally
see, that after several days/weeks of working, all of that software
just consumes more and more RAM and become "completely" unresponsive.
I assume that when there are many "objects" in their databases
(Freenet, YaCy), they starts to "lag" so heavily, that completely
unresponsive to nearly all commands. I tried I2P C++ implementation --
it worked much better, but... without any known reasons, it just
becomes unresponsive to the locally running eepsites: traffic seems
relaying, but locally "hosted" eepsites just stops answering. i2pd was
usable only when restarted possibly once per half and hour
I know that all of that seems pretty strange. Many people uses those
projects. But, most of them do not use FreeBSD and most of people uses
proprietary Java JVMs. So possibly most of that projects just do not run
good on FreeBSD, because of small (or lack) of testing on it, and Java
sucks probably because of OpenJDK (I heard about that many times). But
summary is: there are many projects with interesting and good ideas, but
only Tor works really perfectly good and probably Yggdrasil (I hope
mainly because of Go). But I assume that Tor has several times more
founding that all of other projects together.
Many years ago I was very interested in all that stuff: anonymity,
privacy, and so on, so on, so on. I am still interested, but VERY
disappointed with the fact that hardly any of most of that kind of
software works :-).
So, being the software developer, I just must stop whimpering and
complaining and do the job myself. But... lack of time, damned lack of
time. That is why people just have to used proprietary and far from
ideal solutions because other ones just do not work in practice, or just
much more expensive to run and use. Software development is very
expensive thing. And portable software, as I can see with my FreeBSD
experience, is hard to do right.
But I will look closer at Syncthing, ION(-DTN). Thanks for sharing that
information and thoughts!. And I still remember about asynchronous
checksumming, checksumming during packets receiving and other issues,
that are pretty high-priority tasks for NNCP!
--
Sergey Matveev (http://www.stargrave.org/)
OpenPGP: CF60 E89A 5923 1E76 E263 6422 AE1A 8109 E498 57EF
[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]
^ permalink raw reply [flat|nested] 5+ messages in thread
* Re: Thoughts on flexible mesh-routing of NNCP packets
2021-01-31 7:03 Thoughts on flexible mesh-routing of NNCP packets John Goerzen
2021-01-31 11:19 ` Sergey Matveev
@ 2021-01-31 15:16 ` Sergey Matveev
1 sibling, 0 replies; 5+ messages in thread
From: Sergey Matveev @ 2021-01-31 15:16 UTC (permalink / raw)
To: nncp-devel
[-- Attachment #1: Type: text/plain, Size: 817 bytes --]
*** John Goerzen [2021-01-31 01:03]:
>why not use Syncthing for this?
Just tried it. Well, I am disappointed again :-)
Downloaded tarball, unpacked, "go run build.go build", and then:
% ./syncthing -help
panic: qtls.ConnectionState doesn't match
goroutine 1 [running]:
github.com/marten-seemann/qtls-go1-15.init.0()
github.com/marten-seemann/qtls-go1-15@v0•1.1/unsafe.go:11 +0x245
% ./syncthing
panic: qtls.ConnectionState doesn't match
Actually, it is because I use slightly modified Go version with GOST
cryptography, but syncthing's dependencies prohibits me to use modified
crypto/tls.ConnectionState, so it will take time trying to run it.
--
Sergey Matveev (http://www.stargrave.org/)
OpenPGP: CF60 E89A 5923 1E76 E263 6422 AE1A 8109 E498 57EF
[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]
^ permalink raw reply [flat|nested] 5+ messages in thread
* Re: Thoughts on flexible mesh-routing of NNCP packets
2021-01-31 11:19 ` Sergey Matveev
@ 2021-02-01 6:21 ` John Goerzen
2021-02-01 8:39 ` Sergey Matveev
0 siblings, 1 reply; 5+ messages in thread
From: John Goerzen @ 2021-02-01 6:21 UTC (permalink / raw)
To: nncp-devel
On Sun, Jan 31 2021, Sergey Matveev wrote:
> Greetings!
>
> *** John Goerzen [2021-01-31 01:03]:
>>First I want to make sure I understand how the onion routing in
>>NNCP works,
>>because I think it's somewhat different than Tor (not trying to
>>preserve the
>>anonymity of the sender). Is this description correct?
>
> Yes, it is not intended to preserve identity of the sender. But
> it tries
> to preserve the whole transmission path knowledge for
> intermediate and
> target points.
That makes perfect sense (and is fine by me). Thanks for the
confirmation.
>>Permissions in UUCP are based on what the neighbor can do, not
>>what the
>>source can do, if I remember correctly.
>
> I do not remember too, but seems you are right. NNCP is
> different there.
Yes, and that's good!
>>If that is correct, that would imply that it would be difficult
>>to, say,
>>bypass node2 on the way to node5 -- though it would be
>>(theoretically) easy
>>enough to send from node1 to node1.5 and then on to node2.
>
> Yes, it complicates some things. However actual path can be
> extended in
> reality: node2 can have "node3: via:node2.5" option that will
> send
> transitient packet additionally through node2.5, without node1's
> knowledge. That was just a remark, that "full" path is not so
> "hardcoded" in the packets :-)
That is very interesting. I hadn't realized that!
Which reminds me - when you get back a file from nncp-freq, or any
other event that sends something back, I'm assuming it uses the
via path (if any) defined on the system that's sending back the
response?
>>https://verbnetworks.com/research/store-forward-networks/
>>planted the idea in
>>my head: why not use Syncthing for this?
>
> First of all: do not forget that initially NNCP, with all that
> transition/via packets were developed only for offline data
> transmission, without online/network protocols in mind. So idea
> of
> transition packets/nodes was not about relying between network
> nodes,
> for example because of NAT between them. It is really better to
> use some
> kind of network shared directories in that case: FTP, NFS, 9p,
> rsync and
> all that kind of things.
Interesting. This seems like a perfect fit for Syncthing. It is
sort of like a distributed FTP or NFS.
> I have never used Syncthing and only heard of its name. Seems it
> is
> really some kind of very good tool for nncp-xfer! If you can
> "share"
Yes, I have been using it for some years now and am quite pleased
with it.
> only some part of directory hierarchy (directories you are
> "interested"
> in), if file renames (renaming of temporary file to "normal"
> fully
> written packet) are atomic/fast and even deletions will
> propagate, even
> you can setup relay nodes as you wish, then I see nothing
> against
> Syncthing. It is even written on Go, that is good.
Yes to all of these.
Only part of the directory hierarchy:
https://docs.syncthing.net/users/ignoring.html#ignoring-files
Renames are fast:
https://docs.syncthing.net/users/faq.html#is-synchronization-fast
Can set up relay nodes as you wish:
https://docs.syncthing.net/users/introducer.html
So I wouldn't suggest anybody just rip out nncp-call(er) and go to
Syncthing entirely; nncp-call and the daemon are quite useful and
more efficient than Syncthing would be in fixed topology cases, or
situations with a non-TCP transport.
But the nncp-xfer file format is *perfect* for this. A "leaf"
node can use the .stignore file (which also can have "include"
lines) to register its interest in just certain packets (perhaps
anything going to or from it). Relay nodes could carry the whole
thing. And when nncp-xfer imports packets, the deletion out of
the Syncthing tree would propagate. I hope to have the time to
set up and test this all tomorrow.
One particularly interesting use for this - Syncthing can to local
LAN discovery. Syncthing also runs on Android. So it is entirely
possible to use Syncthing as a NNCP "bridge" between two
disconnected networks, syncing up work whenever it connects to a
local LAN.
Of course, one could probably make nncp work in Termux on Android
also.
> https://verbnetworks.com/research/store-forward-networks/ --
> previously
> I have read documents about DTNs, but never known that software
> like
> ion-dtn exists! Some time I thought about creating fully
> DTN-protocol
> (like in RFC) capable implementation. Must look at it closer.
Same here.
> Syncthing is another thing I should also try :-). However
> personally
> have no tasks for it.
I saw your other message about issues with your patched Go
cryptography layer. I have been using Syncthing's debs for a long
time without issue, and I'm not a Go programmer so probably can't
help there, but Syncthing makes a very nice alternative to things
like Dropbox, or even Nextcloud, since it can be run entirely
serverless and disconnected.
> That article also mentions CJDNS (that has a much better working
> Yggdrasil continuation remake), DAT, IPFS, ZeroNet, TahoeLAFS.
> Below is
> completely my opinion and experience with all of them:
Thanks for this!
>
> * DAT -- JavaScript, npm... no, thanks. However I like the idea
Being an append-only log has me somewhat concerned. It seems to
be something like a "public Syncthing, but unidirectional." IPFS
seems to be catching on more than DAT at the moment.
> Many years ago I was very interested in all that stuff:
> anonymity,
> privacy, and so on, so on, so on. I am still interested, but
> VERY
> disappointed with the fact that hardly any of most of that kind
> of
> software works :-).
I keep trying to do my part around the edges, but yes, lack of
time is always an issue for me as well.
> But I will look closer at Syncthing, ION(-DTN). Thanks for
> sharing that
> information and thoughts!. And I still remember about
> asynchronous
> checksumming, checksumming during packets receiving and other
> issues,
> that are pretty high-priority tasks for NNCP!
And thank you, again, for NNCP. Unlike many of these other
things, it's fairly simple, well-documented, and secure and
private by default.
Heck, a person could even use Google Drive for the packets, if one
wanted.
- John
^ permalink raw reply [flat|nested] 5+ messages in thread
* Re: Thoughts on flexible mesh-routing of NNCP packets
2021-02-01 6:21 ` John Goerzen
@ 2021-02-01 8:39 ` Sergey Matveev
0 siblings, 0 replies; 5+ messages in thread
From: Sergey Matveev @ 2021-02-01 8:39 UTC (permalink / raw)
To: nncp-devel
[-- Attachment #1: Type: text/plain, Size: 1847 bytes --]
*** John Goerzen [2021-02-01 00:21]:
>Which reminds me - when you get back a file from nncp-freq, or any other
>event that sends something back, I'm assuming it uses the via path (if any)
>defined on the system that's sending back the response?
Yes, correct.
>So I wouldn't suggest anybody just rip out nncp-call(er) and go to Syncthing
>entirely; nncp-call and the daemon are quite useful and more efficient than
>Syncthing would be in fixed topology cases, or situations with a non-TCP
>transport.
I understand. There can not be killer solution for all cases.
>One particularly interesting use for this - Syncthing can to local LAN
>discovery.
Once I wanted to create same thing in nncp-daemon/caller -- multicasted
notifications about NNCP daemons presence.
>very nice alternative to things like Dropbox, or even Nextcloud, since it can
>be run entirely serverless and disconnected.
Dropbox is not the choice of course, because it is not free software.
>Being an append-only log has me somewhat concerned. It seems to be something
>like a "public Syncthing, but unidirectional." IPFS seems to be catching on
>more than DAT at the moment.
I like DAT idea because it is very close to WARC:
https://en.wikipedia.org/wiki/Web_ARChive
whole website distribution through single file.
And DAT (seems) allows to effectively publish changes to it.
It seems can be conveniently archived for long term storage.
>And thank you, again, for NNCP. Unlike many of these other things, it's
>fairly simple, well-documented, and secure and private by default.
Hope so! Glad you like it!
>Heck, a person could even use Google Drive for the packets, if one wanted.
Yes, of course, no barriers.
--
Sergey Matveev (http://www.stargrave.org/)
OpenPGP: CF60 E89A 5923 1E76 E263 6422 AE1A 8109 E498 57EF
[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]
^ permalink raw reply [flat|nested] 5+ messages in thread
end of thread, other threads:[~2021-02-01 8:39 UTC | newest]
Thread overview: 5+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2021-01-31 7:03 Thoughts on flexible mesh-routing of NNCP packets John Goerzen
2021-01-31 11:19 ` Sergey Matveev
2021-02-01 6:21 ` John Goerzen
2021-02-01 8:39 ` Sergey Matveev
2021-01-31 15:16 ` Sergey Matveev