public inbox for nncp-devel@lists.cypherpunks.ru
Atom feed
* [EN] NNCP 7.0.0 release announcement
@ 2021-06-30 12:31 Sergey Matveev
2021-06-30 13:28 ` John Goerzen
2021-06-30 23:39 ` [EN] NNCP 7.0.0 release announcement Shawn K. Quinn
0 siblings, 2 replies; 9+ messages in thread
From: Sergey Matveev @ 2021-06-30 12:31 UTC (permalink / raw)
To: nncp-devel
[-- Attachment #1: Type: text/plain, Size: 3026 bytes --]
I am pleased to announce NNCP 7.0.0 release availability!
NNCP (Node to Node copy) is a collection of utilities simplifying
secure store-and-forward files and mail exchanging.
This utilities are intended to help build up small size (dozens of
nodes) ad-hoc friend-to-friend (F2F) statically routed darknet
delay-tolerant networks for fire-and-forget secure reliable files, file
requests, Internet mail and commands transmission. All packets are
integrity checked, end-to-end encrypted (E2EE), explicitly authenticated
by known participants public keys. Onion encryption is applied to
relayed packets. Each node acts both as a client and server, can use
push and poll behaviour model.
Out-of-box offline sneakernet/floppynet, dead drops, sequential and
append-only CD-ROM/tape storages, air-gapped computers support. But
online TCP daemon with full-duplex resumable data transmission exists.
------------------------ >8 ------------------------
The main improvements for that release are:
* Merkle Tree-based Hashing with BLAKE3 (MTH) is used instead of
BLAKE2b. Because of that, there are backward *incompatible*
changes of encrypted files (everything laying in the spool
directory) and ".meta" files of chunked transfer.
Current implementation is far from being optimal: it lacks
parallelizable calculations and has higher memory consumption:
nearly 512 KiB for each 1 GiB of file’s data. Future performance
and memory size optimizations should not lead to packet’s format
change. But it is still several times faster than BLAKE2b.
* Resumed online downloads, because of MTH, require reading only of
the preceding part of file, not the whole one as was before.
* "nncp-hash" utility appeared for calculating file’s MTH hash.
* BLAKE2 KDF and XOF functions are replaced with BLAKE3 in encrypted
packets. Lowering number of used primitives. Also, its encrypted
packet’s header is used as an associated data during encryption.
* MultiCast Discovery uses ff02::4e4e:4350 address instead of
ff02::1.
* "nncp-cfgenc" mistakenly asked passphrase three times during
encryption.
* "nncp-stat" reports about partly downloaded packets.
* Updated dependencies.
------------------------ >8 ------------------------
NNCP's home page is: http://www.nncpgo.org/
Source code and its signature for that version can be found here:
http://www.nncpgo.org/download/nncp-7.0.0.tar.xz (1123 KiB)
http://www.nncpgo.org/download/nncp-7.0.0.tar.xz.sig
SHA256 hash: D4D28E9A CF40FE12 68BDE134 9CD36076 282395BE 70094EFB 0DB75CE8 C32EA664
GPG key ID: 0x2B25868E75A1A953 NNCP releases <releases@nncpgo•org>
Fingerprint: 92C2 F0AE FE73 208E 46BF F3DE 2B25 868E 75A1 A953
Please send questions regarding the use of NNCP, bug reports and patches
to mailing list: http://lists.cypherpunks.ru/nncp_002ddevel.html
--
Sergey Matveev (http://www.stargrave.org/)
OpenPGP: CF60 E89A 5923 1E76 E263 6422 AE1A 8109 E498 57EF
[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]
^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: [EN] NNCP 7.0.0 release announcement
2021-06-30 12:31 [EN] NNCP 7.0.0 release announcement Sergey Matveev
@ 2021-06-30 13:28 ` John Goerzen
2021-06-30 14:09 ` MTH Sergey Matveev
2021-06-30 14:31 ` nncp-redir Sergey Matveev
2021-06-30 23:39 ` [EN] NNCP 7.0.0 release announcement Shawn K. Quinn
1 sibling, 2 replies; 9+ messages in thread
From: John Goerzen @ 2021-06-30 13:28 UTC (permalink / raw)
To: Sergey Matveev; +Cc: nncp-devel
Hi Sergey,
I need to read up more on these algorithms but this all sounds
very nice!
I have a question about memory usage:
On Wed, Jun 30 2021, Sergey Matveev wrote:
> * Merkle Tree-based Hashing with BLAKE3 (MTH) is used instead of
> BLAKE2b. Because of that, there are backward *incompatible*
> changes of encrypted files (everything laying in the spool
> directory) and ".meta" files of chunked transfer.
>
> Current implementation is far from being optimal: it lacks
> parallelizable calculations and has higher memory consumption:
> nearly 512 KiB for each 1 GiB of file’s data. Future
> performance
> and memory size optimizations should not lead to packet’s
> format
> change. But it is still several times faster than BLAKE2b.
I'm not familiar with MTH, but I'm guessing that's what leads to
the memory consumption that scales with the size of the data? I
would assume that BLAKE3, like other hashing algorithms, would be
constant-space?
I often process packets up to about 256GB in size, so I assume
that would lead to about 128MB of RAM usage. That's not ideal on
some of the lower-powered systems I use, particularly Raspberry
Pis, but I can probably live with it in general (the
lowest-powered ones don't tend to process data at that size).
Would I be correct in assuming that this is used in pretty much
all utilities? nncp-file/exec, call, daemon, toss?
I have been meaning to write a message about packet redirection as
well. I don't know if this makes it easier - I don't think so -
but I thought I might mention.
So I have a use case like this.
My node "laptop" sends backups to a machine on the LAN, I'll call
it "baksvr". Usually I only want it to send backups over the LAN
(in case I'm tethered to 4G or something), but occasionally I want
it to send backups remotely.
Now the machine on my home LAN isn't reachable from the internet,
but it does have an NNCP link to a VPS out there, which my laptop
could reach.
So I have packets queued up in my tx directory on the laptop for
baksvr, but I want to redirect them via the VPS.
I eventually wound up using nncp-bundle to bundle them up, sent
that bundle across NNCP, and unpacked it on the destination. That
can even be automated with nncp-exec -- but it might be nice to
have something like "nncp-redir" that could redirect queued
packets, adding additional onion routing layers around what's
there already.
- John
^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: MTH
2021-06-30 13:28 ` John Goerzen
@ 2021-06-30 14:09 ` Sergey Matveev
2021-06-30 14:31 ` nncp-redir Sergey Matveev
1 sibling, 0 replies; 9+ messages in thread
From: Sergey Matveev @ 2021-06-30 14:09 UTC (permalink / raw)
To: nncp-devel
[-- Attachment #1: Type: text/plain, Size: 2578 bytes --]
Greetings!
*** John Goerzen [2021-06-30 06:28]:
>I'm not familiar with MTH, but I'm guessing that's what leads to the memory
>consumption that scales with the size of the data?
Just want to remark that "MTH" is just my own name of the hashing used
in NNCP. Technically it is ordinary BLAKE3 that creates binary tree of
hashes, completely similar to one used in Tiger Tree Hash (TTH),
Certificate Transparency and huge quantity of other projects. So nothing
anything new, except that I use keyed hashing instead of prepending of
0x00/0x01 to leaf/node data to differentiate them: http://www.nncpgo.org/MTH.html
>I would assume that
>BLAKE3, like other hashing algorithms, would be constant-space?
Yes, right.
>I often process packets up to about 256GB in size, so I assume that would
>lead to about 128MB of RAM usage. That's not ideal on some of the
>lower-powered systems I use, particularly Raspberry Pis, but I can probably
>live with it in general (the lowest-powered ones don't tend to process data
>at that size). Would I be correct in assuming that this is used in pretty
>much all utilities? nncp-file/exec, call, daemon, toss?
Yeah, it is used nearly everywhere: everything that deals with encrypted
packets. Higher memory consumption is only a temporary thing now! MTH
hashes data the following way (ordinary Merkle tree):
node3
/ \
/ \
node2 leaf4
/ \ \
/ \ \
/ \ \
/ \ \
/ \ \
node0 node1 leaf4
/ \ / \ \
/ \ / \ \
leaf0 leaf1 leaf2 leaf3 leaf4
| | | | |
block block block block block
and *currently* all leafs (hashes of 128KiB blocks) are kept in memory.
Only during the final hashing all of them "fold" to nodes. So that is
why 1TiB/128KiB*256bit=256MiB required to keep the leafs. And additional
256MiB/2=128MiB is needed for making the first level of nodes (that are
twice less). And of course possible various overheads.
Of course you can collapse/fold huge quantity of all those intermediate
leafs and nodes. Current implementation is only a temporary solution! It
will be optimized and require negligible amount of memory. Of course
without changing the format.
--
Sergey Matveev (http://www.stargrave.org/)
OpenPGP: CF60 E89A 5923 1E76 E263 6422 AE1A 8109 E498 57EF
[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]
^ permalink raw reply [flat|nested] 9+ messages in thread
* nncp-redir
2021-06-30 13:28 ` John Goerzen
2021-06-30 14:09 ` MTH Sergey Matveev
@ 2021-06-30 14:31 ` Sergey Matveev
2021-06-30 17:25 ` nncp-redir Sergey Matveev
1 sibling, 1 reply; 9+ messages in thread
From: Sergey Matveev @ 2021-06-30 14:31 UTC (permalink / raw)
To: nncp-devel
[-- Attachment #1: Type: text/plain, Size: 1728 bytes --]
Greetings!
*** John Goerzen [2021-06-30 06:28]:
>but it might be nice to have something like "nncp-redir"
>that could redirect queued packets, adding additional onion routing layers
>around what's there already.
Well, as I can see that could be done rather easily. Transition packets
literally contains an ordinary encrypted packets, that you already have
in your case.
However I am thinking again about possibility to have multiple
recipients in one encrypted packet. In your case there will be
transitional packet to VPS and backsvr. If it will reach VPS, then it
will unpack it and create "de-proxied" message to the backsvr. If it
will reach backsvr, then it can open both the transitional packet and an
encrypted packet inside it.
It close to the idea of multicasting, where I can send single packet for
multiple recipients to spread the data more efficiently. I just can not
forget the fact that in networks like FidoNet all the time (once per
week, as I remember) an updated "nodelist" is sent to all the nodes in
the network. Making enormous number of copies to each node would be
wasteful, so FidoNet has multicasting abilities. NNCP currently lacks it
at all. But I am not sure that it have to be a part of NNCP itself, but
probably an external echoconference-like processing nncp-exec-ed
software dealing with the "subscription" (multicast group membership)
states and similar things.
I will think about all of it more closely (all of us want simple
solutions :-)) and if won't come to any conclusions, then will make
"nncp-redir" utility, that should be dead simple.
--
Sergey Matveev (http://www.stargrave.org/)
OpenPGP: CF60 E89A 5923 1E76 E263 6422 AE1A 8109 E498 57EF
[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]
^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: nncp-redir
2021-06-30 14:31 ` nncp-redir Sergey Matveev
@ 2021-06-30 17:25 ` Sergey Matveev
2021-07-01 15:03 ` nncp-redir John Goerzen
0 siblings, 1 reply; 9+ messages in thread
From: Sergey Matveev @ 2021-06-30 17:25 UTC (permalink / raw)
To: nncp-devel
[-- Attachment #1: Type: text/plain, Size: 1075 bytes --]
*** Sergey Matveev [2021-06-30 17:31]:
>I will think about all of it more closely (all of us want simple
>solutions :-)) and if won't come to any conclusions, then will make
>"nncp-redir" utility, that should be dead simple.
Your case is when the node can be reached via multiple routes. I can not
think of anything that won't lead to full-featured routing system, at
least that exists in FTN-like networks. And I do not wish to do that
kind of complicated things. NNCP is by simplicity design uses static
manually setup routes. Multiple recipients won't help there: it could
help, with transitional packet inside, only if there is not more than
one hop between. So I have got no thoughts how can simply and
beautifully solve you case. It is too complicated for NNCP, as well as
UUCP too.
I will write nncp-trns (transition, that "trns" is already used in many
places in NNCP) utility for wrapping/redirecting packets through another
node(s).
--
Sergey Matveev (http://www.stargrave.org/)
OpenPGP: CF60 E89A 5923 1E76 E263 6422 AE1A 8109 E498 57EF
[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]
^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: [EN] NNCP 7.0.0 release announcement
2021-06-30 12:31 [EN] NNCP 7.0.0 release announcement Sergey Matveev
2021-06-30 13:28 ` John Goerzen
@ 2021-06-30 23:39 ` Shawn K. Quinn
2021-07-01 7:28 ` lukechampine.com/blake3 build fail Sergey Matveev
1 sibling, 1 reply; 9+ messages in thread
From: Shawn K. Quinn @ 2021-06-30 23:39 UTC (permalink / raw)
To: nncp-devel
On 6/30/21 07:31, Sergey Matveev wrote:
> I am pleased to announce NNCP 7.0.0 release availability!
>
> NNCP (Node to Node copy) is a collection of utilities simplifying
> secure store-and-forward files and mail exchanging.
>
> This utilities are intended to help build up small size (dozens of
> nodes) ad-hoc friend-to-friend (F2F) statically routed darknet
> delay-tolerant networks for fire-and-forget secure reliable files, file
> requests, Internet mail and commands transmission.
[...]
> Please send questions regarding the use of NNCP, bug reports and patches
> to mailing list: http://lists.cypherpunks.ru/nncp_002ddevel.html
skquinn@moonpatrol:~/usr/src/nncp-7.0.0$ redo all
redo all
redo bin/all
redo bin/nncp-bundle
# lukechampine.com/blake3
vendor/lukechampine.com/blake3/blake3.go:71:19: invalid operation: 1 <<
i (shift count type int, must be unsigned integer)
redo bin/nncp-bundle (exit 2)
redo bin/all (exit 1)
redo all (exit 1)
Should be pretty self-explanatory. Anyone else seeing this?
--
Shawn K. Quinn <skquinn@rushpost•com>
http://www.rantroulette.com
http://www.skqrecordquest.com
^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: lukechampine.com/blake3 build fail
2021-06-30 23:39 ` [EN] NNCP 7.0.0 release announcement Shawn K. Quinn
@ 2021-07-01 7:28 ` Sergey Matveev
0 siblings, 0 replies; 9+ messages in thread
From: Sergey Matveev @ 2021-07-01 7:28 UTC (permalink / raw)
To: nncp-devel
[-- Attachment #1: Type: text/plain, Size: 575 bytes --]
Greetings!
*** Shawn K. Quinn [2021-06-30 18:39]:
>(shift count type int, must be unsigned integer)
>Should be pretty self-explanatory. Anyone else seeing this?
Seems that Go 1.12 can not build that code. I checked with 1.13 -- build
fine. I will reflect that minimal version in installation documentation.
https://go.googlesource.com/proposal/+/master/design/19113-signed-shift-counts.md
Similar issue: https://github.com/minio/md5-simd/issues/17
--
Sergey Matveev (http://www.stargrave.org/)
OpenPGP: CF60 E89A 5923 1E76 E263 6422 AE1A 8109 E498 57EF
[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]
^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: nncp-redir
2021-06-30 17:25 ` nncp-redir Sergey Matveev
@ 2021-07-01 15:03 ` John Goerzen
2021-07-01 16:19 ` nncp-redir Sergey Matveev
0 siblings, 1 reply; 9+ messages in thread
From: John Goerzen @ 2021-07-01 15:03 UTC (permalink / raw)
To: Sergey Matveev; +Cc: nncp-devel
On Wed, Jun 30 2021, Sergey Matveev wrote:
> I will write nncp-trns (transition, that "trns" is already used
> in many
> places in NNCP) utility for wrapping/redirecting packets through
> another
> node(s).
Thank you! That makes good sense.
I haven't really had a need for multicasting (though I definitely
see your point about why it can be helpful), and as you say, that
could also be resolved at the software layer.
I have a related question about -via.
I over-simplified slightly in my initial explanation. In reality,
baksvr has no direct Internet access; there's a separate
langateway on the network. So what I had to do is:
nncp-file -via vps,langateway filename baksvr:
Now, the nncp.hjson on vps for baksvr configures it to be via
langateway. Initially I had forgotten about langateway and wrote:
nncp-file -via vps filename baksvr:
which didn't work, as they got queued up on vps for a poll from
baksvr that would never arrive.
So my question is, how does -via interact with the nncp.hjson via?
I vaguely remembered that one wouldn't necessarily have to spell
out the entire route initially, and could let, for example, vps
figure out how to get a packet to its destination if it had a via
entry. But I may be misremembering that.
Thanks again!
- John
^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: nncp-redir
2021-07-01 15:03 ` nncp-redir John Goerzen
@ 2021-07-01 16:19 ` Sergey Matveev
0 siblings, 0 replies; 9+ messages in thread
From: Sergey Matveev @ 2021-07-01 16:19 UTC (permalink / raw)
To: nncp-devel
[-- Attachment #1: Type: text/plain, Size: 1074 bytes --]
*** John Goerzen [2021-07-01 08:03]:
>So my question is, how does -via interact with the nncp.hjson via? I vaguely
>remembered that one wouldn't necessarily have to spell out the entire route
>initially, and could let, for example, vps figure out how to get a packet to
>its destination if it had a via entry. But I may be misremembering that.
I am afraid that transition packets always were just laid in the
corresponding spool outbound directory of the target node, without any
logic related to further relaying/transitioning. So you always had to
specify the full route.
But I also can not find any reason why not to look at destination node's
"via" option too. I think it would be safe to add automatic
transitioning if "via" is not empty for the destination node. It should
perfectly fit in your case where you do not specify the full path.
"via" option in nncp.hjson is just a default value for the specified
node of the -via option.
--
Sergey Matveev (http://www.stargrave.org/)
OpenPGP: CF60 E89A 5923 1E76 E263 6422 AE1A 8109 E498 57EF
[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]
^ permalink raw reply [flat|nested] 9+ messages in thread
end of thread, other threads:[~2021-07-01 16:19 UTC | newest]
Thread overview: 9+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2021-06-30 12:31 [EN] NNCP 7.0.0 release announcement Sergey Matveev
2021-06-30 13:28 ` John Goerzen
2021-06-30 14:09 ` MTH Sergey Matveev
2021-06-30 14:31 ` nncp-redir Sergey Matveev
2021-06-30 17:25 ` nncp-redir Sergey Matveev
2021-07-01 15:03 ` nncp-redir John Goerzen
2021-07-01 16:19 ` nncp-redir Sergey Matveev
2021-06-30 23:39 ` [EN] NNCP 7.0.0 release announcement Shawn K. Quinn
2021-07-01 7:28 ` lukechampine.com/blake3 build fail Sergey Matveev