public inbox for nncp-devel@lists.cypherpunks.ru
Atom feed
* [EN] NNCP 8.6.0 release announcement
@ 2022-03-02 14:57 Sergey Matveev
2022-03-02 16:18 ` John Goerzen
0 siblings, 1 reply; 8+ messages in thread
From: Sergey Matveev @ 2022-03-02 14:57 UTC (permalink / raw)
To: nncp-devel
[-- Attachment #1.1: Type: text/plain, Size: 2461 bytes --]
I am pleased to announce NNCP 8.6.0 release availability!
NNCP (Node to Node copy) is a collection of utilities simplifying
secure store-and-forward files and mail exchanging.
This utilities are intended to help build up small size (dozens of
nodes) ad-hoc friend-to-friend (F2F) statically routed darknet
delay-tolerant networks for fire-and-forget secure reliable files, file
requests, Internet mail and commands transmission. All packets are
integrity checked, end-to-end encrypted (E2EE), explicitly authenticated
by known participants public keys. Onion encryption is applied to
relayed packets. Each node acts both as a client and server, can use
push and poll behaviour model. Also there is multicasting areas support.
Out-of-box offline sneakernet/floppynet, dead drops, sequential and
append-only CD-ROM/tape storages, air-gapped computers support. But
online TCP daemon with full-duplex resumable data transmission exists.
------------------------ >8 ------------------------
The main improvements for that release are:
* "nncp-xfer" checks if locally copied packet’s checksum differs from
the source’s one.
* "nncp-call" has bad return code if its queues contain unfinished
jobs.
* "nncp-ack" command appeared, that sends explicit packet receipt
acknowledgement (ACK packet). That acknowledgement deletes
referenced packet from the outbound spool.
* Ability to turn "fsync" operation off using "$NNCPNOSYNC=1"
environment variable.
* Added various indices in documentation, that should help searching
in it.
------------------------ >8 ------------------------
NNCP's home page is: http://www.nncpgo.org/
Source code and its signature for that version can be found here:
http://www.nncpgo.org/download/nncp-8.6.0.tar.xz (1670 KiB)
http://www.nncpgo.org/download/nncp-8.6.0.tar.xz.sig
SHA256 hash: AE16F0A0 9C1B7D1D A3767E1C 0F410D9C 29ACCAE6 32A448A9 55DB4A0F 15BF838F
GPG key ID: 0x2B25868E75A1A953 NNCP releases <releases@nncpgo•org>
Fingerprint: 92C2 F0AE FE73 208E 46BF F3DE 2B25 868E 75A1 A953
There are mirrors where you can also get the source code tarballs:
http://www.nncpgo.org/Mirrors.html
Please send questions regarding the use of NNCP, bug reports and patches
to mailing list: http://lists.cypherpunks.ru/nncp_002ddevel.html
--
Sergey Matveev (http://www.stargrave.org/)
OpenPGP: CF60 E89A 5923 1E76 E263 6422 AE1A 8109 E498 57EF
[-- Attachment #1.2: nncp-8.6.0.tar.xz.meta4 --]
[-- Type: application/metalink4+xml, Size: 1337 bytes --]
[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]
^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: [EN] NNCP 8.6.0 release announcement
2022-03-02 14:57 [EN] NNCP 8.6.0 release announcement Sergey Matveev
@ 2022-03-02 16:18 ` John Goerzen
2022-03-02 16:22 ` John Goerzen
2022-03-02 18:47 ` Sergey Matveev
0 siblings, 2 replies; 8+ messages in thread
From: John Goerzen @ 2022-03-02 16:18 UTC (permalink / raw)
To: Sergey Matveev; +Cc: nncp-devel
On Wed, Mar 02 2022, Sergey Matveev wrote:
> I am pleased to announce NNCP 8.6.0 release availability!
Thank you, Sergey! I will get to work packaging this version right
away. This will be very helpful right away!
I'm starting to plan out my implementation...
I have two questions about ACK packet generation.
First, let's say I have two machines, A and B, that are airgapped and
they communicate using nncp-xfer and a USB stick that moves between
them. I want to use nncp-ack for all the reasons we talked about, on
BOTH ends.
That raises the question of: how do we handle the ACK packets
themselves?
In theory, we sort of have two options:
1) Never keep ACK packets on the source machine, and never send ACKs for
them on the destination.
2) Handle them like any other packet. But this implies an endless
stream of re-ACKing.
I think option #1 is preferable. If an ACK is lost, there's no harm
(particularly if -seen is in use, which it should be in this cast [might
want to mention that hint on the nncp-ack documentation page]). The
sender will simply retransmit, the recipient will re-ACK and then toss
will ignore it as it's been seen.
That implies two questions then: how to prevent nncp-ack from itself
ACKing an ACK, and how to cause nncp-xfer to not keep ACKs?
I *THINK* we can do this with niceness. If so, maybe this could be
included in the doc page also, or maybe there should be some code in
there so that nncp-ack doesn't ack an ack...
So...
# First receive ACKs
nncp-xfer -rx -nice 1 /mnt/shared
# Toss them, which will delete them before they can be acked.
nncp-toss -nice 1
# Now receive the rest of the packets
nncp-xfer -rx /mnt/shared
# Ack them
nncp-ack -nice 1 -node alice
# Send the ACKs, without -keep
nncp-xfer -tx -nice 1 -node alice /mnt/shared
# Toss the rest
nncp-toss
# Send the other replies, with -keep
nncp-xfer -tx -keep -node alice /mnt/shared
Somewhat related - I have nncp-toss running as a daemon (-cycle) mode on
some nodes that I will want to enable ack generation for. That implies
a race condition; will nncp-ack or nncp-toss get to the packets first,
after nncp-xfer copies them into the rx directory?
I would be fine with an -ack option to nncp-toss (similar to the -seen
option we already have). Yes, that may generate unnecessary ACKs to
systems that are communicated with by call/daemon, but it probably makes
things very simple and I'm fine with that very small overhead.
Alternatively, there could be an ack option added to the neigh
configuration for a particular node, which would cause toss to always
generate ack packets when it processes packets inbound from that node.
... in thinking about this a bit more, I suppose this may need some
additional logic such that toss keeps track of what packets it's already
ACKd. With something like -cycle 60 and a packet for which toss fails,
that could generate thousands of ACKs per day. I guess nncp-ack would
also generate duplicate ACKs, but most likely in a far smaller quantity.
This may just over-complicate things.
- John
^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: [EN] NNCP 8.6.0 release announcement
2022-03-02 16:18 ` John Goerzen
@ 2022-03-02 16:22 ` John Goerzen
2022-03-02 18:55 ` Sergey Matveev
2022-03-02 18:47 ` Sergey Matveev
1 sibling, 1 reply; 8+ messages in thread
From: John Goerzen @ 2022-03-02 16:22 UTC (permalink / raw)
To: Sergey Matveev; +Cc: nncp-devel
On Wed, Mar 02 2022, John Goerzen wrote:
> I think option #1 is preferable. If an ACK is lost, there's no harm
> (particularly if -seen is in use, which it should be in this cast [might
> want to mention that hint on the nncp-ack documentation page]). The
> sender will simply retransmit, the recipient will re-ACK and then toss
> will ignore it as it's been seen.
Hmm, now I see that nncp-xfer -rx will ignore packets that have been
seen. That would mean that if an ACK is lost, the sender will keep
retransmitting the packet forever, right? I'm not quite sure what to do
about this yet, hmm.
^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: [EN] NNCP 8.6.0 release announcement
2022-03-02 16:18 ` John Goerzen
2022-03-02 16:22 ` John Goerzen
@ 2022-03-02 18:47 ` Sergey Matveev
2022-03-02 18:59 ` John Goerzen
1 sibling, 1 reply; 8+ messages in thread
From: Sergey Matveev @ 2022-03-02 18:47 UTC (permalink / raw)
To: nncp-devel
[-- Attachment #1: Type: text/plain, Size: 4684 bytes --]
*** John Goerzen [2022-03-02 10:18]:
>In theory, we sort of have two options:
>
>1) Never keep ACK packets on the source machine, and never send ACKs for
>them on the destination.
>
>2) Handle them like any other packet. But this implies an endless
>stream of re-ACKing.
I honestly forget about an endless reACKing problem. Somehow we must not
send ACK for an ACK obviously. Currently nncp-ack is a separate command
call, that just generates ACK for everything it saw in rx-directory.
nncp-toss is a candidate for that job instead, because it sees the plain
(unencrypted) packet's type. But that will lead to more configuration
file's options to control if ACKs need to be generated. However,
nncp-ack already has access to private keys (that is required to
generate outbound encrypted packets), so it also can decrypt packets.
>That implies two questions then: how to prevent nncp-ack from itself
>ACKing an ACK, and how to cause nncp-xfer to not keep ACKs?
First part is solved by partly decrypting the packet in rx directory. We
need to decrypt only its first chunk, to view the plain packet's header
to determine the type. Basically it requires opening of each file,
reading up to 128+KiBs of data and two asymmetric cryptography
operations. I think that is acceptable.
Second part currently is not solvable. Because all encrypted packets are
indistinguishable from the noise, except for some header information,
with niceness level, that actually is not directly related to payload
type. I did not want to add visible packet's type outside the ciphertext
very much. nncp-xfer does not have access to private keys by design.
For example there can be an additional state which keeps knowledge of
ACK packets identifiers. Even empty "PKT.ack" nearby the "PKT" in the
spool area would satisfy that, but that means leaking of exactly that
special packet type knowledge. Currently it can be done for example by
enabling -debug logging during nncp-ack, parsing output recfile to get
generated ACK packets filenames. Then it can be used to remove those
packets after the transfer for example. I agree that lost ACK is not a
big problem. nncp-ack could have something like:
nncp-ack -list-of-generated-acks acks.txt ...
nncp-xfer / whatever
nncp-rm -list-of-generated-acks acks.txt
That file can be placed in memory, because of the small size.
>I *THINK* we can do this with niceness.
Probably, but I do not like that idea at all. Niceness is not a "type of
the packet", however it is tempting to use it for that. Currently there
are different predefined default niceness levels for each type of
packet, but user is free to override it in anyway, for example to
completely remove any difference between all packets at all (of course
loosing ability to prioritize the packets).
>Somewhat related - I have nncp-toss running as a daemon (-cycle) mode on
>some nodes that I will want to enable ack generation for.
I tend to treat "nncp-toss -cycle" like a mistake done previously. It
complicates things very much, because of concurrent access, because
every option is have to be in configuration file, instead of
simpler-to-process command line arguments. But ok, -cycle exists, I
won't remove it, but I think that it will be much better to use
shell-script wrapper around it, probably taking more resources.
Currently you have to stop nncp-toss-cycle when you work with nncp-xfer,
to be able to run nncp-ack (because tossing will remove the packet)
before tossing. Moving ACKing to tosser -- will be a huge complication.
Currently I am in position that nncp-toss-cycle is not friendly with
nncp-xfer/nncp-bundle. It is even harmful, because while you copy files
from the external media, tosser will do tossing in background, requiring
to do read operations (at least) while nncp-xfer/bundle does writing at
the same time. So do not want to think about how all of that will work
with nncp-toss-cycle :-)
So currently I see two relatively simple changes I have to done:
* nncp-ack must (partly) decrypt packets, to determine if it ACK and skip it
* nncp-ack must generate list of generated ACK encrypted packets, to use
it at least with nncp-rm, to be able to remove them after xfer transfer.
* nncp-rm could take a list of PKTs from stdin for convenience there
* probably nncp-xfer can take a list of packets (from the file/stdin) to
skip. To be able to leave ACK packets in the spool just for a while
nncp-ack won't generate ACK for ACK, fixing the biggest problem.
--
Sergey Matveev (http://www.stargrave.org/)
OpenPGP: CF60 E89A 5923 1E76 E263 6422 AE1A 8109 E498 57EF
[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]
^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: [EN] NNCP 8.6.0 release announcement
2022-03-02 16:22 ` John Goerzen
@ 2022-03-02 18:55 ` Sergey Matveev
0 siblings, 0 replies; 8+ messages in thread
From: Sergey Matveev @ 2022-03-02 18:55 UTC (permalink / raw)
To: nncp-devel
[-- Attachment #1: Type: text/plain, Size: 1154 bytes --]
*** John Goerzen [2022-03-02 10:22]:
>Hmm, now I see that nncp-xfer -rx will ignore packets that have been
>seen. That would mean that if an ACK is lost, the sender will keep
>retransmitting the packet forever, right? I'm not quite sure what to do
>about this yet, hmm.
Seems so indeed. Well, that can be solved if nncp-xfer will output the
list of packets it transferred now and before, and nncp-ack can use that
list of files he had to ACK. That "state sharing" has never been done
before in NNCP, that seems like some kind of overcomplication now.
However I knew since the very beginning that ACKs will be painful :-)
Will think about all of that. I like that nncp-ack/toss/xfer are more or
less separated and isolated things from each other: xfer does not need
private keys, toss does not know anything about how the packet is
transferred, and so on. Making a combine that does everything, having
all the state it wants -- tempting, but ugly, in my opinion.
Intermediate state files currently seem at least less ugly.
--
Sergey Matveev (http://www.stargrave.org/)
OpenPGP: CF60 E89A 5923 1E76 E263 6422 AE1A 8109 E498 57EF
[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]
^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: [EN] NNCP 8.6.0 release announcement
2022-03-02 18:47 ` Sergey Matveev
@ 2022-03-02 18:59 ` John Goerzen
2022-03-02 19:26 ` Sergey Matveev
0 siblings, 1 reply; 8+ messages in thread
From: John Goerzen @ 2022-03-02 18:59 UTC (permalink / raw)
To: Sergey Matveev; +Cc: nncp-devel
On Wed, Mar 02 2022, Sergey Matveev wrote:
> *** John Goerzen [2022-03-02 10:18]:
>>In theory, we sort of have two options:
>>
>>1) Never keep ACK packets on the source machine, and never send ACKs for
>>them on the destination.
>>
>>2) Handle them like any other packet. But this implies an endless
>>stream of re-ACKing.
>
> I honestly forget about an endless reACKing problem. Somehow we must not
> send ACK for an ACK obviously. Currently nncp-ack is a separate command
> call, that just generates ACK for everything it saw in rx-directory.
> nncp-toss is a candidate for that job instead, because it sees the plain
> (unencrypted) packet's type. But that will lead to more configuration
> file's options to control if ACKs need to be generated. However,
> nncp-ack already has access to private keys (that is required to
> generate outbound encrypted packets), so it also can decrypt packets.
Hi Sergey,
I just had another thought - maybe the ideal place to generate ACKs is
in the nncp-xfer/bundle rx code itself?
That would allow it to:
- Generate an ACK for an incoming packet even if there is a .seen for it
- Avoid the race with toss
- Decrypt the first chunk and avoid generating ACKs for ACKs
> Probably, but I do not like that idea at all. Niceness is not a "type of
> the packet", however it is tempting to use it for that. Currently there
Agreed, that is a workaround.
[ regarding toss -cycle ]
I find it very useful for machines that primarily interact via
daemon/call/caller where there is not always a definite "done" state
even.
> So currently I see two relatively simple changes I have to done:
>
> * nncp-ack must (partly) decrypt packets, to determine if it ACK and skip it
> * nncp-ack must generate list of generated ACK encrypted packets, to use
> it at least with nncp-rm, to be able to remove them after xfer transfer.
> * nncp-rm could take a list of PKTs from stdin for convenience there
> * probably nncp-xfer can take a list of packets (from the file/stdin) to
> skip. To be able to leave ACK packets in the spool just for a while
>
> nncp-ack won't generate ACK for ACK, fixing the biggest problem.
So generating ACKs in nncp-xfer/bundle still leaves the problem of keep
for them. Maybe the answer would be a simple option to xfer/bundle tx:
-always-delete-sent-acks or something. Then an ACK that's transmitted
would always be deleted, regardless of -keep. Combined with -seen for
toss, that should provide good behavior in all cases. (If an ACK is
lost, the sender will retransmit the packet, but since it's already been
seen, the recipient will immediately generate a new ACK)
- John
^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: [EN] NNCP 8.6.0 release announcement
2022-03-02 18:59 ` John Goerzen
@ 2022-03-02 19:26 ` Sergey Matveev
2022-03-03 3:10 ` John Goerzen
0 siblings, 1 reply; 8+ messages in thread
From: Sergey Matveev @ 2022-03-02 19:26 UTC (permalink / raw)
To: nncp-devel
[-- Attachment #1: Type: text/plain, Size: 3287 bytes --]
*** John Goerzen [2022-03-02 12:59]:
>I just had another thought - maybe the ideal place to generate ACKs is
>in the nncp-xfer/bundle rx code itself?
Actually initially I started exactly in them. But:
* if ACKs are in use, xfer/bundle (let's call them xfers) must use the
config with private keys. I do not use that use-case, but last time I
checked, you can use xfers with nncp-cfgmin-imized configuration file
version, that you safely can keep on unsafe storage/device. For
example you take a laptop with minimized configuration, keep the spool
on it, go outside, get connection with xfers in a place, where your
laptop can be stolen/compromised. Anyway it is used only for just
copying of files from one place (storages, tapes, whatever) to
another. No private keys (even encrypted ones (nncp-cfgenc)) are used
If we move all ACKing only to xfers, then there will be no way to
use xfers only for transport and some other thing only for ACKing.
Those are two completely different tasks with different requirements
* I used bundle with the tape drive, that REALLY has to be able just to
stream the data without any interruptions. That is why nncp-bundle
does not check checksum on the fly (I used all of that previously on
pretty low-performance CPU and BLAKE3 was not used in NNCP at that
time). And that is also desireable for HDDs too, to be able to do as
much sequential operation as it could (but it is not as critical as
with tapes). If xfers will generate ACKs on the fly after each
received packet, then it will interrupt sequential operations, it will
"touch" our spool or xfer's mounted directory
* we could just "remember" what ACKs we have to generate at the end of
the process, but... if the process is suddenly interrupted? We loose
that state and any knowledge of what was received and what we must ACK
>- Avoid the race with toss
Currently I am sure that it should not be run at all, because of
obviously undefined behaviour.
>I find it very useful for machines that primarily interact via
>daemon/call/caller where there is not always a definite "done" state
>even.
I do not find the idea of running it sometimes in background, while
those (long-live!) daemons are running, bad. I find that there sould be
used daemontools'es supervise service with it, like some kind of trivial:
$ cat services/nncp-toss/run
sleep 60
exec nncp-toss ...
which you will "down" when you insert your removable media to do
xfering and acking:
$ svc -d services/nncp-toss
$ nncp-xfer ...
$ nncp-ack ...
$ svc -u services/nncp-toss
>So generating ACKs in nncp-xfer/bundle still leaves the problem of keep
>for them.
Yes, that is independent problem. That I currently plan to solve with
temporary intermediate file generated by xfers with a list of packets
you need to ACK, that is used by nncp-ack/rm after.
>Maybe the answer would be a simple option to xfer/bundle tx:
>-always-delete-sent-acks or something.
If xfers will be capable to generated packets, then yes. But I am
against that, for the reasons at top of that letter.
--
Sergey Matveev (http://www.stargrave.org/)
OpenPGP: CF60 E89A 5923 1E76 E263 6422 AE1A 8109 E498 57EF
[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]
^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: [EN] NNCP 8.6.0 release announcement
2022-03-02 19:26 ` Sergey Matveev
@ 2022-03-03 3:10 ` John Goerzen
0 siblings, 0 replies; 8+ messages in thread
From: John Goerzen @ 2022-03-03 3:10 UTC (permalink / raw)
To: Sergey Matveev; +Cc: nncp-devel
On Wed, Mar 02 2022, Sergey Matveev wrote:
> * I used bundle with the tape drive, that REALLY has to be able just to
I remember this about tape drives, yes :-)
> * we could just "remember" what ACKs we have to generate at the end of
> the process, but... if the process is suddenly interrupted? We loose
> that state and any knowledge of what was received and what we must ACK
Makes sense.
> which you will "down" when you insert your removable media to do
> xfering and acking:
>
> $ svc -d services/nncp-toss
> $ nncp-xfer ...
> $ nncp-ack ...
> $ svc -u services/nncp-toss
Of course, that could still work with -cycle as a service. service
nncp-toss stop, service nncp-toss start, etc.
> If xfers will be capable to generated packets, then yes. But I am
> against that, for the reasons at top of that letter.
Understood. Your state file idea would work for me too.
Thanks!
- John
^ permalink raw reply [flat|nested] 8+ messages in thread
end of thread, other threads:[~2022-03-03 3:11 UTC | newest]
Thread overview: 8+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2022-03-02 14:57 [EN] NNCP 8.6.0 release announcement Sergey Matveev
2022-03-02 16:18 ` John Goerzen
2022-03-02 16:22 ` John Goerzen
2022-03-02 18:55 ` Sergey Matveev
2022-03-02 18:47 ` Sergey Matveev
2022-03-02 18:59 ` John Goerzen
2022-03-02 19:26 ` Sergey Matveev
2022-03-03 3:10 ` John Goerzen