public inbox for nncp-devel@lists.cypherpunks.ru Atom feed
* [EN] NNCP 8.6.0 release announcement
@ 2022-03-02 14:57 Sergey Matveev
2022-03-02 16:18 ` John Goerzen
0 siblings, 1 reply; 8+ messages in thread
From: Sergey Matveev @ 2022-03-02 14:57 UTC (permalink / raw)
To: nncp-devel
[-- Attachment #1.1: Type: text/plain, Size: 2461 bytes --]
I am pleased to announce NNCP 8.6.0 release availability!
NNCP (Node to Node copy) is a collection of utilities simplifying
secure store-and-forward files and mail exchanging.
This utilities are intended to help build up small size (dozens of
nodes) ad-hoc friend-to-friend (F2F) statically routed darknet
delay-tolerant networks for fire-and-forget secure reliable files, file
requests, Internet mail and commands transmission. All packets are
integrity checked, end-to-end encrypted (E2EE), explicitly authenticated
by known participants public keys. Onion encryption is applied to
relayed packets. Each node acts both as a client and server, can use
push and poll behaviour model. Also there is multicasting areas support.
Out-of-box offline sneakernet/floppynet, dead drops, sequential and
append-only CD-ROM/tape storages, air-gapped computers support. But
online TCP daemon with full-duplex resumable data transmission exists.
------------------------ >8 ------------------------
The main improvements for that release are:
* "nncp-xfer" checks if locally copied packet’s checksum differs from
the source’s one.
* "nncp-call" has bad return code if its queues contain unfinished
jobs.
* "nncp-ack" command appeared, that sends explicit packet receipt
acknowledgement (ACK packet). That acknowledgement deletes
referenced packet from the outbound spool.
* Ability to turn "fsync" operation off using "$NNCPNOSYNC=1"
environment variable.
* Added various indices in documentation, that should help searching
in it.
------------------------ >8 ------------------------
NNCP's home page is: http://www.nncpgo.org/
Source code and its signature for that version can be found here:
http://www.nncpgo.org/download/nncp-8.6.0.tar.xz (1670 KiB)
http://www.nncpgo.org/download/nncp-8.6.0.tar.xz.sig
SHA256 hash: AE16F0A0 9C1B7D1D A3767E1C 0F410D9C 29ACCAE6 32A448A9 55DB4A0F 15BF838F
GPG key ID: 0x2B25868E75A1A953 NNCP releases <releases@nncpgo•org>
Fingerprint: 92C2 F0AE FE73 208E 46BF F3DE 2B25 868E 75A1 A953
There are mirrors where you can also get the source code tarballs:
http://www.nncpgo.org/Mirrors.html
Please send questions regarding the use of NNCP, bug reports and patches
to mailing list: http://lists.cypherpunks.ru/nncp_002ddevel.html
--
Sergey Matveev (http://www.stargrave.org/)
OpenPGP: CF60 E89A 5923 1E76 E263 6422 AE1A 8109 E498 57EF
[-- Attachment #1.2: nncp-8.6.0.tar.xz.meta4 --]
[-- Type: application/metalink4+xml, Size: 1337 bytes --]
[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]
^ permalink raw reply [flat|nested] 8+ messages in thread* Re: [EN] NNCP 8.6.0 release announcement 2022-03-02 14:57 [EN] NNCP 8.6.0 release announcement Sergey Matveev @ 2022-03-02 16:18 ` John Goerzen 2022-03-02 16:22 ` John Goerzen 2022-03-02 18:47 ` Sergey Matveev 0 siblings, 2 replies; 8+ messages in thread From: John Goerzen @ 2022-03-02 16:18 UTC (permalink / raw) To: Sergey Matveev; +Cc: nncp-devel On Wed, Mar 02 2022, Sergey Matveev wrote: > I am pleased to announce NNCP 8.6.0 release availability! Thank you, Sergey! I will get to work packaging this version right away. This will be very helpful right away! I'm starting to plan out my implementation... I have two questions about ACK packet generation. First, let's say I have two machines, A and B, that are airgapped and they communicate using nncp-xfer and a USB stick that moves between them. I want to use nncp-ack for all the reasons we talked about, on BOTH ends. That raises the question of: how do we handle the ACK packets themselves? In theory, we sort of have two options: 1) Never keep ACK packets on the source machine, and never send ACKs for them on the destination. 2) Handle them like any other packet. But this implies an endless stream of re-ACKing. I think option #1 is preferable. If an ACK is lost, there's no harm (particularly if -seen is in use, which it should be in this cast [might want to mention that hint on the nncp-ack documentation page]). The sender will simply retransmit, the recipient will re-ACK and then toss will ignore it as it's been seen. That implies two questions then: how to prevent nncp-ack from itself ACKing an ACK, and how to cause nncp-xfer to not keep ACKs? I *THINK* we can do this with niceness. If so, maybe this could be included in the doc page also, or maybe there should be some code in there so that nncp-ack doesn't ack an ack... So... # First receive ACKs nncp-xfer -rx -nice 1 /mnt/shared # Toss them, which will delete them before they can be acked. nncp-toss -nice 1 # Now receive the rest of the packets nncp-xfer -rx /mnt/shared # Ack them nncp-ack -nice 1 -node alice # Send the ACKs, without -keep nncp-xfer -tx -nice 1 -node alice /mnt/shared # Toss the rest nncp-toss # Send the other replies, with -keep nncp-xfer -tx -keep -node alice /mnt/shared Somewhat related - I have nncp-toss running as a daemon (-cycle) mode on some nodes that I will want to enable ack generation for. That implies a race condition; will nncp-ack or nncp-toss get to the packets first, after nncp-xfer copies them into the rx directory? I would be fine with an -ack option to nncp-toss (similar to the -seen option we already have). Yes, that may generate unnecessary ACKs to systems that are communicated with by call/daemon, but it probably makes things very simple and I'm fine with that very small overhead. Alternatively, there could be an ack option added to the neigh configuration for a particular node, which would cause toss to always generate ack packets when it processes packets inbound from that node. ... in thinking about this a bit more, I suppose this may need some additional logic such that toss keeps track of what packets it's already ACKd. With something like -cycle 60 and a packet for which toss fails, that could generate thousands of ACKs per day. I guess nncp-ack would also generate duplicate ACKs, but most likely in a far smaller quantity. This may just over-complicate things. - John ^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: [EN] NNCP 8.6.0 release announcement 2022-03-02 16:18 ` John Goerzen @ 2022-03-02 16:22 ` John Goerzen 2022-03-02 18:55 ` Sergey Matveev 2022-03-02 18:47 ` Sergey Matveev 1 sibling, 1 reply; 8+ messages in thread From: John Goerzen @ 2022-03-02 16:22 UTC (permalink / raw) To: Sergey Matveev; +Cc: nncp-devel On Wed, Mar 02 2022, John Goerzen wrote: > I think option #1 is preferable. If an ACK is lost, there's no harm > (particularly if -seen is in use, which it should be in this cast [might > want to mention that hint on the nncp-ack documentation page]). The > sender will simply retransmit, the recipient will re-ACK and then toss > will ignore it as it's been seen. Hmm, now I see that nncp-xfer -rx will ignore packets that have been seen. That would mean that if an ACK is lost, the sender will keep retransmitting the packet forever, right? I'm not quite sure what to do about this yet, hmm. ^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: [EN] NNCP 8.6.0 release announcement 2022-03-02 16:22 ` John Goerzen @ 2022-03-02 18:55 ` Sergey Matveev 0 siblings, 0 replies; 8+ messages in thread From: Sergey Matveev @ 2022-03-02 18:55 UTC (permalink / raw) To: nncp-devel [-- Attachment #1: Type: text/plain, Size: 1154 bytes --] *** John Goerzen [2022-03-02 10:22]: >Hmm, now I see that nncp-xfer -rx will ignore packets that have been >seen. That would mean that if an ACK is lost, the sender will keep >retransmitting the packet forever, right? I'm not quite sure what to do >about this yet, hmm. Seems so indeed. Well, that can be solved if nncp-xfer will output the list of packets it transferred now and before, and nncp-ack can use that list of files he had to ACK. That "state sharing" has never been done before in NNCP, that seems like some kind of overcomplication now. However I knew since the very beginning that ACKs will be painful :-) Will think about all of that. I like that nncp-ack/toss/xfer are more or less separated and isolated things from each other: xfer does not need private keys, toss does not know anything about how the packet is transferred, and so on. Making a combine that does everything, having all the state it wants -- tempting, but ugly, in my opinion. Intermediate state files currently seem at least less ugly. -- Sergey Matveev (http://www.stargrave.org/) OpenPGP: CF60 E89A 5923 1E76 E263 6422 AE1A 8109 E498 57EF [-- Attachment #2: signature.asc --] [-- Type: application/pgp-signature, Size: 833 bytes --] ^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: [EN] NNCP 8.6.0 release announcement 2022-03-02 16:18 ` John Goerzen 2022-03-02 16:22 ` John Goerzen @ 2022-03-02 18:47 ` Sergey Matveev 2022-03-02 18:59 ` John Goerzen 1 sibling, 1 reply; 8+ messages in thread From: Sergey Matveev @ 2022-03-02 18:47 UTC (permalink / raw) To: nncp-devel [-- Attachment #1: Type: text/plain, Size: 4684 bytes --] *** John Goerzen [2022-03-02 10:18]: >In theory, we sort of have two options: > >1) Never keep ACK packets on the source machine, and never send ACKs for >them on the destination. > >2) Handle them like any other packet. But this implies an endless >stream of re-ACKing. I honestly forget about an endless reACKing problem. Somehow we must not send ACK for an ACK obviously. Currently nncp-ack is a separate command call, that just generates ACK for everything it saw in rx-directory. nncp-toss is a candidate for that job instead, because it sees the plain (unencrypted) packet's type. But that will lead to more configuration file's options to control if ACKs need to be generated. However, nncp-ack already has access to private keys (that is required to generate outbound encrypted packets), so it also can decrypt packets. >That implies two questions then: how to prevent nncp-ack from itself >ACKing an ACK, and how to cause nncp-xfer to not keep ACKs? First part is solved by partly decrypting the packet in rx directory. We need to decrypt only its first chunk, to view the plain packet's header to determine the type. Basically it requires opening of each file, reading up to 128+KiBs of data and two asymmetric cryptography operations. I think that is acceptable. Second part currently is not solvable. Because all encrypted packets are indistinguishable from the noise, except for some header information, with niceness level, that actually is not directly related to payload type. I did not want to add visible packet's type outside the ciphertext very much. nncp-xfer does not have access to private keys by design. For example there can be an additional state which keeps knowledge of ACK packets identifiers. Even empty "PKT.ack" nearby the "PKT" in the spool area would satisfy that, but that means leaking of exactly that special packet type knowledge. Currently it can be done for example by enabling -debug logging during nncp-ack, parsing output recfile to get generated ACK packets filenames. Then it can be used to remove those packets after the transfer for example. I agree that lost ACK is not a big problem. nncp-ack could have something like: nncp-ack -list-of-generated-acks acks.txt ... nncp-xfer / whatever nncp-rm -list-of-generated-acks acks.txt That file can be placed in memory, because of the small size. >I *THINK* we can do this with niceness. Probably, but I do not like that idea at all. Niceness is not a "type of the packet", however it is tempting to use it for that. Currently there are different predefined default niceness levels for each type of packet, but user is free to override it in anyway, for example to completely remove any difference between all packets at all (of course loosing ability to prioritize the packets). >Somewhat related - I have nncp-toss running as a daemon (-cycle) mode on >some nodes that I will want to enable ack generation for. I tend to treat "nncp-toss -cycle" like a mistake done previously. It complicates things very much, because of concurrent access, because every option is have to be in configuration file, instead of simpler-to-process command line arguments. But ok, -cycle exists, I won't remove it, but I think that it will be much better to use shell-script wrapper around it, probably taking more resources. Currently you have to stop nncp-toss-cycle when you work with nncp-xfer, to be able to run nncp-ack (because tossing will remove the packet) before tossing. Moving ACKing to tosser -- will be a huge complication. Currently I am in position that nncp-toss-cycle is not friendly with nncp-xfer/nncp-bundle. It is even harmful, because while you copy files from the external media, tosser will do tossing in background, requiring to do read operations (at least) while nncp-xfer/bundle does writing at the same time. So do not want to think about how all of that will work with nncp-toss-cycle :-) So currently I see two relatively simple changes I have to done: * nncp-ack must (partly) decrypt packets, to determine if it ACK and skip it * nncp-ack must generate list of generated ACK encrypted packets, to use it at least with nncp-rm, to be able to remove them after xfer transfer. * nncp-rm could take a list of PKTs from stdin for convenience there * probably nncp-xfer can take a list of packets (from the file/stdin) to skip. To be able to leave ACK packets in the spool just for a while nncp-ack won't generate ACK for ACK, fixing the biggest problem. -- Sergey Matveev (http://www.stargrave.org/) OpenPGP: CF60 E89A 5923 1E76 E263 6422 AE1A 8109 E498 57EF [-- Attachment #2: signature.asc --] [-- Type: application/pgp-signature, Size: 833 bytes --] ^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: [EN] NNCP 8.6.0 release announcement 2022-03-02 18:47 ` Sergey Matveev @ 2022-03-02 18:59 ` John Goerzen 2022-03-02 19:26 ` Sergey Matveev 0 siblings, 1 reply; 8+ messages in thread From: John Goerzen @ 2022-03-02 18:59 UTC (permalink / raw) To: Sergey Matveev; +Cc: nncp-devel On Wed, Mar 02 2022, Sergey Matveev wrote: > *** John Goerzen [2022-03-02 10:18]: >>In theory, we sort of have two options: >> >>1) Never keep ACK packets on the source machine, and never send ACKs for >>them on the destination. >> >>2) Handle them like any other packet. But this implies an endless >>stream of re-ACKing. > > I honestly forget about an endless reACKing problem. Somehow we must not > send ACK for an ACK obviously. Currently nncp-ack is a separate command > call, that just generates ACK for everything it saw in rx-directory. > nncp-toss is a candidate for that job instead, because it sees the plain > (unencrypted) packet's type. But that will lead to more configuration > file's options to control if ACKs need to be generated. However, > nncp-ack already has access to private keys (that is required to > generate outbound encrypted packets), so it also can decrypt packets. Hi Sergey, I just had another thought - maybe the ideal place to generate ACKs is in the nncp-xfer/bundle rx code itself? That would allow it to: - Generate an ACK for an incoming packet even if there is a .seen for it - Avoid the race with toss - Decrypt the first chunk and avoid generating ACKs for ACKs > Probably, but I do not like that idea at all. Niceness is not a "type of > the packet", however it is tempting to use it for that. Currently there Agreed, that is a workaround. [ regarding toss -cycle ] I find it very useful for machines that primarily interact via daemon/call/caller where there is not always a definite "done" state even. > So currently I see two relatively simple changes I have to done: > > * nncp-ack must (partly) decrypt packets, to determine if it ACK and skip it > * nncp-ack must generate list of generated ACK encrypted packets, to use > it at least with nncp-rm, to be able to remove them after xfer transfer. > * nncp-rm could take a list of PKTs from stdin for convenience there > * probably nncp-xfer can take a list of packets (from the file/stdin) to > skip. To be able to leave ACK packets in the spool just for a while > > nncp-ack won't generate ACK for ACK, fixing the biggest problem. So generating ACKs in nncp-xfer/bundle still leaves the problem of keep for them. Maybe the answer would be a simple option to xfer/bundle tx: -always-delete-sent-acks or something. Then an ACK that's transmitted would always be deleted, regardless of -keep. Combined with -seen for toss, that should provide good behavior in all cases. (If an ACK is lost, the sender will retransmit the packet, but since it's already been seen, the recipient will immediately generate a new ACK) - John ^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: [EN] NNCP 8.6.0 release announcement 2022-03-02 18:59 ` John Goerzen @ 2022-03-02 19:26 ` Sergey Matveev 2022-03-03 3:10 ` John Goerzen 0 siblings, 1 reply; 8+ messages in thread From: Sergey Matveev @ 2022-03-02 19:26 UTC (permalink / raw) To: nncp-devel [-- Attachment #1: Type: text/plain, Size: 3287 bytes --] *** John Goerzen [2022-03-02 12:59]: >I just had another thought - maybe the ideal place to generate ACKs is >in the nncp-xfer/bundle rx code itself? Actually initially I started exactly in them. But: * if ACKs are in use, xfer/bundle (let's call them xfers) must use the config with private keys. I do not use that use-case, but last time I checked, you can use xfers with nncp-cfgmin-imized configuration file version, that you safely can keep on unsafe storage/device. For example you take a laptop with minimized configuration, keep the spool on it, go outside, get connection with xfers in a place, where your laptop can be stolen/compromised. Anyway it is used only for just copying of files from one place (storages, tapes, whatever) to another. No private keys (even encrypted ones (nncp-cfgenc)) are used If we move all ACKing only to xfers, then there will be no way to use xfers only for transport and some other thing only for ACKing. Those are two completely different tasks with different requirements * I used bundle with the tape drive, that REALLY has to be able just to stream the data without any interruptions. That is why nncp-bundle does not check checksum on the fly (I used all of that previously on pretty low-performance CPU and BLAKE3 was not used in NNCP at that time). And that is also desireable for HDDs too, to be able to do as much sequential operation as it could (but it is not as critical as with tapes). If xfers will generate ACKs on the fly after each received packet, then it will interrupt sequential operations, it will "touch" our spool or xfer's mounted directory * we could just "remember" what ACKs we have to generate at the end of the process, but... if the process is suddenly interrupted? We loose that state and any knowledge of what was received and what we must ACK >- Avoid the race with toss Currently I am sure that it should not be run at all, because of obviously undefined behaviour. >I find it very useful for machines that primarily interact via >daemon/call/caller where there is not always a definite "done" state >even. I do not find the idea of running it sometimes in background, while those (long-live!) daemons are running, bad. I find that there sould be used daemontools'es supervise service with it, like some kind of trivial: $ cat services/nncp-toss/run sleep 60 exec nncp-toss ... which you will "down" when you insert your removable media to do xfering and acking: $ svc -d services/nncp-toss $ nncp-xfer ... $ nncp-ack ... $ svc -u services/nncp-toss >So generating ACKs in nncp-xfer/bundle still leaves the problem of keep >for them. Yes, that is independent problem. That I currently plan to solve with temporary intermediate file generated by xfers with a list of packets you need to ACK, that is used by nncp-ack/rm after. >Maybe the answer would be a simple option to xfer/bundle tx: >-always-delete-sent-acks or something. If xfers will be capable to generated packets, then yes. But I am against that, for the reasons at top of that letter. -- Sergey Matveev (http://www.stargrave.org/) OpenPGP: CF60 E89A 5923 1E76 E263 6422 AE1A 8109 E498 57EF [-- Attachment #2: signature.asc --] [-- Type: application/pgp-signature, Size: 833 bytes --] ^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: [EN] NNCP 8.6.0 release announcement 2022-03-02 19:26 ` Sergey Matveev @ 2022-03-03 3:10 ` John Goerzen 0 siblings, 0 replies; 8+ messages in thread From: John Goerzen @ 2022-03-03 3:10 UTC (permalink / raw) To: Sergey Matveev; +Cc: nncp-devel On Wed, Mar 02 2022, Sergey Matveev wrote: > * I used bundle with the tape drive, that REALLY has to be able just to I remember this about tape drives, yes :-) > * we could just "remember" what ACKs we have to generate at the end of > the process, but... if the process is suddenly interrupted? We loose > that state and any knowledge of what was received and what we must ACK Makes sense. > which you will "down" when you insert your removable media to do > xfering and acking: > > $ svc -d services/nncp-toss > $ nncp-xfer ... > $ nncp-ack ... > $ svc -u services/nncp-toss Of course, that could still work with -cycle as a service. service nncp-toss stop, service nncp-toss start, etc. > If xfers will be capable to generated packets, then yes. But I am > against that, for the reasons at top of that letter. Understood. Your state file idea would work for me too. Thanks! - John ^ permalink raw reply [flat|nested] 8+ messages in thread
end of thread, other threads:[~2022-03-03 3:11 UTC | newest] Thread overview: 8+ messages (download: mbox.gz / follow: Atom feed) -- links below jump to the message on this page -- 2022-03-02 14:57 [EN] NNCP 8.6.0 release announcement Sergey Matveev 2022-03-02 16:18 ` John Goerzen 2022-03-02 16:22 ` John Goerzen 2022-03-02 18:55 ` Sergey Matveev 2022-03-02 18:47 ` Sergey Matveev 2022-03-02 18:59 ` John Goerzen 2022-03-02 19:26 ` Sergey Matveev 2022-03-03 3:10 ` John Goerzen