Suggestions to make ACK easier

public inbox for nncp-devel@lists.cypherpunks.ru
Atom feed

* Suggestions to make ACK easier
@ 2023-07-11 14:28 John Goerzen
  2023-07-11 15:55 ` John Goerzen
                   ` (2 more replies)
  0 siblings, 3 replies; 7+ messages in thread
From: John Goerzen @ 2023-07-11 14:28 UTC (permalink / raw)
  To: nncp-devel

Hi,

A question about ACKs came up on the Matrix channel today, which
prompted me to revisit my previous article about it, at
https://www.complete.org/dead-usb-drives-are-fine-building-a-reliable-sneakernet/

There are two sort of pain points:

1) There is a race condition in that nncp-ack must run before nncp-toss
for incoming packets.

2) We have to save off the list of generated ACKs so that we can remove
them from the queue later.

I think the situation could be simplified with two tweaks:

First, nncp-xfer (and all the others, nncp-call, caller, daemon, bundle,
etc) could have a -ack mode that will cause it to automatically generate
an ack after successful receipt of a packet, eliminating the nncp-toss
race condition.  (Or maybe it would make more sense to put this in toss?
But there I think it would be more easy to generate multiple ACKs)

Then, nncp-xfer -tx could have a -keepexceptacks or something, that would
cause it to keep all packets it sends, EXCEPT for the acks.  That
eliminates the need to save off the list of generated ACKs from an
earlier step.  It may be that the status of a packet as an ACK isn't
available at this point, in which case a separate flag file or something
may be useful.

Thanks,

John

^ permalink raw reply	[flat|nested] 7+ messages in thread

* Re: Suggestions to make ACK easier
  2023-07-11 14:28 Suggestions to make ACK easier John Goerzen
@ 2023-07-11 15:55 ` John Goerzen
  2023-07-30  7:47 ` Sergey Matveev
  2023-08-09 12:04 ` Sergey Matveev
  2 siblings, 0 replies; 7+ messages in thread
From: John Goerzen @ 2023-07-11 15:55 UTC (permalink / raw)
  To: nncp-devel

I guess one other thought - doing it at toss time, while it may have a
risk of acking more than once, could also be nice as it would make it
convenient to do end-to-end ACKs even on cases where things traverse
several network hops.

- John

On Tue, Jul 11 2023, John Goerzen wrote:

> Hi,
>
> A question about ACKs came up on the Matrix channel today, which
> prompted me to revisit my previous article about it, at
> https://www.complete.org/dead-usb-drives-are-fine-building-a-reliable-sneakernet/
>
> There are two sort of pain points:
>
> 1) There is a race condition in that nncp-ack must run before nncp-toss
> for incoming packets.
>
> 2) We have to save off the list of generated ACKs so that we can remove
> them from the queue later.
>
> I think the situation could be simplified with two tweaks:
>
> First, nncp-xfer (and all the others, nncp-call, caller, daemon, bundle,
> etc) could have a -ack mode that will cause it to automatically generate
> an ack after successful receipt of a packet, eliminating the nncp-toss
> race condition.  (Or maybe it would make more sense to put this in toss?
> But there I think it would be more easy to generate multiple ACKs)
>
> Then, nncp-xfer -tx could have a -keepexceptacks or something, that would
> cause it to keep all packets it sends, EXCEPT for the acks.  That
> eliminates the need to save off the list of generated ACKs from an
> earlier step.  It may be that the status of a packet as an ACK isn't
> available at this point, in which case a separate flag file or something
> may be useful.
>
> Thanks,
>
> John

^ permalink raw reply	[flat|nested] 7+ messages in thread

* Re: Suggestions to make ACK easier
  2023-07-11 14:28 Suggestions to make ACK easier John Goerzen
  2023-07-11 15:55 ` John Goerzen
@ 2023-07-30  7:47 ` Sergey Matveev
  2023-08-09 12:04 ` Sergey Matveev
  2 siblings, 0 replies; 7+ messages in thread
From: Sergey Matveev @ 2023-07-30  7:47 UTC (permalink / raw)
  To: nncp-devel

[-- Attachment #1: Type: text/plain, Size: 428 bytes --]

Greetings!

*** John Goerzen [2023-07-11 09:28]: [...]

Very sorry for late answer, but I still do not have enough time even to
think about the suggestions. My vacation on work was lost and currently
do not know exactly when I will have enough free time. Hoping the next
month I will ne on the "rail" again!

-- 
Sergey Matveev (http://www.stargrave.org/)
OpenPGP: 12AD 3268 9C66 0D42 6967  FD75 CB82 0563 2107 AD8A

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 228 bytes --]

^ permalink raw reply	[flat|nested] 7+ messages in thread

* Re: Suggestions to make ACK easier
  2023-07-11 14:28 Suggestions to make ACK easier John Goerzen
  2023-07-11 15:55 ` John Goerzen
  2023-07-30  7:47 ` Sergey Matveev
@ 2023-08-09 12:04 ` Sergey Matveev
  2023-08-10  2:04   ` John Goerzen
  2 siblings, 1 reply; 7+ messages in thread
From: Sergey Matveev @ 2023-08-09 12:04 UTC (permalink / raw)
  To: nncp-devel

[-- Attachment #1: Type: text/plain, Size: 1608 bytes --]

Greetings!

*** John Goerzen [2023-07-11 09:28]:
>First, nncp-xfer (and all the others, nncp-call, caller, daemon, bundle,
>etc) could have a -ack mode that will cause it to automatically generate
>an ack after successful receipt of a packet, eliminating the nncp-toss
>race condition.  (Or maybe it would make more sense to put this in toss?
>But there I think it would be more easy to generate multiple ACKs)

I am afraid I dislike all those ideas unfortunately, basically because
of non-atomic operations.

If "nncp-xfer -rx" will generate ACK packet, then what if something will
go wrong and outbound ACK packet could not be saved? We loose that ACK
and our running nncp-toss can already toss retrieved packet. -xfer +
-ack + -toss guarantees that if -ack will fail, then no packet will
disappear because of tossing. We can rerun -ack and be sure that ACKs
are generated (or will catch some error again (missing free disk space
for example)), and only after that we will toss them. So that won't
eliminate the race between reception and tossing.

>Then, nncp-xfer -tx could have a -keepexceptacks or something, that would
>cause it to keep all packets it sends, EXCEPT for the acks.

Unfortunately it can not, because -xfer sees only already encrypted
packets, which lacks packet type identification. It does not know what
it transfers. Or we have to keep some state, list of encrypted ACK
packets that can be safely removed, that leads to current workflow again.

-- 
Sergey Matveev (http://www.stargrave.org/)
OpenPGP: 12AD 3268 9C66 0D42 6967  FD75 CB82 0563 2107 AD8A

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 228 bytes --]

^ permalink raw reply	[flat|nested] 7+ messages in thread

* Re: Suggestions to make ACK easier
  2023-08-09 12:04 ` Sergey Matveev
@ 2023-08-10  2:04   ` John Goerzen
  2023-08-13 20:30     ` Sergey Matveev
  0 siblings, 1 reply; 7+ messages in thread
From: John Goerzen @ 2023-08-10  2:04 UTC (permalink / raw)
  To: Sergey Matveev; +Cc: nncp-devel


On Wed, Aug 09 2023, Sergey Matveev wrote:

> [[PGP Signed Part:Undecided]]
> Greetings!
>
> *** John Goerzen [2023-07-11 09:28]:
>>First, nncp-xfer (and all the others, nncp-call, caller, daemon, bundle,
>>etc) could have a -ack mode that will cause it to automatically generate
>>an ack after successful receipt of a packet, eliminating the nncp-toss
>>race condition.  (Or maybe it would make more sense to put this in toss?
>>But there I think it would be more easy to generate multiple ACKs)
>
> I am afraid I dislike all those ideas unfortunately, basically because
> of non-atomic operations.
>
> If "nncp-xfer -rx" will generate ACK packet, then what if something will
> go wrong and outbound ACK packet could not be saved? We loose that ACK
> and our running nncp-toss can already toss retrieved packet. -xfer +

That makes sense.  Though if -seen is in use, I guess it will just cause
a harmless retransmission.

> -ack + -toss guarantees that if -ack will fail, then no packet will
> disappear because of tossing. We can rerun -ack and be sure that ACKs

But there is the problem that toss may run before -ack.  Basically if
anything at all on the system is going to use -ack, then you can't use
-autotoss, or nncp-toss with -cycle, because that then means a race
condition -- packets may be tossed before nncp-ack is run.  You
basically have to ensure that nothing at all might run toss when you're
loading stuff up for -ack, and that means some outside-of-NNCP locking
in some cases.

Maybe the thing to do here is extend nncp.hjson to add an "autoack"
feature to a given node?  That way, nncp-toss could generate an ack for
it at toss time.  This may imply needing a seen-like method to prevent
sending multiple ACKs for a packet that fails to toss the first time.

One other thing I've been pondering: end-to-end ACKs.  So if you send
from node A to node D via B and C, what happens if B receives the packet
and then its disk dies?  This is similar to the USB drive case.  I
suppose we could have each piece along the chain generate ACKs to the
previous one, or could we have D generate an ack back to A when the
packet arrives at D?  That may be too complicated, but just a thought
I've been having.

> are generated (or will catch some error again (missing free disk space
> for example)), and only after that we will toss them. So that won't
> eliminate the race between reception and tossing.
>
>>Then, nncp-xfer -tx could have a -keepexceptacks or something, that would
>>cause it to keep all packets it sends, EXCEPT for the acks.
>
> Unfortunately it can not, because -xfer sees only already encrypted
> packets, which lacks packet type identification. It does not know what
> it transfers. Or we have to keep some state, list of encrypted ACK
> packets that can be safely removed, that leads to current workflow again.

Ahh, OK, that makes sense.

Thanks,

- John

^ permalink raw reply	[flat|nested] 7+ messages in thread

* Re: Suggestions to make ACK easier
  2023-08-10  2:04   ` John Goerzen
@ 2023-08-13 20:30     ` Sergey Matveev
  2023-09-13 17:29       ` John Goerzen
  0 siblings, 1 reply; 7+ messages in thread
From: Sergey Matveev @ 2023-08-13 20:30 UTC (permalink / raw)
  To: nncp-devel

[-- Attachment #1: Type: text/plain, Size: 2547 bytes --]

Greetings!

*** John Goerzen [2023-08-09 21:04]:
>> If "nncp-xfer -rx" will generate ACK packet, then what if something will
>> go wrong and outbound ACK packet could not be saved? We loose that ACK
>> and our running nncp-toss can already toss retrieved packet. -xfer +
>
>That makes sense.  Though if -seen is in use, I guess it will just cause
>a harmless retransmission.

Problem here is that ACK won't be generated at all, because packet is
already tossed. Duplicate ACKs is not a problem indeed, but completely
lost (non-generated) ones are bad of course.

>But there is the problem that toss may run before -ack.  Basically if
>anything at all on the system is going to use -ack, then you can't use
>-autotoss, or nncp-toss with -cycle, because that then means a race
>condition -- packets may be tossed before nncp-ack is run.  You
>basically have to ensure that nothing at all might run toss when you're
>loading stuff up for -ack, and that means some outside-of-NNCP locking
>in some cases.

Everything is right. ACK-based workflow was not compatible with autotossing.

>Maybe the thing to do here is extend nncp.hjson to add an "autoack"
>feature to a given node?  That way, nncp-toss could generate an ack for
>it at toss time.

As with -xfer/-bundle commands, you have to store the knowledge of
packets who carry ACKs. It was made by printing their identifiers to
stdout by nncp-ack command itself. With tossers (that are not only
-toss, but -daemon and -caller) you will have to create some kind of
atomically commitable database of those ACKs. It can be a directory with
just empty files, whose names will be ACK-packets identifiers.

I made ability to automatically generate ACKs during tossing in 8.9.0
release. nncp-toss -gen-ack, nncp-* -autotoss-gen-ack. Although I did
only quick testing if it works fine. It saves knowledge about ACK
packets in spool/tx/ack/PKT empty files. nncp-rm -ack will remove them
and corresponding spool/tx/PKT packets.

>One other thing I've been pondering: end-to-end ACKs.  So if you send
>from node A to node D via B and C, what happens if B receives the packet
>and then its disk dies?

Something badly unexpected will happen :-)
Well, end-to-end acknowledgments is more complicated, because
recipient's node does not know identity of the very first packet that
was sent. Something much more complicated must be done to deal with it.

-- 
Sergey Matveev (http://www.stargrave.org/)
OpenPGP: 12AD 3268 9C66 0D42 6967  FD75 CB82 0563 2107 AD8A

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 228 bytes --]

^ permalink raw reply	[flat|nested] 7+ messages in thread

* Re: Suggestions to make ACK easier
  2023-08-13 20:30     ` Sergey Matveev
@ 2023-09-13 17:29       ` John Goerzen
  0 siblings, 0 replies; 7+ messages in thread
From: John Goerzen @ 2023-09-13 17:29 UTC (permalink / raw)
  To: Sergey Matveev; +Cc: nncp-devel

On Sun, Aug 13 2023, Sergey Matveev wrote:

> I made ability to automatically generate ACKs during tossing in 8.9.0
> release. nncp-toss -gen-ack, nncp-* -autotoss-gen-ack. Although I did
> only quick testing if it works fine. It saves knowledge about ACK
> packets in spool/tx/ack/PKT empty files. nncp-rm -ack will remove them
> and corresponding spool/tx/PKT packets.

Hi Sergey,

Thank you for this!

It's been a busy month, but I've finally had the time to update my
documentation, scripts, and such, and test this out.

I think I've found two issues.

The first, if -gen-ack is used in conjunction with -seen, then -seen
fires first and the ACK isn't generated.  This is relevant if the ACK
itself was lost at some point; we'd want to keep generating them as long
as the relevant packet keeps coming in.  Ideally the ACK would be
processed before applying the -seen filter.

The second, I'm a little less sure of, but it seems that when responding
to packets that arrived using a via designation, the ACK is sent to the
wrong node -- not the immediate peer.

I had a situation where a packet flowed like this:

A -> Z -> V

where the path between Z and V is using a USB stick and runs ACK.  The
ACK generated on V might have been sent to A via Z, rather than to Z
directly.

I'm a little unsure of my diagnosis here; I did see A process an ACK,
but it is also theoretically possible that a nncp-toss -gen-ack running
on Z generated that ACK.

In any case, somehow there were a few packets that don't have ACKs and
are sticking around (hence the first issue).

Thanks!

- John

^ permalink raw reply	[flat|nested] 7+ messages in thread

end of thread, other threads:[~2023-09-13 17:33 UTC | newest]

Thread overview: 7+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2023-07-11 14:28 Suggestions to make ACK easier John Goerzen
2023-07-11 15:55 ` John Goerzen
2023-07-30  7:47 ` Sergey Matveev
2023-08-09 12:04 ` Sergey Matveev
2023-08-10  2:04   ` John Goerzen
2023-08-13 20:30     ` Sergey Matveev
2023-09-13 17:29       ` John Goerzen