public inbox for nncp-devel@lists.cypherpunks.ru
Atom feed

From: John Goerzen <jgoerzen@complete•org>
To: Sergey Matveev <stargrave@stargrave•org>
Cc: nncp-devel@lists.cypherpunks.ru
Subject: Re: Issues with very large packets
Date: Sat, 20 Feb 2021 22:31:12 -0600	[thread overview]
Message-ID: <87o8ge59f3.fsf@complete.org> (raw)
In-Reply-To: <YDFpX3GKBdJu0qV9@stargrave.org>


On Sat, Feb 20 2021, Sergey Matveev wrote:

> much more linearly on the disk, much more denser, because of 
> sequential
> writings of checkpoints on ZFS. And after that nncp-stat works a
> magnitude faster:
>         1.62 real         0.01 user         0.04 sys

Impressive!

> Also I have made support of ".nock" (non-checksummed) files. 
> After
> online daemons receives the file, it is renamed from ".part" to 
> ".nock".
> By default it is checksummed in the background and then renamed 
> to
> ordinary fully verified packet. But with -nock 
> command-line/configuration
> option you can disable its checksumming at all, doing that after
> daemon/caller invocations with nncp-check command.

Interesting.  How do these different options interact with 
removing the packet from the remote end?

I gather that, right now, the packet is preserved on the remote 
end until the recipient has checked and verified the checksum. 
With background checksumming, I assume that could happen later, 
possibly even after the call drops.  Will the sender still keep 
the packet around until the recipient confirms the checksum?  If 
not, that could result in data loss.  And if so, how exactly would 
that happen, especially since the call my drop by then?  (eg, 
onlinedeadline of 20s and it takes 200s to checksum a big file)  I 
worry there may be a race condition there where nncp-toss may see 
and receive a successfully-processed file before this can be 
indicated back to the sender.

> There is only one exception: if the file is received from the 
> very
> beginning (from zero offset), then it is hashed immediately in 
> the
> background during receiving. And if it is received till the end, 
> then no
> ".nock" is created and file is immediately renamed to ordinary 
> received
> packet (of course if checksum is good). If it is interrupted and 
> resumed
> later, then ".nock" is used anyway.

That's great!

> All of that is in the "develop" branch in git and everything 
> seems
> working. But currently I did not test it much, so there should 
> be some
> bugs. I will work a couple of days under that updated versions 
> and will
> make a release.

Thanks for all your work on this!

- John

next prev parent reply	other threads:[~2021-02-21  4:32 UTC|newest]

Thread overview: 8+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2021-02-18 21:35 Issues with very large packets John Goerzen
2021-02-19 12:36 ` Sergey Matveev
2021-02-19 19:18   ` John Goerzen
2021-02-19 19:46     ` Sergey Matveev
2021-02-19 20:34       ` John Goerzen
2021-02-20 19:56         ` Sergey Matveev
2021-02-21  4:31           ` John Goerzen [this message]
2021-02-21  8:27             ` Sergey Matveev