*** John Goerzen [2021-02-19 14:34]:
>But how much of that is caused by head seeking vs. reading an
>extra 128ish K?  I mean, I would expect the cost of reading 128K vs. reading
>200 bytes to be tiny compared to the latency of the seeks to get to the file
>in the first place.

That is also true.

Well, I made that ".hdr" files support, which keep copy of the header. I
tried it on the following spool directory: Tx:    303 GiB,  839 pkts.
Non-modified nncp-stat takes:
       39.61 real         0.07 user         0.10 sys
I made it on ZFS that was exported/imported, so completely lacking any
cached information. .hdr supporting version took:
       16.97 real         0.05 user         0.08 sys
So it helps. Also there is some kind of hack: you can remove all .hdr
and recreate them simply by running nncp-stat. Because there are no
other heavy writing on the filesystem, those .hdr files will be located
much more linearly on the disk, much more denser, because of sequential
writings of checkpoints on ZFS. And after that nncp-stat works a
magnitude faster:
        1.62 real         0.01 user         0.04 sys

So random I/O operations is the bottleneck here. Sequential ZFS write of
bunch of buffered operations helps much (as in last case), but it is
hacky of course. .hdr files usage can be disabled in the configuration at all.

Also I have made support of ".nock" (non-checksummed) files. After
online daemons receives the file, it is renamed from ".part" to ".nock".
By default it is checksummed in the background and then renamed to
ordinary fully verified packet. But with -nock command-line/configuration
option you can disable its checksumming at all, doing that after
daemon/caller invocations with nncp-check command.

There is only one exception: if the file is received from the very
beginning (from zero offset), then it is hashed immediately in the
background during receiving. And if it is received till the end, then no
".nock" is created and file is immediately renamed to ordinary received
packet (of course if checksum is good). If it is interrupted and resumed
later, then ".nock" is used anyway.

All of that is in the "develop" branch in git and everything seems
working. But currently I did not test it much, so there should be some
bugs. I will work a couple of days under that updated versions and will
make a release.

-- 
Sergey Matveev (http://www.stargrave.org/)
OpenPGP: CF60 E89A 5923 1E76 E263  6422 AE1A 8109 E498 57EF