public inbox for nncp-devel@lists.cypherpunks.ru
Atom feed
* Issues with very large packets
@ 2021-02-18 21:35 John Goerzen
2021-02-19 12:36 ` Sergey Matveev
0 siblings, 1 reply; 8+ messages in thread
From: John Goerzen @ 2021-02-18 21:35 UTC (permalink / raw)
To: nncp-devel
Hi folks!
So this week I had the occasion to send some 250GB packets through
NNCP. This generally worked in the end, but had two interesting
foibles along the way.
First, nncp-stat was extremely slow on a system that had packets
like that queued for transmission. I'm wondering if it is trying
to read all the packets, even when called with no parameters?
The other is a new twist on the timeout problem I mentioned
previously. I previously encountered it post-receive when the
receiving end would do a time-consuming check on a largeish (say,
10GB) packet.
With these very large packets, I also encountered it at the
beginning of a call. nncp-daemon on the transmitting side
apparently was taking too long at call establishment, and
nncp-call would timeout (with the same error I'd mentioned
previously) without receiving anything. Interestingly, if
nncp-call would try again immediately, it would work. By cycling
through this timeout/working/timeout/working pattern, eventually
all the data was transmitted.
It makes me wonder if nncp-daemon was doing some sort of expensive
scan at the beginning of the call, and either it or the OS cached
the results? Not quite sure there.
Anyhow, thanks again!
- John
^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: Issues with very large packets
2021-02-18 21:35 Issues with very large packets John Goerzen
@ 2021-02-19 12:36 ` Sergey Matveev
2021-02-19 19:18 ` John Goerzen
0 siblings, 1 reply; 8+ messages in thread
From: Sergey Matveev @ 2021-02-19 12:36 UTC (permalink / raw)
To: nncp-devel
[-- Attachment #1: Type: text/plain, Size: 3401 bytes --]
Greetings!
*** John Goerzen [2021-02-18 15:35]:
>First, nncp-stat was extremely slow on a system that had packets like that
>queued for transmission. I'm wondering if it is trying to read all the
>packets, even when called with no parameters?
>It makes me wonder if nncp-daemon was doing some sort of expensive scan at
>the beginning of the call, and either it or the OS cached the results?
Both of them, as all other commands uses (very simple) src/jobs.go code
to get the list of available encrypted packets in the spool. It does
I/O, but I would not call it expensive:
* list files in the directory (node's spool)
* open each file and read its XDR-encoded header
* this header (currently) takes only 172 bytes of data
* seek each file to the beginning
* return all that jobs metainformation with opened file descriptors
So the only I/O is directory and 172 bytes of each files reading. So if
you have many of files, then yes, it would take some time. But it will
read the whole block/record (up to 128KiB by default) on ZFS. And yes,
if you repeat the operation, then ZFS ARC cache should contain those
blocks, that is why it is much more faster.
First of all: I have no ideas how file's size can affect anyhow that
algorithm above. Possibly read-ahead configuration may read more than
single ZFS block, but it is question about just several ones in the
worst case. So I see no difference in 1TiB or 1GiB file's
metainformation getting. I am talking much about ZFS there only because
reading of that little piece of information will be more expensive on
it, comparing to other filesystems (and I think it is ok, because in
most real world use-cases you wish to read the whole file after that).
If we will keep that metainformation nearby, then it should help much:
* Keeping separate database/cache-file of course is not an option,
because of complexity and raised consistency questions.
* In nearly all code I see in NNCP, only the niceness level is the only
thing used everywhere. And a long time ago I actually thought about
keeping it in the filename itself. But I did not like that clumsy
optimization and still do not like, because niceness is some kind of
private valuable metainformation itself (if someone send a packet with
non-common niceness of 145 -- the fact that some packet with it
appeared somewhere may be valuable). That is not nice :-)
* We can keep copy of that metainformation (that 172-byte) header in
".meta"/whatever file nearby. It it does not exist -- then read the
beginning of file as used to. It can be recreated anytime atomically.
I like that solution
* That kind of information can also be kept in filesystem's extended
attributes. Honestly I have never ever worked with xattrs at all.
Today was my first setextattr/getextattr command invocations. But
seems that it should be even more optimal and faster access
information than separate file on any filesystem. It xattrs are
missing (disabled?), then fallback to ordinary file reading
* But as I can see, OpenBSD does not support xattrs at all. So fallback
to separate ".meta" could be valuable anyway
So seems I am going to write that header keeping in xattrs with fallback
to separate file storage.
--
Sergey Matveev (http://www.stargrave.org/)
OpenPGP: CF60 E89A 5923 1E76 E263 6422 AE1A 8109 E498 57EF
[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]
^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: Issues with very large packets
2021-02-19 12:36 ` Sergey Matveev
@ 2021-02-19 19:18 ` John Goerzen
2021-02-19 19:46 ` Sergey Matveev
0 siblings, 1 reply; 8+ messages in thread
From: John Goerzen @ 2021-02-19 19:18 UTC (permalink / raw)
To: Sergey Matveev; +Cc: nncp-devel
On Fri, Feb 19 2021, Sergey Matveev wrote:
> Greetings!
>
> *** John Goerzen [2021-02-18 15:35]:
>>First, nncp-stat was extremely slow on a system that had packets
>>like that
>>queued for transmission. I'm wondering if it is trying to read
>>all the
>>packets, even when called with no parameters?
>>It makes me wonder if nncp-daemon was doing some sort of
>>expensive scan at
>>the beginning of the call, and either it or the OS cached the
>>results?
>
> Both of them, as all other commands uses (very simple)
> src/jobs.go code
> to get the list of available encrypted packets in the spool. It
> does
> I/O, but I would not call it expensive:
Indeed that doesn't seem like it should be very slow. I will see
if I can do some experimentation with strace next week (when I'll
next have packets of this size) and see if I can validate it.
I do also sometimes have about a thousand packets in the rx
directory. That slows it down too, but to the point of taking a
few dozen seconds, not to the point of taking a minute+ as this
is. Though of course to read an entire file of this size would
take a lot more than a minute, so it wasn't that. I am sure I saw
this when there were way less than 1000 packets in the directory,
and also reasonably sure that I saw it when there were 3.
Random other question: do .seen files impact this at all?
> metainformation getting. I am talking much about ZFS there only
> because
> reading of that little piece of information will be more
> expensive on
> it, comparing to other filesystems (and I think it is ok,
> because in
> most real world use-cases you wish to read the whole file after
> that).
It is indeed ZFS here.
> If we will keep that metainformation nearby, then it should help
> much:
>
> * Keeping separate database/cache-file of course is not an
> option,
> because of complexity and raised consistency questions.
Agreed. This use case is not worth a lot of optimizations, it
makes sense not to have it in the filename. .meta could work but
again this one little thing may not justify it. (Of course it
should still be encrypted if it's in .meta or xattrs)
> * That kind of information can also be kept in filesystem's
> extended
> attributes. Honestly I have never ever worked with xattrs at
> all.
> Today was my first setextattr/getextattr command invocations.
> But
> seems that it should be even more optimal and faster access
> information than separate file on any filesystem. It xattrs
> are
> missing (disabled?), then fallback to ordinary file reading
> * But as I can see, OpenBSD does not support xattrs at all. So
> fallback
> to separate ".meta" could be valuable anyway
I'd discourage that path. xattrs are really add-ons in POSIXland.
The support for them is often not present, sometimes broken,
sometimes has weird compatibility issues. Probably not worth the
hassle.
> So seems I am going to write that header keeping in xattrs with
> fallback
> to separate file storage.
Hey, if you want to <grin> But I don't think it's really worth
doing this for my (or really most any) use case. If indeed you're
reading just a few bytes from the start of the file, this wouldn't
make much difference (or at least shouldn't) - and may even hurt,
because now things like ls -l could have 2000 files to look at
instead of 1000.
Thanks again!
- John
^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: Issues with very large packets
2021-02-19 19:18 ` John Goerzen
@ 2021-02-19 19:46 ` Sergey Matveev
2021-02-19 20:34 ` John Goerzen
0 siblings, 1 reply; 8+ messages in thread
From: Sergey Matveev @ 2021-02-19 19:46 UTC (permalink / raw)
To: nncp-devel
[-- Attachment #1: Type: text/plain, Size: 2482 bytes --]
*** John Goerzen [2021-02-19 13:18]:
>Random other question: do .seen files impact this at all?
There is only checking of their existence. Of course that can lead to
additional I/O (reading "inodes" of directory), but it should be cached
anyway. I am sure .seen impact is negligible.
>I'd discourage that path. xattrs are really add-ons in POSIXland. The
>support for them is often not present, sometimes broken, sometimes has weird
>compatibility issues. Probably not worth the hassle.
Believe you. And exactly that feeling I had when read about all that
xattrs stuff. Well, better forget about xattrs.
>Hey, if you want to <grin> But I don't think it's really worth doing this
>for my (or really most any) use case. If indeed you're reading just a few
>bytes from the start of the file, this wouldn't make much difference (or at
>least shouldn't) - and may even hurt, because now things like ls -l could
>have 2000 files to look at instead of 1000.
Probably I am wrong, but I really believe that especially on ZFS that
leads to huge read amplification. With default recordsize=128KiB it
plays no role to read 200B or 100KiB -- ZFS will anyway read the whole
record (it had to -- to check the integrity) (assume that compression
plays no role, because of encryption). But reading 200B file will lead
only reading of that 200B, that is even much smaller that disk sector
size. So thousand of files is many megabytes of random reads, that is
really heavy.
Looking for metainformation of 2000 files of course is an additional
I/O, but ZFS keeps directory information in pretty compact dictionaries,
those chunks are located in the recordsize blocks on the disk. And many
of them located nearby on the disk because of sequential ZFS transaction
write. So even thousands of additional files is not a comparable load at
all as read-amplificated reading of all that files'es first record. So
it is worth of it, however is an optimization for ZFS. But pretty simple
and optional.
>.meta could work but again this one little
>thing may not justify it. (Of course it should still be encrypted if it's in
>.meta or xattrs)
It is anyway plaintext already in the encrypted packet. .meta will (I
assume) keep just a copy of the header of a file. If someone can read
encrypted packets -- it can read corresponding .meta ones.
--
Sergey Matveev (http://www.stargrave.org/)
OpenPGP: CF60 E89A 5923 1E76 E263 6422 AE1A 8109 E498 57EF
[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]
^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: Issues with very large packets
2021-02-19 19:46 ` Sergey Matveev
@ 2021-02-19 20:34 ` John Goerzen
2021-02-20 19:56 ` Sergey Matveev
0 siblings, 1 reply; 8+ messages in thread
From: John Goerzen @ 2021-02-19 20:34 UTC (permalink / raw)
To: Sergey Matveev; +Cc: nncp-devel
On Fri, Feb 19 2021, Sergey Matveev wrote:
> Probably I am wrong, but I really believe that especially on ZFS
> that
> leads to huge read amplification. With default recordsize=128KiB
> it
> plays no role to read 200B or 100KiB -- ZFS will anyway read the
> whole
> record (it had to -- to check the integrity) (assume that
> compression
> plays no role, because of encryption). But reading 200B file
> will lead
> only reading of that 200B, that is even much smaller that disk
> sector
> size. So thousand of files is many megabytes of random reads,
> that is
> really heavy.
I don't think you're wrong, but in my experience it just hasn't
been a huge issue. Yes nncp-stat can take a dozen seconds when
there are a thousand packets there. But how much of that is
caused by head seeking vs. reading an extra 128ish K? I mean, I
would expect the cost of reading 128K vs. reading 200 bytes to be
tiny compared to the latency of the seeks to get to the file in
the first place. This is all on HDD, of course; with SSD, I would
imagine the 128K sequential read also to be fairly
inconsequential. But I guess the one way to find out is to test
it!
-- John
^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: Issues with very large packets
2021-02-19 20:34 ` John Goerzen
@ 2021-02-20 19:56 ` Sergey Matveev
2021-02-21 4:31 ` John Goerzen
0 siblings, 1 reply; 8+ messages in thread
From: Sergey Matveev @ 2021-02-20 19:56 UTC (permalink / raw)
To: nncp-devel
[-- Attachment #1: Type: text/plain, Size: 2509 bytes --]
*** John Goerzen [2021-02-19 14:34]:
>But how much of that is caused by head seeking vs. reading an
>extra 128ish K? I mean, I would expect the cost of reading 128K vs. reading
>200 bytes to be tiny compared to the latency of the seeks to get to the file
>in the first place.
That is also true.
Well, I made that ".hdr" files support, which keep copy of the header. I
tried it on the following spool directory: Tx: 303 GiB, 839 pkts.
Non-modified nncp-stat takes:
39.61 real 0.07 user 0.10 sys
I made it on ZFS that was exported/imported, so completely lacking any
cached information. .hdr supporting version took:
16.97 real 0.05 user 0.08 sys
So it helps. Also there is some kind of hack: you can remove all .hdr
and recreate them simply by running nncp-stat. Because there are no
other heavy writing on the filesystem, those .hdr files will be located
much more linearly on the disk, much more denser, because of sequential
writings of checkpoints on ZFS. And after that nncp-stat works a
magnitude faster:
1.62 real 0.01 user 0.04 sys
So random I/O operations is the bottleneck here. Sequential ZFS write of
bunch of buffered operations helps much (as in last case), but it is
hacky of course. .hdr files usage can be disabled in the configuration at all.
Also I have made support of ".nock" (non-checksummed) files. After
online daemons receives the file, it is renamed from ".part" to ".nock".
By default it is checksummed in the background and then renamed to
ordinary fully verified packet. But with -nock command-line/configuration
option you can disable its checksumming at all, doing that after
daemon/caller invocations with nncp-check command.
There is only one exception: if the file is received from the very
beginning (from zero offset), then it is hashed immediately in the
background during receiving. And if it is received till the end, then no
".nock" is created and file is immediately renamed to ordinary received
packet (of course if checksum is good). If it is interrupted and resumed
later, then ".nock" is used anyway.
All of that is in the "develop" branch in git and everything seems
working. But currently I did not test it much, so there should be some
bugs. I will work a couple of days under that updated versions and will
make a release.
--
Sergey Matveev (http://www.stargrave.org/)
OpenPGP: CF60 E89A 5923 1E76 E263 6422 AE1A 8109 E498 57EF
[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]
^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: Issues with very large packets
2021-02-20 19:56 ` Sergey Matveev
@ 2021-02-21 4:31 ` John Goerzen
2021-02-21 8:27 ` Sergey Matveev
0 siblings, 1 reply; 8+ messages in thread
From: John Goerzen @ 2021-02-21 4:31 UTC (permalink / raw)
To: Sergey Matveev; +Cc: nncp-devel
On Sat, Feb 20 2021, Sergey Matveev wrote:
> much more linearly on the disk, much more denser, because of
> sequential
> writings of checkpoints on ZFS. And after that nncp-stat works a
> magnitude faster:
> 1.62 real 0.01 user 0.04 sys
Impressive!
> Also I have made support of ".nock" (non-checksummed) files.
> After
> online daemons receives the file, it is renamed from ".part" to
> ".nock".
> By default it is checksummed in the background and then renamed
> to
> ordinary fully verified packet. But with -nock
> command-line/configuration
> option you can disable its checksumming at all, doing that after
> daemon/caller invocations with nncp-check command.
Interesting. How do these different options interact with
removing the packet from the remote end?
I gather that, right now, the packet is preserved on the remote
end until the recipient has checked and verified the checksum.
With background checksumming, I assume that could happen later,
possibly even after the call drops. Will the sender still keep
the packet around until the recipient confirms the checksum? If
not, that could result in data loss. And if so, how exactly would
that happen, especially since the call my drop by then? (eg,
onlinedeadline of 20s and it takes 200s to checksum a big file) I
worry there may be a race condition there where nncp-toss may see
and receive a successfully-processed file before this can be
indicated back to the sender.
> There is only one exception: if the file is received from the
> very
> beginning (from zero offset), then it is hashed immediately in
> the
> background during receiving. And if it is received till the end,
> then no
> ".nock" is created and file is immediately renamed to ordinary
> received
> packet (of course if checksum is good). If it is interrupted and
> resumed
> later, then ".nock" is used anyway.
That's great!
> All of that is in the "develop" branch in git and everything
> seems
> working. But currently I did not test it much, so there should
> be some
> bugs. I will work a couple of days under that updated versions
> and will
> make a release.
Thanks for all your work on this!
- John
^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: Issues with very large packets
2021-02-21 4:31 ` John Goerzen
@ 2021-02-21 8:27 ` Sergey Matveev
0 siblings, 0 replies; 8+ messages in thread
From: Sergey Matveev @ 2021-02-21 8:27 UTC (permalink / raw)
To: nncp-devel
[-- Attachment #1: Type: text/plain, Size: 1627 bytes --]
*** John Goerzen [2021-02-20 22:31]:
>Interesting. How do these different options interact with removing the
>packet from the remote end?
Packet is deleted only and only when confirmation is sent. And
confirmation is sent only if either packet with the same hash exists in
the spool directory (completely checksummed and fully downloaded), or
corresponding .seen packet exists (that can appear only after tossing of
completely checksummed and fully downloaded packet).
By default, after packet is received, it is renamed .part->.nock and it
is added to background checker queue. When checker finishes
checksumming, it renames .nock->"no extension" and sends confirmation to
the remote side. If remote side is unavailable (connection is lost),
then that confirmation is lost. But ordinary packet (or .seen after its
tossing) will be in the spool. Each online session starts with sending
the whole list of packets (taking niceness in advance) to the remote
side. And during the next session, because of packet/.seen, confirmation
will be sent and packet deleted on remote side. Also, when session
begins, background checker is started with pre-filled queue of currently
existing .nock files (by default, if no -nock option is turned on), that
will send confirmation when .nock-s are checked.
So no loss is possible (.nock files are not enough for confirmation).
Lost connections are not a problem, if you keep state of seen packets --
confirmation will be sent during the next sessions.
--
Sergey Matveev (http://www.stargrave.org/)
OpenPGP: CF60 E89A 5923 1E76 E263 6422 AE1A 8109 E498 57EF
[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]
^ permalink raw reply [flat|nested] 8+ messages in thread
end of thread, other threads:[~2021-02-21 8:27 UTC | newest]
Thread overview: 8+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2021-02-18 21:35 Issues with very large packets John Goerzen
2021-02-19 12:36 ` Sergey Matveev
2021-02-19 19:18 ` John Goerzen
2021-02-19 19:46 ` Sergey Matveev
2021-02-19 20:34 ` John Goerzen
2021-02-20 19:56 ` Sergey Matveev
2021-02-21 4:31 ` John Goerzen
2021-02-21 8:27 ` Sergey Matveev