public inbox for nncp-devel@lists.cypherpunks.ru
Atom feed
From: Sergey Matveev <stargrave@stargrave•org>
To: nncp-devel@lists.cypherpunks.ru
Subject: Re: Issues with very large packets
Date: Fri, 19 Feb 2021 22:46:38 +0300 [thread overview]
Message-ID: <YDAVqtiqRuYU5DPW@stargrave.org> (raw)
In-Reply-To: <87czwvj27u.fsf@complete.org>
[-- Attachment #1: Type: text/plain, Size: 2482 bytes --]
*** John Goerzen [2021-02-19 13:18]:
>Random other question: do .seen files impact this at all?
There is only checking of their existence. Of course that can lead to
additional I/O (reading "inodes" of directory), but it should be cached
anyway. I am sure .seen impact is negligible.
>I'd discourage that path. xattrs are really add-ons in POSIXland. The
>support for them is often not present, sometimes broken, sometimes has weird
>compatibility issues. Probably not worth the hassle.
Believe you. And exactly that feeling I had when read about all that
xattrs stuff. Well, better forget about xattrs.
>Hey, if you want to <grin> But I don't think it's really worth doing this
>for my (or really most any) use case. If indeed you're reading just a few
>bytes from the start of the file, this wouldn't make much difference (or at
>least shouldn't) - and may even hurt, because now things like ls -l could
>have 2000 files to look at instead of 1000.
Probably I am wrong, but I really believe that especially on ZFS that
leads to huge read amplification. With default recordsize=128KiB it
plays no role to read 200B or 100KiB -- ZFS will anyway read the whole
record (it had to -- to check the integrity) (assume that compression
plays no role, because of encryption). But reading 200B file will lead
only reading of that 200B, that is even much smaller that disk sector
size. So thousand of files is many megabytes of random reads, that is
really heavy.
Looking for metainformation of 2000 files of course is an additional
I/O, but ZFS keeps directory information in pretty compact dictionaries,
those chunks are located in the recordsize blocks on the disk. And many
of them located nearby on the disk because of sequential ZFS transaction
write. So even thousands of additional files is not a comparable load at
all as read-amplificated reading of all that files'es first record. So
it is worth of it, however is an optimization for ZFS. But pretty simple
and optional.
>.meta could work but again this one little
>thing may not justify it. (Of course it should still be encrypted if it's in
>.meta or xattrs)
It is anyway plaintext already in the encrypted packet. .meta will (I
assume) keep just a copy of the header of a file. If someone can read
encrypted packets -- it can read corresponding .meta ones.
--
Sergey Matveev (http://www.stargrave.org/)
OpenPGP: CF60 E89A 5923 1E76 E263 6422 AE1A 8109 E498 57EF
[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]
next prev parent reply other threads:[~2021-02-19 19:46 UTC|newest]
Thread overview: 8+ messages / expand[flat|nested] mbox.gz Atom feed top
2021-02-18 21:35 Issues with very large packets John Goerzen
2021-02-19 12:36 ` Sergey Matveev
2021-02-19 19:18 ` John Goerzen
2021-02-19 19:46 ` Sergey Matveev [this message]
2021-02-19 20:34 ` John Goerzen
2021-02-20 19:56 ` Sergey Matveev
2021-02-21 4:31 ` John Goerzen
2021-02-21 8:27 ` Sergey Matveev