public inbox for nncp-devel@lists.cypherpunks.ru
Atom feed
* Announcing Filespooler
@ 2022-05-30 0:23 John Goerzen
2022-05-31 17:13 ` John Goerzen
` (2 more replies)
0 siblings, 3 replies; 11+ messages in thread
From: John Goerzen @ 2022-05-30 0:23 UTC (permalink / raw)
To: nncp-devel
Hi folks,
After writing a forced-ordering program in shell for the nth time - for
handling things like incremental backups that can arrive in arbitrary
order from nncp-exec or nncp-file - I finally wrote it properly.
https://www.complete.org/filespooler/
I use it with NNCP, though it is a generic enough algorithm that it can
also use Syncthing, S3, etc. as a transport. Filespooler itself is a
Unixy tool, and provides no transport mechanism by itself; instead, it
can rely on anything you can pipe to, or anything like a filesystem. In
fact, both nncp-exec and nncp-file and feed to it:
https://www.complete.org/using-filespooler-over-nncp/
My other tool gitsync-nncp can also run over Filespooler:
https://www.complete.org/gitsync-nncp-over-filespooler/
Enjoy,
- John
^ permalink raw reply [flat|nested] 11+ messages in thread
* Re: Announcing Filespooler
2022-05-30 0:23 Announcing Filespooler John Goerzen
@ 2022-05-31 17:13 ` John Goerzen
2022-05-31 18:07 ` Jonathan Lane
2022-06-07 10:32 ` Sergey Matveev
2 siblings, 0 replies; 11+ messages in thread
From: John Goerzen @ 2022-05-31 17:13 UTC (permalink / raw)
To: John Goerzen; +Cc: nncp-devel
I realized that Filespooler without queues can make a perfect
augmentation to nncp-exec, allowing you to pass along environment
variables in addition to commands. I wrote a page for that here:
https://www.complete.org/using-filespooler-without-queues-to-pass-more-metadata/
On Sun, May 29 2022, John Goerzen wrote:
> Hi folks,
>
> After writing a forced-ordering program in shell for the nth time - for
> handling things like incremental backups that can arrive in arbitrary
> order from nncp-exec or nncp-file - I finally wrote it properly.
>
> https://www.complete.org/filespooler/
>
> I use it with NNCP, though it is a generic enough algorithm that it can
> also use Syncthing, S3, etc. as a transport. Filespooler itself is a
> Unixy tool, and provides no transport mechanism by itself; instead, it
> can rely on anything you can pipe to, or anything like a filesystem. In
> fact, both nncp-exec and nncp-file and feed to it:
>
> https://www.complete.org/using-filespooler-over-nncp/
>
> My other tool gitsync-nncp can also run over Filespooler:
>
> https://www.complete.org/gitsync-nncp-over-filespooler/
>
> Enjoy,
>
> - John
^ permalink raw reply [flat|nested] 11+ messages in thread
* Re: Announcing Filespooler
2022-05-30 0:23 Announcing Filespooler John Goerzen
2022-05-31 17:13 ` John Goerzen
@ 2022-05-31 18:07 ` Jonathan Lane
2022-06-02 0:03 ` John Goerzen
2022-06-07 10:32 ` Sergey Matveev
2 siblings, 1 reply; 11+ messages in thread
From: Jonathan Lane @ 2022-05-31 18:07 UTC (permalink / raw)
To: nncp-devel
On Sun, May 29, 2022 at 07:23:43PM -0500, John Goerzen wrote:
> Hi folks,
>
> After writing a forced-ordering program in shell for the nth time - for
> handling things like incremental backups that can arrive in arbitrary
> order from nncp-exec or nncp-file - I finally wrote it properly.
>
> https://www.complete.org/filespooler/
>
> I use it with NNCP, though it is a generic enough algorithm that it can
> also use Syncthing, S3, etc. as a transport. Filespooler itself is a
> Unixy tool, and provides no transport mechanism by itself; instead, it
> can rely on anything you can pipe to, or anything like a filesystem. In
> fact, both nncp-exec and nncp-file and feed to it:
>
> https://www.complete.org/using-filespooler-over-nncp/
>
> My other tool gitsync-nncp can also run over Filespooler:
>
> https://www.complete.org/gitsync-nncp-over-filespooler/
>
> Enjoy,
>
> - John
>
Very cool! One thing I'd add to your Syncthing article is that Syncthing
supports the concept of a folder being send-only or receive only, so you
could potentially wire up a full send-receive pair over a pair of
folders if you only needed to write between two machines. With a bit of
clever integration with Postfix you could probably use this as a mail
bridge.
--
tidux@sdf•org
SDF Public Access UNIX System - http://sdf.org
^ permalink raw reply [flat|nested] 11+ messages in thread
* Re: Announcing Filespooler
2022-05-31 18:07 ` Jonathan Lane
@ 2022-06-02 0:03 ` John Goerzen
0 siblings, 0 replies; 11+ messages in thread
From: John Goerzen @ 2022-06-02 0:03 UTC (permalink / raw)
To: Jonathan Lane; +Cc: nncp-devel
On Tue, May 31 2022, Jonathan Lane wrote:
>
> Very cool! One thing I'd add to your Syncthing article is that Syncthing
> supports the concept of a folder being send-only or receive only, so you
> could potentially wire up a full send-receive pair over a pair of
> folders if you only needed to write between two machines. With a bit of
> clever integration with Postfix you could probably use this as a mail
> bridge.
Quite true, one could tunnel Postfix or Exim over Filespooler as easily
as over NNCP. However, the problem with a unidirectional Syncthing
folder is the one of knowing when to delete the file. With
bidirectional syncing, the sender creates, and the receiver deletes
after processing.
- John
^ permalink raw reply [flat|nested] 11+ messages in thread
* Re: Announcing Filespooler
2022-05-30 0:23 Announcing Filespooler John Goerzen
2022-05-31 17:13 ` John Goerzen
2022-05-31 18:07 ` Jonathan Lane
@ 2022-06-07 10:32 ` Sergey Matveev
2022-06-10 13:14 ` John Goerzen
2 siblings, 1 reply; 11+ messages in thread
From: Sergey Matveev @ 2022-06-07 10:32 UTC (permalink / raw)
To: nncp-devel
[-- Attachment #1: Type: text/plain, Size: 357 bytes --]
Greetings!
*** John Goerzen [2022-05-29 19:23]:
>https://www.complete.org/filespooler/
Cool! Although I can not build it (because of Rust), I like its simple
and Unix friendly architecture, according to its documentation.
Looks great.
--
Sergey Matveev (http://www.stargrave.org/)
OpenPGP: CF60 E89A 5923 1E76 E263 6422 AE1A 8109 E498 57EF
[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]
^ permalink raw reply [flat|nested] 11+ messages in thread
* Re: Announcing Filespooler
2022-06-07 10:32 ` Sergey Matveev
@ 2022-06-10 13:14 ` John Goerzen
2022-06-10 18:48 ` Sergey Matveev
0 siblings, 1 reply; 11+ messages in thread
From: John Goerzen @ 2022-06-10 13:14 UTC (permalink / raw)
To: Sergey Matveev; +Cc: nncp-devel
On Tue, Jun 07 2022, Sergey Matveev wrote:
> Greetings!
>
> *** John Goerzen [2022-05-29 19:23]:
>>https://www.complete.org/filespooler/
>
> Cool! Although I can not build it (because of Rust), I like its simple
> and Unix friendly architecture, according to its documentation.
> Looks great.
Thanks! Out of curiosity, what are your challenges with Rust? Maybe I
can help in that area.
I have been using it in conjunction with NNCP for my backups now and the
combination is working quite well!
- John
^ permalink raw reply [flat|nested] 11+ messages in thread
* Re: Announcing Filespooler
2022-06-10 13:14 ` John Goerzen
@ 2022-06-10 18:48 ` Sergey Matveev
2022-06-10 20:46 ` John Goerzen
0 siblings, 1 reply; 11+ messages in thread
From: Sergey Matveev @ 2022-06-10 18:48 UTC (permalink / raw)
To: nncp-devel
[-- Attachment #1: Type: text/plain, Size: 952 bytes --]
*** John Goerzen [2022-06-10 08:14]:
>Out of curiosity, what are your challenges with Rust?
When I looked at it, it does not tell how to compile itself from the
sources. Its official suggestion is just to download some prebuilt
binary from the Internet and build Rust with it. So officially there is
(was? I did not check it for several years, because of no trust because
of all of that) no way to build it from the sources. Completely
unacceptable in my opinion and no trust for its creators. As I remember,
they wrote it initially on OCaml, that in turn was written on C.
Just out of curiosity I tried mrustc to build at least some of Rust's
version to be able to use it for bootstrapping further. I succeeded on
Devuan, but could not build it on FreeBSD. So several years ago I just
could not build it from the sources.
--
Sergey Matveev (http://www.stargrave.org/)
OpenPGP: CF60 E89A 5923 1E76 E263 6422 AE1A 8109 E498 57EF
[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]
^ permalink raw reply [flat|nested] 11+ messages in thread
* Re: Announcing Filespooler
2022-06-10 18:48 ` Sergey Matveev
@ 2022-06-10 20:46 ` John Goerzen
2022-06-10 21:24 ` Rust building Sergey Matveev
0 siblings, 1 reply; 11+ messages in thread
From: John Goerzen @ 2022-06-10 20:46 UTC (permalink / raw)
To: Sergey Matveev; +Cc: nncp-devel
On Fri, Jun 10 2022, Sergey Matveev wrote:
> *** John Goerzen [2022-06-10 08:14]:
>>Out of curiosity, what are your challenges with Rust?
>
> When I looked at it, it does not tell how to compile itself from the
> sources. Its official suggestion is just to download some prebuilt
> binary from the Internet and build Rust with it. So officially there is
> (was? I did not check it for several years, because of no trust because
> of all of that) no way to build it from the sources. Completely
> unacceptable in my opinion and no trust for its creators. As I remember,
> they wrote it initially on OCaml, that in turn was written on C.
>
> Just out of curiosity I tried mrustc to build at least some of Rust's
> version to be able to use it for bootstrapping further. I succeeded on
> Devuan, but could not build it on FreeBSD. So several years ago I just
> could not build it from the sources.
Thanks for this - you sent me down a path of exploring this.
rustc is written in Rust, as the Go compiler is written in Go. (You are
correct that rustc was once written in OCaml, but nowadays it's Rust).
https://rustc-dev-guide.rust-lang.org/building/how-to-build-and-run.html
describes the process. It does need a bootstrapping compiler, which is
described in
https://rustc-dev-guide.rust-lang.org/building/how-to-build-and-run.html
.
It looks like the story is similar with Go;
https://rustc-dev-guide.rust-lang.org/building/how-to-build-and-run.html
describes that "you need a Go compiler installed" to build it.
- John
^ permalink raw reply [flat|nested] 11+ messages in thread
* Re: Rust building
2022-06-10 20:46 ` John Goerzen
@ 2022-06-10 21:24 ` Sergey Matveev
2022-06-14 13:59 ` John Goerzen
0 siblings, 1 reply; 11+ messages in thread
From: Sergey Matveev @ 2022-06-10 21:24 UTC (permalink / raw)
To: nncp-devel
[-- Attachment #1: Type: text/plain, Size: 2449 bytes --]
*** John Goerzen [2022-06-10 15:46]:
>rustc is written in Rust, as the Go compiler is written in Go.
But the huge difference is that Go's documentation explicitly tells that
you only have to build Go 1.4 with C-compiler and then use it for any
other further versions. Clear, "non-secret" way to build it using native
C-compiler from C-sources and then from Go-sources. 10min and you will
get modern Go compiler.
>https://rustc-dev-guide.rust-lang.org/building/how-to-build-and-run.html
Tried it now. It went downloading
https://static.rust-lang.org/dist/2022-05-20/rust-std-beta-x86_64-unknown-freebsd.tar.xz
without any questions or warnings, trying to run it after. No
information how you can build that binary by self. I know that Rust 1.23
is built using 1.22, that was built with 1.21, and so on. If there were
any clear path how you can make all that path to the code you can build
from native OS'es C-compiler, then it will be ok. But there is no
official information how to do that. Just binaries downloaded from the
Internet. Like I am visiting some modern website sending me yet another
JS/WASM application my computer has to run. Authors just hardcoded
"download and run the binary". No way I can accept that behaviour.
Just out of curiosity I tried to build mrustc now. And it gives many
errors:
src/hir_conv/constant_evaluation.cpp:2003:41: error: use of undeclared identifier 'v'
return (v >> 8) | (v << 8);
src/hir_conv/constant_evaluation.cpp:2002:45: error: unknown type name '__builtin_constant_p'
static uint16_t bswap16(uint16_t v) {
src/hir_conv/constant_evaluation.cpp:2021:42: error: expected unqualified-id
rv = U128(H::bswap16(val.truncate_u64()));
I succeeded several years ago building it on Devuan only by taking the
notes and steps from GNU Guix'es ports, where they used mrustc to
bootstrap Rust building. It was not trivial task too, because many
things were slightly broken, some things required some cargo packages
downloading of some exact versions and patches, and so on. Maybe I could
cross-build Rust for FreeBSD from that Devuan's installation? No
information about that and FreeBSD is not officially claimed to be
supported by mrustc.
--
Sergey Matveev (http://www.stargrave.org/)
OpenPGP: CF60 E89A 5923 1E76 E263 6422 AE1A 8109 E498 57EF
[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]
^ permalink raw reply [flat|nested] 11+ messages in thread
* Re: Rust building
2022-06-10 21:24 ` Rust building Sergey Matveev
@ 2022-06-14 13:59 ` John Goerzen
2022-06-14 18:19 ` Sergey Matveev
0 siblings, 1 reply; 11+ messages in thread
From: John Goerzen @ 2022-06-14 13:59 UTC (permalink / raw)
To: Sergey Matveev; +Cc: nncp-devel
On Fri, Jun 10 2022, Sergey Matveev wrote:
>>https://rustc-dev-guide.rust-lang.org/building/how-to-build-and-run.html
>
> Tried it now. It went downloading
> https://static.rust-lang.org/dist/2022-05-20/rust-std-beta-x86_64-unknown-freebsd.tar.xz
>
I don't want to derail too much, but there is a FreeBSD port of Rust,
and a team around it in FreeBSD:
https://wiki.freebsd.org/Rust
> without any questions or warnings, trying to run it after. No
> information how you can build that binary by self. I know that Rust 1.23
> is built using 1.22, that was built with 1.21, and so on. If there were
> any clear path how you can make all that path to the code you can build
> from native OS'es C-compiler, then it will be ok. But there is no
On the one hand, I can see your point here. Many (definitely not all)
languages have a compiler or interpreter implemented in C. So we can
expect to be able to begin with binaries for cc, as, ld, and friends,
and then from that, build other things.
On the other hand, I think that Rust (and also Go in a different way)
make a pretty credible case that the era of C dominance ought to be
coming to an end before long. Already one can write device drivers for
both Linux and FreeBSD in Rust. So, I don't want to say "has a C
implementation" ought to be a requirement for a language I use
More broadly, I don't want to "bless" C as somehow special. "Well it's
OK if we have to download a distribution-supplied precompiled binary of
cc, but not other languages." If cc can be written in C, why not rustc
in Rust?
- John
^ permalink raw reply [flat|nested] 11+ messages in thread
* Re: Rust building
2022-06-14 13:59 ` John Goerzen
@ 2022-06-14 18:19 ` Sergey Matveev
0 siblings, 0 replies; 11+ messages in thread
From: Sergey Matveev @ 2022-06-14 18:19 UTC (permalink / raw)
To: nncp-devel
[-- Attachment #1: Type: text/plain, Size: 1339 bytes --]
*** John Goerzen [2022-06-14 08:59]:
>If cc can be written in C, why not rustc in Rust?
It can, no problems, but it should be bootstrapable, especially taking
into account that it was already initially written on OCaml, high-level
language, that can be in turn bootstrapped with CC. Initial commits in
Rust repository even contains its OCaml version, but why its authors do
not provide instructions how can I build it up from the ground, to
reproduce their job? Even CC can be bootstrapped more or less with GNU
Mes project as it is done in GNU Guix, where they build Scheme
interpreter, TinyCC, early GCC with only relatively tiny binary blob,
being lighter with its latest releases. CC is just so old, initially
written on BCPL, assembly, for some hardware is not used anymore. But at
least it is some common denominator existing in virtually all Unix-like
OSes out-of-box. Clear official rules and guides how Rust can be build
from the ground, how can I reproduce the work done by their authors will
close that problem (of course by the price of CPU time spent compiling a
dozens of its versions). Or at least clear direction to use something
like mrustc, but making it buildable not only on three OSes.
--
Sergey Matveev (http://www.stargrave.org/)
OpenPGP: CF60 E89A 5923 1E76 E263 6422 AE1A 8109 E498 57EF
[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]
^ permalink raw reply [flat|nested] 11+ messages in thread
end of thread, other threads:[~2022-06-14 18:20 UTC | newest]
Thread overview: 11+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2022-05-30 0:23 Announcing Filespooler John Goerzen
2022-05-31 17:13 ` John Goerzen
2022-05-31 18:07 ` Jonathan Lane
2022-06-02 0:03 ` John Goerzen
2022-06-07 10:32 ` Sergey Matveev
2022-06-10 13:14 ` John Goerzen
2022-06-10 18:48 ` Sergey Matveev
2022-06-10 20:46 ` John Goerzen
2022-06-10 21:24 ` Rust building Sergey Matveev
2022-06-14 13:59 ` John Goerzen
2022-06-14 18:19 ` Sergey Matveev