public inbox for nncp-devel@lists.cypherpunks.ru
Atom feed

* Streaming data of unknown length
@ 2022-11-08 23:03 Jonathan Hemingway
  2022-11-09 19:23 ` Sergey Matveev
  0 siblings, 1 reply; 2+ messages in thread
From: Jonathan Hemingway @ 2022-11-08 23:03 UTC (permalink / raw)
  To: nncp-devel

Hello,

I have an interesting use-case for NNCP and I'm wondering how best to go about it.

Imagine I have a security camera with a fallback battery power supply. Normally the
camera stream is recorded to local storage and discarded after some arbitrary period.

In the event that main power is cut or some other emergency conditions are met the
video stream should immediately feed to some remote storage as well. The remote
storage channel would be NNCP so that the video stream can be transmitted to a
dependable relay and decrypted later.

This situation is different from how NNCP normally works because it must be assumed
that the sender may be taken permantly offline at any momement during transmission.
In this case, what is the best way to use NNCP? A file-transfer with chunking?

Here is an example script to show what I mean:
---------------------------------------------
#!/bin/sh
NODE="${1}"
FILE="$(mktemp --dry-run XXXX.ismv)"

# encode from webcam into nncp-file
ffmpeg -nostats \
                -f alsa -i default -f v4l2 -i /dev/video0 \
                -b:a 48K -b:v 128K \
                -timestamp now \
                -f ismv pipe: \
        | nncp-file -chunked 128 -quiet -nice F - "${NODE}:${FILE}" \
        &

while true; do
        sleep 2s
        nncp-call -onlinedeadline 60 -tx "${NODE}"
done
---------------------------------------------

I'm using chunking here because I think it is the easiest way to reconcstruct
an interrupted stream, but I'm not sure if that is how it works.


Cheers,
Emery

^ permalink raw reply	[flat|nested] 2+ messages in thread

• * Re: Streaming data of unknown length
    2022-11-08 23:03 Streaming data of unknown length Jonathan Hemingway
  @ 2022-11-09 19:23 ` Sergey Matveev
    0 siblings, 0 replies; 2+ messages in thread
  From: Sergey Matveev @ 2022-11-09 19:23 UTC (permalink / raw)
    To: nncp-devel
  
  [-- Attachment #1: Type: text/plain, Size: 2120 bytes --]
  
  Greetings!
  
  *** Jonathan Hemingway [2022-11-08 23:03]:
  >In this case, what is the best way to use NNCP? A file-transfer with chunking?
  
  Personally I have never tried real streaming use-cases with NNCP,
  moreover only relatively recently it was able to send data without
  storing it in temporary file (to determine resulting packet's size),
  but seems that the only friendly way is indeed just to use chunked
  transfer and to dump packets to remote side in background.
  
  Although you won't be able to automatically reassemble (nncp-reass)
  those chunks to the single file, because .nncp.meta (that holds all
  information needed for reliable reassembling) is sent only when stdin is
  dried up -- but .nncp.chunk files are numbered and hardly can be damaged
  if successfully already tossed. You can just literally concatenate them
  (preserving the order). .nncp.meta file additionally holds just chunk's
  checksums.
  
  But pay attention that while nncp-file is generating new chunks, it
  accumulates their checksums (for resulting .nncp.meta) in memory.
  32-byte for each chunk, but if it is long-running process with thousands
  of chunks already generated, then it could noticeably grow.
  
  Another problem I see is that you can fill your spool of encrypted
  queued packets quicker than you transfer them. And soon you (nncp-file)
  will just file because of lack of disk space. Another problem is that
  each time nncp-call connects and online-protocol is activated -- there
  is no particular order of packets delivery. None of NNCP component's
  look at mtime/ctime/whatever and have got any kind of date-based
  ordering. You can queue hundred of packets, but remote side will
  download them in random order. But if amount of generated data is
  definitely not a problem for available disk space and outgoing online
  bandwidth, then seems it should work. However it is definitely not the
  very good solution, taking in account that NNCP never tried to preserve
  any kind of packets transmission ordering.
  
  -- 
  Sergey Matveev (http://www.stargrave.org/)
  OpenPGP: 12AD 3268 9C66 0D42 6967  FD75 CB82 0563 2107 AD8A
  
  [-- Attachment #2: signature.asc --]
  [-- Type: application/pgp-signature, Size: 228 bytes --]
  
  ^ permalink raw reply	[flat|nested] 2+ messages in thread

end of thread, other threads:[~2022-11-09 19:24 UTC | newest]

Thread overview: 2+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2022-11-08 23:03 Streaming data of unknown length Jonathan Hemingway
2022-11-09 19:23 ` Sergey Matveev