00:37 <sb0> rjo, you have to determine and read yourself the correct amount of bytes, buffer it, and send it to pickle

00:38 <sb0> rjo, when reading without asyncio, you could pass the file object directly to pickle, which would call read() on it for you (and do the size determination which already exists in the pickle format it seems)

00:39 <sb0> but since it does "file_object.read()" and not "yield from file_object.read()", you can't use asyncio...

00:39 <sb0> asyncio has this sort of integration issue with about everything

00:42 <sb0> stekern, I just took their generated fpgatop.v and fpgatop_mem.v (or something like that), added `define SYNTHESIS on top of each, and put them into a ISE GUI project

00:50 <sb0> rjo, btw, surprisingly, pickle is slower than json

01:48 <rjo> sb0: oh. but that has nothing to do with asyncio specifically. you will see that with any asynchronous/coroutine style framework.

01:48 <rjo> sb0: none of them can allow blocking ops.

01:49 <rjo> sb0: in serializing/deserializing speed?

01:49 <sb0> it would be nice, though, if the language would automatically turn functions that do blocking IO into coroutines somehow

01:49 <sb0> it would also remove all the "yield from" clutter

01:50 <sb0> yes

01:53 <rjo> sb0: yes. "yield from" is a bit ugly. but much nicer than the equivalent in twisted (e.g.).

01:55 <rjo> sb0: well. i guess you could do the usual: expose the pickle read() blocking in a thread and feed data to that thread from the event loop in asyncio-style.

01:55 <sb0> yeah, meh

01:55 <rjo> sb0: i never liked pickle somehow. tried it about a hundred times for different things. but usually something else is much nicer (sqlite, numpy, json, yaml)...

01:56 <rjo> sb0: ;)

01:56 <sb0> right now what I'm doing is using readline() for each message instead, since json escapes \n

01:56 <sb0> and buffer it

02:00 <rjo> sb0: in strings, that is?

02:00 <rjo> sb0: if you pretty-print json it has lots of \n ...

02:00 <sb0> yes. and I disable the pretty-printing

02:02 <rjo> sb0: looks like you would be better of with some datagram or messaging style transport and not a "stream"

02:02 <rjo> s/of/off/

02:03 <sb0> I need it at two places:

02:03 <sb0> 1) communication between master and client: datagram protocol is UDP, message size is limited by MTU

02:04 <sb0> 2) communication between master and worker process: unix sockets are messy, stdout/stdin is stream

02:04 <sb0> and modules like zeromq, multiprocessing, etc. are not asyncio-compatible

02:05 <rjo> https://github.com/aio-libs/aiozmq

02:05 <rjo> what are worker processes? the distributed adapters between devices and the master?

02:06 <sb0> yeah, I saw it... and I'm a bit hesitant to add dependencies like this, which can end up messy, e.g. llvmpy

02:06 <rjo> sb0: "But Windows is a second-class citizen in ZeroMQ world, sorry.

02:06 <rjo> sb0: nice

02:07 <sb0> and it doesn't solve the multiprocessing problem. using the asyncio process functions + stdin/stdout as asyncio streams is not too bad

02:07 <rjo> sb0: yep. maybe 0mq is in fact overkill.

02:07 <rjo> did github change syntax highlighting colors recently?

02:08 <sb0> worker process - I'd like to have the master run the experiment code in a child process, which makes it easier to implement e.g. timeouts and helps contain crashes

02:08 <sb0> especially if that worker process is going to load $DISGUSTING_BINARY_DRIVER

02:09 <sb0> the idea is to have the master handle client connections on one side, and a child process on the other - all with asyncio

02:10 <sb0> the master process would do the communication with clients, scheduling/queuing, extraction of the code from the experiment repository, and tell its worker process to run module X or Y

02:11 <rjo> sb0: yep. i called them "controllers" in the original design.

02:12 <rjo> sb0: the core device controller/worker can also be separate from the master.

02:13 <sb0> by 'separate' you mean on a different machine?

02:14 <rjo> separate process actually. but that then allows different machines.

02:15 <sb0> yes, I'm planning to do that (instead of a thread) for crash/memory leak containment

02:16 <rjo> sb0: good. i like that. the only thing i am a bit worried about is startup and shutdown of that bundle of processes potentially also across machines.

02:17 <rjo> gtg. adjust some vacuum bakeout heaters...

02:17 <sb0> different machines is a bit harder, I'm planning to checkout all the experiment git into some folder in /tmp (to allow experiments that are spread across several modules in several files), and I'm also simply doing a "asyncio.create_subprocess_exec(sys.executable, ..."

02:18 <sb0> the worker process is only run once in the best case. it stays online until it crashes, or is killed.

02:20 <rjo> sb0: you will need some heartbeat protocol and a cascade of mechanisms to SIGTERM/SIGKILL processes.

02:22 <sb0> we don't necessarily need a heartbeat protocol. if the worker process doesn't reply within a reasonable time to the next "run" command, or if its results are excessively delayed, kill + rerun

02:24 <sb0> what are you baking out? :)

17:32 <rjo> sb0: timeout on run() might be a bit long. especially for long running experiments (infinitely long running).

17:32 <rjo> sb0: baking one of my trap chambers.

23:44 <rjo> sb0: i forgot about the example yesterday. will do that now.