01:56 <rjo> sb0: https://github.com/nist-ionstorage/artiq/blob/master/examples/transport.py

01:58 <rjo> sb0: i think that is a feasible API. i don't see how we could magically extract the pdq preparation stuff from @kernel code otherwise.

01:59 <rjo> sb0: this way, all that needs to be passed from host-side prepare() to @kernel code is the frame id and segment metadata (duration and triggering).

02:00 <rjo> sb0: if you find bugs or unclear/impossible behaviour, let me know ;)

02:02 <rjo> sb0: could'nt we automatically add **kwargs that are passed to AutoContext.__init__() and are not yet in AutoContext.parameters?

02:06 <rjo> s/d'nt/dn't/

02:34 <sb0> rjo, it's already doing that, isn't it?

02:35 <sb0> parameters only lists what must be requested from the database unless it's passed as kwarg

02:38 <sb0> "ensures all segments have been advanced() through" << what should it do when not the case? throw an exception?

02:39 <sb0> btw you can use array() in non-kernel code, instead of [0] * self.nbins

02:40 <sb0> its advantage there, besides consistency with kernel code, is it avoids the list reference problems with [[0]*x]*y

02:48 <sb0> rjo, you can also just do self.histograms = [] in scan() and then self.histograms.append(hist[i]) in kernel. since the last round of inline transform refactoring/bugfixing, it RPCs the right thing...

03:09 <sb0> rjo, seems to me the compiler should be able to handle all this stuff already.

03:10 <sb0> one small thing we can add is supporting iterators for parameters (in addition to the spaces), to avoid having to use a space at the end of the 1st line

03:13 <sb0> where should your *.npz files live? in the same repository as the experiment sources?

05:11 <rjo_> sb0: re additional parameters. nice! i didn't check.

05:12 <rjo_> sb0: yes. if the method can not do what it is asked to do because it is in the middle of another frame, that is an error in any case. otherwise the voltages would jump.

05:21 <rjo_> sb0: ... that is even it it was possible to leave a frame

05:27 <rjo_> sb0: re array(): ack. but [[0]*x]*y] does not have an array() equivalent :)

05:27 <sb0> array(array(0, x), y)

05:27 <rjo_> sb0: ah. yed.

05:27 <rjo_> sb0: ah. yes.

05:27 <sb0> works both in interpreter and kernel

05:27 <rjo_> true.

05:28 <rjo_> i had leaped to complex slices array(0, nd, ny)[x, y] ...

05:29 <rjo_> sb0: yeah. you done quite a bit of refactoring. i have to geat my bearings straight...

05:31 <rjo_> sb0: oh. it rpc's methods. that is nice.

05:32 <sb0> right now the main limitation is all arguments need to be 32-bit ints, but I plan to improve that

05:32 <rjo_> sb0: re npz. they are very much like the other nomeric parameters that represent some "calibration" things, like duration of pulses, frequencies etc.

05:33 <sb0> so they should rather be in the parameter database?

05:34 <rjo_> sb0: yeah. i tried to think through a way have prepare() and transport() -- that is the host/slow part and the kernel part -- interleaved and have the transforms do the rest but i think that will be very difficult.

05:34 <sb0> the main issue is when to call prepare()

05:35 <rjo_> sb0: yes. conceptually the parameter database should support "large" data, like numpy arrays.

05:35 <sb0> and with which waveforms

05:36 <rjo_> sb0: yes. and to ensure that all methods to frame and pdq2 do not break the notion of not changing the waveform between repeats...

05:38 <rjo_> sb0: i think having split prepare() and transport() is just fine. especially since the code to verify the lock-step between pdq2 segments and rtio does not seem too difficult.

05:40 <sb0> how large? too large for json?

05:43 <sb0> rjo_, we can also call an auto-built prepare() every time before entering kernel mode, with all the possible waveforms that the core device might want

05:44 <sb0> (for that kernel execution)

05:45 <sb0> though I'm not sure about performance problems, and how they could be alleviated by caching and omitting the programming of waveforms that are already there - since they'd need to be extracted everytime anyway

05:46 <rjo_> sb0: well master-to-controller json is going to be fine. i am doing tens of MB with json easily. i'll have to think about larger data for e.g. controller-to-core

05:46 <rjo_> sb0: the remark about large data would be for the versioning.

05:47 <rjo_> sb0: pdq2 has typical order of magnitude of 500 points per channel. that is 500*36 knots for all channels in a single pdq2 rack.

05:49 <sb0> git would only version the diffs, no?

05:50 <sb0> do you expect a lot of diffs?

05:52 <rjo_> sb0: i see yes. something like a "with self.electrodes.prepared(): ..." context from within the kernel. but autobuilding prepare() is difficult. how do you telll if something that may or may note touch the waveform is suppposed to end up in prepare()?

05:53 <rjo_> sb0: yeah. but the npz-diffs are unreadable (by humans).

05:54 <rjo_> sb0: no. the waveforms are rarely goig to change.

05:55 <rjo_> s/goig/going/ and so forth.

05:59 <sb0> how about not using npz, and storing arrays as json?

06:01 <sb0> (waveforms) the main issue there is that the sequence of calls that trigger waveform parts can be inside a complicated algorithm

06:02 <rjo_> sb0: i think json is fine.

06:03 <sb0> one would have to unroll that algorithm, which may or may not be possible, to find out what the sequence of waveform parts is

06:03 <rjo_> sb0: yes. unraveling waveform construction from actual rtio-like things would be difficult. if you see a way to do it that would be great, otherwise i think the split is readable.

06:05 <sb0> the general case is essentially solving the halting problem. one would have to put potentially user-unfriendly restrictions on what algorithms can benefit from the auto-extraction

06:05 <rjo_> sb0: that example with appending the reverse transport is just a simple one. there can be more array juggling.

06:07 <sb0> by user-unfriendly, I mean we can have some heuristics, but they will always be some cases where they fail. in that case, the user would have to rewrite their code to use the split prepare/trigger form

06:07 <sb0> *there will

06:07 <rjo_> sb0: yes. looks like the halting problem. what a relief!

06:08 <rjo_> :)

06:09 <rjo_> the advantage is also that it is obvious what the code will actually do if it is split().

06:09 <rjo_> damn. just "split."

06:10 <rjo_> not every word is a function...

06:11 <rjo_> also. one could actually name the segments like append("a_to_b", t, u) and then recall them by name advance("a_to_b") and just error out if the sequence is wrong.

06:12 <rjo_> that would make it explicit what segment advance() actually plays and force the user to do the right thing.

14:29 <sb0> http://copsub.com/

15:53 <sb0> so there's asyncio.open_connection(address, port) and socket.create_connection((address, port)). meh.

16:23 <sb0> rjo, can the pdq be easily connected to the ppro using the current adapter board?

16:24 <sb0> it'd be nice if the openrisc folks would debloat their stuff to make room for the extra rtio channels *g*

16:24 <sb0> we can probably trim some fat from the sdram controller too