sb0 changed the topic of #m-labs to: ARTIQ, Migen, MiSoC, Mixxeo & other M-Labs projects :: fka #milkymist :: Logs http://irclog.whitequark.org/m-labs
<whitequark>
rjo: python's parser is unsuitable for the kind of tooling we are building
<whitequark>
it only allows to return crude error messages based on line number and sometimes column
<whitequark>
the one I'll make will give you the column numbers for every token, expression, etc
<whitequark>
similar to what clang can do
<whitequark>
it's not a significant amount of work (i already wrote one such parser and i'm just rewriting chunks of it in python) and it improves usability greatly
sturmflut_ has joined #m-labs
SturmFlut has quit [Ping timeout: 264 seconds]
sb0 has joined #m-labs
<sb0>
whitequark, what's the reason for not reusing actual ast nodes?
<sb0>
with ast nodes, you could easily run modified asts - and that's something we may need later
<sb0>
also, code becomes more directly reusable (there are often tests like isinstance(x, ast.yyy))
<whitequark>
sb0: reusing as inheriting or reusing as in taking whatever python's parser gave out?
<whitequark>
I don't have an opinion on the former except it's possible that python's AST nodes are some weird C thing you can't properly inherit from
<whitequark>
(as in, I'll do it if it's possible)
<whitequark>
the latter doesn't really make sense to me, it's just more work for no benefit
<whitequark>
(you have to match /your/ nodes to /its/ nodes but you still have to parse everything anyway)
fengling has joined #m-labs
<whitequark>
from a quick check it appears that I will be able to inherit from ast's nodes, yes
<GitHub148>
[pyparser] whitequark pushed 1 new commit to master: http://git.io/j1pf
<sb0>
whitequark, inherit or just patch attributes dynamically
<sb0>
the latter works (the transforms already do some of that)
fengling has quit [Ping timeout: 272 seconds]
fengling has joined #m-labs
kugg has quit [Ping timeout: 240 seconds]
kugg has joined #m-labs
<rjo>
whitequark: in that case the right way would seem to be to fix the cpython parser and upstream those features would it not?
<whitequark>
rjo: there are several issues with that
<rjo>
whitequark: where do we need more information than the usual level of cpython ast (without column)?
<whitequark>
rjo: py2llvm diagnostics?
<whitequark>
I mean... compare pre-4.7 (or so) gcc and clang
<rjo>
carrying over the ast level of metadata into py2llvm.
<whitequark>
sorry?
<rjo>
whitequark: yes. i see why good debugging symbols are nice and i enjoy it ;) but is this the most efficient way to spend development time right now?
<whitequark>
sure why not
<whitequark>
i'm nearly done with the lexer
<whitequark>
a few more days and it'll be finished. python is substantially simpler than i thought
<rjo>
whitequark: a parsed python source code snipped gives you an ast with some level of debugging information (line numbers, not columns).
<whitequark>
yes, I know
<whitequark>
I've looked at the builtin ast module and its implementation
<rjo>
and that could just be carried over (to the extend reasonably possible) through py2llvm into IR.
<whitequark>
thus i can say that extending upstream is /not/ a good way to spend development time, plus i'm not even sure if upstream /wants/ it, plus shipping a modified python binary is a pain
<whitequark>
yes
<whitequark>
well, it's not just IR
<whitequark>
it's py2llvm itself's diagnostics
<whitequark>
IR is mainly useful for backtraces; while convenient, proper location information doesn't have that much impact there
<rjo>
if you had to guess, what is the slow-down of lexing/parsing wrt cpython ast?
<whitequark>
2x or less
<whitequark>
parsing overhead is negligible, for lexing I'm heavily leaning back at python's re module
<whitequark>
one token = one re.match invocation
<sb0>
hmm, parsing isn't that fast
<rjo>
yes. that is what i meant with carrying over. the py2llvm passes need to rais their own errors based with the metadata.
<whitequark>
sb0: I'm not speculating. I'm basing this on data from my ruby parser
<whitequark>
ruby parser lexes in pure ruby (using a ragel-built dfa), not even via regexps
<whitequark>
the parsing overhead is in single %s
<whitequark>
the lexing overhead is significant but python's simple enough you can trick re into giving you complete tokens
<whitequark>
I'm assuming re has a decent implementation
<sb0>
well, by "parsing" I meant the call to "ast.parse", which does lexing + parsing
<rjo>
well. i also worry about having to maintain another piece in the puzzle. language might change, pyparsing and cpython might diverge in hidden ways...
<sb0>
rjo, ad9726 is a 400MSPS DAC. should that be used like the PDQ?
<sb0>
I thought you wanted a slow serial DAC
<whitequark>
rjo: I will have integration tests verifying ast equality against current python, plus if that was ruby, it'd be a problem
<whitequark>
rjo: python has a clear grammar specification you can simply diff
<rjo>
sb0: it is the pdq2
<sb0>
so what does "spi to rtio-bus for ad9726" mean?
<rjo>
who calls for ad9726?
<rjo>
ah. did i write that. i meant ad5370 ...
<sb0>
ah, ok. makes more sense
<rjo>
whitequark: my complaint is not about ability but about having to do it ;)
<sb0>
so the API would be "dac_channel.set_voltage()" at now()?
<whitequark>
rjo: to rephrase: I don't expect that to require any significant amount of time
<whitequark>
I've reviewed the changes in grammar from 2.6 to 3.4
<whitequark>
to arrive to this
<rjo>
sb0: could there be a mediator that converts from the pdq2 style programs to set_voltage() and interleaves?
<rjo>
whitequark: ok. lets see. but i will have a warm and fuzzy feeling when i find the first bug and can say "i told you so" ;)
* whitequark
shrugs
<rjo>
and carrying over all that heavy debugging information (with column numbers, ranges etc) through py2llvm is no biggie?
<whitequark>
i want high quality tooling
<whitequark>
no, not a problem
<sb0>
just pass by reference
<whitequark>
sb0: i create a bunch of heap objects, each 3 words big
<whitequark>
if I remember cpython's optimization correctly, every source range is 6 heap words or so
<whitequark>
if this becomes unacceptable, I know how to optimize it to a single fixnum, but that's unlikely to be necessary
<rjo>
also acting on it and unwinding a funny stack with inlining and interleaving?
<sb0>
rjo, the mediator could call wavesynth to convert the program to samples, put that in a big list, and then the kernel would loop over the list and call set_voltage and delay
<rjo>
sb0: sounds about right.
<whitequark>
rjo: that's not really any different to existing ast debugging info
<rjo>
whitequark: there zero ast debugging info carried over into IR at the moment AFAIK.
<sb0>
rjo, this won't work in parallel blocks because the delay will appear dynamic (as it's inside a loop), but we can add detection of simple cases like this (delay in a for loop of a fixed length without break)
<whitequark>
rjo: yeah, I only meant it's not relevant to pyparser
<rjo>
sb0: ack.
<sb0>
hm. though the for loop can be broken (through an exception) in case set_voltage underflows rtio
<whitequark>
if we're talking about just carrying over location info to IR, yes, it's not that hard, our transforms just have to be aware of location info
<rjo>
whitequark: ok. i am not the expert on that. i'll leave it to you guys. but it does sound weird that we need to replace the cpython parser/lexer.
<rjo>
sb0: yes. the context would be pretty restricted where the interleaving would work. that is fine.
<whitequark>
rjo: i wrote a ruby parser a while ago specifically because existing ruby parser's lineno reporting was not enough for a compiler from ruby-like lang to LLVM IR
<rjo>
conceptually, why do you need more debugging info if you do X-to-llvm than if you do X-interpreted?
<sb0>
rjo, I think we'll soon end up with a multicore system and a crossbar between the cores and each rtio fifo...
<sb0>
and each core running one branch of parallel blocks. also for performance reasons...
<whitequark>
rjo: fundamentally it is not about to-llvm but about introducing a type system (and py2llvm effectively has one)
<whitequark>
adding unusual control flow has the same effect. if some nontrivial invariant is violated, you better be able to explain to the user really well what is wrong with it
<whitequark>
and interpreted dynlangs have comparatively few nontrivial invariants, it's almost exclusively "this value doesn't respond to that method"
<rjo>
sb0: "end up" as in "accidential" or as in "according to plan"?
<rjo>
ah
<rjo>
not the comms-cpu/rtio-cpu split
<rjo>
but the rtio1/rtio2 cpu split?
<rjo>
hyperthreading...
<rjo>
whitequark: hmm. do you have an example?
<whitequark>
say if a type of a variable was inferred as int for some reason, and you're adding a float to it
<whitequark>
you want to show where exactly it was inferred as int, where you add a float, possibly where the second variable was inferred as float
<sb0>
rjo, well, the more channels you need to control, the more cpu power you need
<rjo>
whitequark: ok. you should sell that parsing/debugging feature-set to numba. would you not expect them to constantly complaining about the problem?
<sb0>
rjo, the Oxford people want a hundred or more dds channels...
<sb0>
rjo, was there a decision made about duty cycle management?
<whitequark>
rjo: that's why I'm writing this as a separate package from artiq
<rjo>
sb0: that does not make sense to me. there is > 10k$ worth of equipment on each dds channel.
<whitequark>
when I wrote the ruby parser, it was a hit
<rjo>
sb0: Joe and John who I asked to look into this feature did not seem to feel any pain having to do it the pedestrian way and did apparently not understand how the hardware logic analyzer would be used to do the heavy lifting automatically.
<whitequark>
rjo: I've barely crossed paths with people using python but I would be surprised if it's not a problem for numba
<rjo>
sb0: i would conclude that we delay it.
<sb0>
rjo, so we don't put it in this extension?
<rjo>
i would live to see it and it is a prime use case for the hard-LA but there is apparently no market.
<rjo>
so yes.
<sb0>
I wouldn't do it with the hard-LA, but with counters attached to each RTIO channels
<sb0>
ok
<rjo>
but they would sit right where the RLE encoder for the hard-LA would is.
<rjo>
*would be
<rjo>
to me the two are very related. hard-LA gives you history, duty-cycle-counters give you an average since reset.
<sb0>
yes. but the difficulty in the hard-LA is RLE encoding, putting all the data together, and managing DRAM
<rjo>
whitequark: ok.
<sb0>
getting the data from the rtio-bus is straightforward
<rjo>
sb0: ack. the hard-LA is bigger.
<sb0>
rjo, and how do we keep duty cycle during kernel handovers? preprogram enough pulses and leave the RTIO core running?
<rjo>
sb0: i guess all you would need to do is If(rtio, duty_cycle.storage.eq(duty_cycle.storage + 1))...
<rjo>
sb0: yes. how fast is a handover with a back-buffered kernel?
<sb0>
rjo, what if a large part of the experiment is in non-kernel mode, e.g. CPU intensive on the PC, or making a RPC call to a particularly slow device?
<sb0>
if everything is already buffered, a handover is the same order of magnitude as a function call
<rjo>
sb0: split the rpc or disallow it.
<rjo>
that is what i would have guessed.
<rjo>
the cpu intensive stuff would need to go into the pipelined prepare() or analyze()
<rjo>
sb0: re duty cycle: that register transfer between the clock domains is similar to the one for the debug interface, right?
<sb0>
yes, but the debug interfaces does not need to be precise
<rjo>
the duty cycle counters dont need to be precise either. i guess knowing the duty cycle to 8 bits is sufficient for virtually all cases.
<rjo>
that is however more than the single bit
<rjo>
that is needed for the debug if.
<sb0>
whereas with duty cycle counters, you want a clear definition of what the duty cycle is for a given value of now()
<rjo>
but now() is in rtio cycles.
<sb0>
yes. and the duty cycle counters will use rtio cycles, but the accounting of rtio cycles will happen in the CPU clock domain
<rjo>
ah. i get it.
<rjo>
you want to accumulate based on what is put into the fifo.
<sb0>
yes
<rjo>
the delta between pulses.
<rjo>
ok.
<rjo>
fine. that is perfect.
<rjo>
would the hard-LA be in rtio-domain or in cpu?
<rjo>
sounds like we have the duty-cycle thing sorted out. agree?
<rjo>
then onto wb2rtio.
<sb0>
we can do both, since we just want a full dump over a length of time and not a quick readout from the CPU at a given time
<sb0>
doing it in CPU domain spares us one (easy) clock domain transfer
<sb0>
yes, duty cycle is ok
<rjo>
what do you mean by both?
<sb0>
I mean: either will work
<rjo>
ok.
<sb0>
doing it in CPU domain is slightly easier imo
<rjo>
whatever is easier as long as it does not tax the cpu at all in normal mode.
<rjo>
once you want to do download the post-mortem all bets can be off.
<sb0>
it will DMA all the time and use DRAM bandwidth. other than that the CPU doesn't have to worry about it.
<rjo>
ack
<rjo>
re wb2rtio. imho the naturral way to represent a wb-read in rtio would be to use that rtio-channel as output for address and then put the the data into an input fifo. then you can process the read data at your convenience and independent of the wb cyc length and wb device latency.
<rjo>
that feature should be called rtio2wb (rtio-slave, wishbone-master)
<sb0>
wait, there are 2 things:
<rjo>
the wb2rtio would replace the test mode for the dds in the runtime.
<sb0>
1) an adapter that takes a device with a RTIO-bus interface on it and wraps it so it can be connected to a wishbone bus in the same clock domain
<sb0>
2) injection of RTIO commands from the CPU in debug mode
<sb0>
#2 would just use the same type of register interface used right now I think
<rjo>
hmm. 1) seems a bit boring. my use case would be instead of doing all these complicated writes to the rtio-bus fifos, just writing/reading to wishbone. thus not only exposing one rtio device on wb via synthesis changes but exposing the entire rtio-bus on wb at runtime.
<sb0>
#1 is trivial, yes. and basically a development thing...
<rjo>
is that then equal to 2)?
<sb0>
there is no entire rtio-bus. each fifo has a point-to-point link to its device.
<sb0>
the demultiplexing is done in the cpu domain, before the fifos
<rjo>
yes. they all lie flat on wb or csr.
<rjo>
let's step back.
<sb0>
rtio-bus is a bit of a misnomer, since the links are point-to-point. I guess it came from dds *bus* on rtio...
<rjo>
ack.
<rjo>
my use case would be writing rtio-spi.
<rjo>
more rpecisely rtio-ad5370
<rjo>
one approach could be to start with wb-ad5370 with mapped registers and debug that thoroughly.
<rjo>
then if we had rtio2wb, we could just hide the wishbone device behind rtio and be happy
<rjo>
the other approach would be start with rtio-ad5370 right away but wrap the rtio in wb2rtio and again get memmapped registers
<sb0>
so a rtio-bus device would have an optional (as ttls won't need it) abstract concept of addresses
<rjo>
either is ok by me. which is easier?
<sb0>
that rtio2wb would map to the wishbone address, if it exists?
<rjo>
yes
<rjo>
yes
<sb0>
and then data, of an arbitrary number of bits (1 for ttls, 32 for dds, etc.)
<rjo>
the fifo data layout would be (time, addr, dat_w, w_en) for the output fifo and (time, addr, dat_r) for the input fifo.
<sb0>
there should also be the configuration signals (e.g. ttl oe) somewhere
<sb0>
though that can be done with a special address, maybe
<sb0>
and then ttls would also have addresses
<sb0>
memory-mapped-IO within RTIO
<rjo>
dat_{rw} being something like (o, oe, rising, falling) for a hypothetical ttl wishbone module that is to be wrapped in rtio2wb
<sb0>
oe management isn't that simple, since the action of the output commands depend on the oe state (set level on the line for output, open/close the gate for input)
<sb0>
but a special address that switches oe would work
<rjo>
you had suggested wb2rtio last time. and the more i think about having to choose at synthesis time is ok.
<rjo>
you could just ignore o if !oe the way tstriple already does it, no?
<sb0>
you don't want to ignore o. you want to recycle the output commands to open/close the gate when in input mode
<sb0>
well I guess that once we have addresses, opening/closing the gate can be done with another address than the one used for o
<sb0>
then you can even do loopback tests without having to connect two RTIO pins on the board
<rjo>
ack. iirc i had that in ventilator like that (but the ttls were a 32-bit bank) and there were 4 addresses.
<rjo>
yes.
<rjo>
but isn't wb2rtio (at synthesis time) the pragmatic solution for this debugging?
<rjo>
(i guess we have drifted of into partly reorganizing the rtio-ttl registers)
<sb0>
the loopback test does more than testing the hardware... it also tests some of the interleaving, drivers, etc.
<rjo>
absolutely. i meant rtio2wb vs wb2rtio for debugging rtio-ad5370.
<sb0>
yes
<rjo>
ok. then this is also agreed, right? lets go for a slim wb2rtio wrapper that can be used at synthesis time.
<sb0>
yes
<rjo>
afaict wb2rtio would choose a suitable timestamp and push into the fifo for writes
<rjo>
for reads it would have a flag and a register with the data (stripping the timestamp?)
<sb0>
wb2rtio would give the same register-based interface as the current one
<rjo>
a readable flag
<sb0>
the cpu would be responsible for choosing the timestamp (e.g. by reading the current counter value and adding a margin)
<sb0>
basically when a channel is in debug mode the rtio core would hand over the associated control registers to the comm-CPU
<sb0>
(by "debug" I mean "override")
<rjo>
? you mean wb2rtio would consist of a timestamp register, a write register and a bunch of pseudo-mem-mapped registers?
<rjo>
isn't the current rtio interface already wishbone?
<sb0>
there will be no wb2rtio. just the possibility to access the FIFOs from the comm-CPU, and disable access to the same from the kernel-CPU.
<sb0>
memory-mapped CSRs on wishbone, yes
<rjo>
ok. yeah. i guess that is fine. doing the mapping in software is easier than in gateware.
<rjo>
and debug_rtio_write_soon(rtio_channel, fifo_data) would be it. and fifo_data the aforementioned (addr, data) or the like.
<rjo>
easier than the *memapped_wb2rtio_address = data with a complicated wb2rtio translator that decodes addresses, guesses timestamps etc in gateware.
<sb0>
yes
<sb0>
and we don't need performance for the debug interface
<whitequark>
sb0: i've just discovered that pep8 can't automatically fix violations
<whitequark>
see this is why tooling aware parsers are necessary :p
<whitequark>
neither seems pylint
<whitequark>
rubocop, ruby's pylint, specifically uses the rewrite module of my parser to fix these while not breaking the AST by accident
sturmflut-work has quit [Remote host closed the connection]
sturmflut-work has joined #m-labs
stekern has quit [Quit: Lost terminal]
stekern has joined #m-labs
sh[4]rm4 has joined #m-labs
sh4rm4 has quit [Ping timeout: 265 seconds]
fengling has quit [Quit: WeeChat 1.0]
kyak has quit [Ping timeout: 252 seconds]
kyak has joined #m-labs
kyak has joined #m-labs
kyak has quit [Ping timeout: 264 seconds]
kyak has joined #m-labs
kyak has joined #m-labs
kyak has quit [Changing host]
stekern has quit [Quit: Lost terminal]
stekern has joined #m-labs
<kristianpaul>
sb0: remenbers the parameter for not not optimize a design in ise/xst?
<kristianpaul>
hmm signal keep
<kristianpaul>
(* KEEP = "TRUE" *)
sb0 has quit [Read error: Connection reset by peer]
sh[4]rm4 has quit [Remote host closed the connection]
sh4rm4 has joined #m-labs
sb0 has joined #m-labs
Zougloub has quit [Ping timeout: 256 seconds]
<rjo>
sb0: yes. tmux was my "tool of the year 2014". 2013 was and 2015 will be vim ;)
digshadow-w has joined #m-labs
<digshadow-w>
I heard some physics folks hang out here, so...anyone have a user manual for a LeCroy 6010 magic controller (GPIB <=> CAMAC)? A friend is trying to get some instruments running but doesn't have the manual. I found something similar, but it would be good to have the actual user manual. I've collected what I have so far here: http://siliconpr0n.org/wiki/doku.php?id=lecroy:6010_magic_controller
sh[4]rm4 has joined #m-labs
sh4rm4 has quit [Ping timeout: 265 seconds]
<rjo>
digshadow-w: ha. got nothing here. nothing in the boat anchor manual archives?
<digshadow-w>
nada
rofl__ has joined #m-labs
sh[4]rm4 has quit [Ping timeout: 265 seconds]
sh[4]rm4 has joined #m-labs
rofl__ has quit [Ping timeout: 265 seconds]
Alain has quit [Remote host closed the connection]
sh[4]rm4 has quit [Remote host closed the connection]