cfbolz changed the topic of #pypy to: PyPy, the flexible snake (IRC logs: https://botbot.me/freenode/pypy/ ) | use cffi for calling C | mac OS and Fedora are not Windows
antocuni has quit [Ping timeout: 268 seconds]
marvin_ has quit [Remote host closed the connection]
marvin_ has joined #pypy
marvin_ has quit [Remote host closed the connection]
<cfbolz>
antocuni: there is still no GC statistics about how much *time* we spend in the GC, is there?
<antocuni>
no, we don't have any direct stat about that
<cfbolz>
antocuni: would it be ok to add that?
<antocuni>
you can compute it using PYPYLOG
<cfbolz>
yes, but that's from outside of the process
<antocuni>
cfbolz: what API do you have in mind?
<antocuni>
but in general yes, I think it would be nice
<cfbolz>
TOTAL_TIME_GC, simply
abrown has joined #pypy
<antocuni>
if you want to be advanced, you could add a field to the stats passed to the gc hooks
<cfbolz>
yes, that would be the plan
<antocuni>
I already thought about doing that; then I stopped because I didn't want to think what kind of clock use to compute the time
<antocuni>
time.time? read_timestamp? time.clock?
<cfbolz>
hah, I would just use time.time
<antocuni>
which is probably fine, apart that later you get different results if you examine PYPYLOG. But it's probably fine
<cfbolz>
yes
tayfun27 has quit [Ping timeout: 246 seconds]
tayfun26 has joined #pypy
tayfun27 has joined #pypy
tayfun26 has quit [Ping timeout: 268 seconds]
adamholmberg has quit [Ping timeout: 268 seconds]
Zaab1t has quit [Ping timeout: 246 seconds]
<abrown>
What are possible reasons for getting a VirtualizableArrayField exception?
<abrown>
VirtualizableArrayField: using virtualizable array in illegal way in <FunctionGraph of (src.environments.environment_without_display:51)Environment.__init__ at 0x7f6e8019a250>
marky1991_2 has quit [Read error: Connection reset by peer]
<abrown>
I have messed around with the constructor but I still seem to be getting the error; not actually sure why
marky1991_2 has joined #pypy
tayfun27 has quit [Remote host closed the connection]
arigato has quit [Quit: Leaving]
Zaab1t has joined #pypy
<abrown>
Looks like I can only use `_virtualizable_ = '...'` and not `_virtualizable_ = ['...', '...']`; is this correct?
<fijal>
abrown: number_of_names have to be constant, one way or another
<fijal>
jit-time constant
<fijal>
virtualizable[*] is a bit of a hack - it assumes the array has known size at jitting time and all the index accesses are constants
<simpson>
abrown: Can you refactor to avoid virtualizable entirely? You only need them when you're doing something like pausing an execution mid-frame.
<abrown>
Possibly; in my simplified example, I think I observed that removing the virtualizable annotation had little to no effect if I had everything else set up correctly (e.g. immutable)
<abrown>
fijal: number_of_names should be constant in the trace
<fijal>
abrown: you usually need to dig to a specific place in the pdb, see the graphs and see where is it exploding
<fijal>
virtualizables are not for the faint hearted
<abrown>
:) so I'm learning
<fijal>
does __locate__ always return a constant?
<fijal>
you can always promote the result, to make double sure it's a constant
<fijal>
in fact we usually have a setup where we don't unroll those functions - we just slap @jit.elidable on top and make sure we promote args
jcea has quit [Remote host closed the connection]
jcea has joined #pypy
<abrown>
fijal: can you explain a bit more what elidable does? it doesn't memoize or anything, right?
themsay has quit [Ping timeout: 250 seconds]
<abrown>
so the "promote args; elide some function" thing only is a benefit if we call the function more than once in the trace?
<Alex_Gaynor>
abrown: elidable means "the JIT may always replace f(*args) with the result if args are the same"
<Alex_Gaynor>
abrown: this means a) if you have f(*args), f(*args) it'll remove the second call and reuse the result, b) if the JIT sees f(*constants) it'll remove the call entirely and replace it with the result from tracing
<Alex_Gaynor>
it's very close in definition to "pure", but elidable is allowed to do unpure things like caching and whatever in the background, as long as it's safe if they dont' happen
<fijal>
what he said, listen to him
<abrown>
:)
<abrown>
I need to think about this some more; at one point I added @elidable to __locate__ based on a very hazy notion of what Alex clearly stated
<abrown>
but performance was worse so I went with @unroll_safe
<Alex_Gaynor>
elidable functions will not be inlined by the JIT. If you have a function which is "sometimes called with all constants and should be treated as elidable, but sometimes we want to inline it" then you need to play clever tricks
<fijal>
yes, but just @elidable is not good enough
<fijal>
you want this to be always called with constant arguments (which might not be possible, because self!), so you need to do clever tricks
<abrown>
awesome
<fijal>
Environment is a frame right?
<abrown>
what I'm thinking about is that the trace of __locate__ should always be the same for the same arguments but the object retrieved may be different as the interpreter changes the environment
<abrown>
yes, basically
<abrown>
it is for a let-binding
<fijal>
ok, so the Environment can change, yes
<abrown>
yup
<fijal>
but the *shape* of the environment should not, right?
<abrown>
correct
<abrown>
it should always have the same number of bindings at each level
<simpson>
Huh. I know that you can assert a name to be green, but you can't check conditionally whether it's green, right?
<fijal>
simpson: you can, for even more clever tricks
<fijal>
abrown: right, so __locate__ should probably return you a number
<fijal>
and should only depend on some objects that are genuinely constant
<fijal>
so Environment should have a Code object, which is constant
<fijal>
you need to split the concepts, essentially
<fijal>
pypy is quite magic, but it won't do *all* the work for you
<abrown>
fijal: __locate__ returns the level the binding is in and the index inside that level
<abrown>
the trace of finding that tuple (level, index) should always be the same...
<fijal>
yes, so you can write it in a way that it really walks constant objects
<fijal>
... and constant fold away everything
<abrown>
yes; so it can't fold the environment levels away?
<fijal>
no, it cannot because the pointer to Environment is not constant
<fijal>
so it gets lost in one way or another
<abrown>
hm
<abrown>
ok, in a previous iteration of this I had an object that wrapped all of this and maintained a pointer to it that never changed
<abrown>
but I ended up running into basically the same situation I am in now: __locate__ never was really walking constant objects because the program flow changed the hierarchy of levels
<abrown>
(I think)
<abrown>
but if I understand you correctly, if my immutable AST node had the environment level assigned to it then this could get constant-folded away (i.e. the code object you mentioned above)
<simpson>
Yes. This sort of annotation/bookkeeping stuff, if you can make it immutable and add it to your immutable AST, should smoothly be removed by the JIT.
<fijal>
abrown: you might need to add a few @jit.promote in the things too
<fijal>
we ended up having a few unnecessary ones in pypy too - something that's known to be constant and I can tell you exactly which constant, but it's easier to just promote it being read from somewhere than try to explain to the compiler
<abrown>
so if I make this change, will I even need @elidable? shouldn't the JIT just inline and constant-fold the operations in __locate__ away?
<fijal>
maybe, maybe not
<abrown>
ok, I'll play around with it
<fijal>
but @elidable makes it much much easier for JIT to do that - it just removes the call
<fijal>
as opposed to trying to optimize the shit out of it (which e.g. runs into the trace limit and time it takes etc.)
<abrown>
:)
<abrown>
as a side note: why do I sometimes need to use @unroll_safe?
<abrown>
why aren't all these functions inlined by the JIT?
<simpson>
The JIT avoids loops by default. At translation time, the JIT sees where loops are, and by default it will assume that loops are unsafe to trace. (Because tracing a non-constant loop is really bad in terms of how much code is generated and how often the guards will fail.)
<abrown>
ok, that makes sense
<fijal>
yes, as simpson said, this seems to be a sensible strategy
<fijal>
inline by default things with no loops and don't inline things with loops
<fijal>
you can control both with @jit.unroll_safe and @jit.dont_look_inside
<abrown>
what about recursive function calls? what does it do about that?
<simpson>
I don't understand the details. cfbolz has assured me that recursion is fine; the JIT will understand that it's entering itself again and set up a bridge and all that.
<simpson>
I don't know exactly what happens to the RPython/C stack frame, but otherwise you get a loop.
<abrown>
yeah, I had some issue with a jitdriver being recursive and came to a similar conclusion; but I mean with the inlining... like, the JIT can't perpetually inline recursive calls, right? or at least it should be scared of doing that?
Rhy0lite has joined #pypy
<fijal>
recursion is a bit bad
<fijal>
there are rules specifically for inlining the portal
<fijal>
but it will eventually mark something as non-inlinable and trace from start
<cfbolz>
abrown: yes, I am not at all sure what happens if you have a recursive rpython level function
<abrown>
ok, well I will try to avoid recursion on the RPython then
<abrown>
*the RPython side then
<cfbolz>
abrown: recursion in the interpreter to implement recursion in the language is fine though
<abrown>
simpson, fijal: thanks so much for the discussion; it was very helpful
<cfbolz>
abrown: I forgot again, what kind of language are you implementing?
<abrown>
Tiger
<fijal>
cfbolz: I kinda wish we had done better job with RPython
<fijal>
yay, hindsight!
<abrown>
it has let-bindings but is pretty simple
<cfbolz>
fijal: all of us wish that, yes ;-)
<simpson>
cfbolz: So far, in Monte, we do this on our ASTs which are green and we get a full unroll. I actually have been thinking of getting even faster, by visiting the AST and writing out a fully-runnable immutable object tree into the heap. But the JIT hasn't minded yet.
<abrown>
one thing you guys might be interested in: as I'm writing this paper I wrote up an abstract model of how I think the meta-interpretation works; is anyone interested in looking at it?
<fijal>
abrown: sure
<fijal>
I'm 80% convinced cfbolz will be too
<simpson>
Sounds interesting.
<abrown>
ok, let me clean it up and I'll post it here later
<abrown>
i've found it difficult to explain/discuss this stuff with others so I tried to explain just the key parts of it--still too complex but I'd be interested in your feedback
<cfbolz>
abrown: sounds cool. I tried something similar in my thesis, if you are interested
<simpson>
No worries. It's tough to grok partial evaluation. It's even tougher for somebody who understands to teach others how to grok. But the good news is that, if you can teach somebody how to teach how to teach, then you're done! There are only three~
* simpson
not marking jokes well
<abrown>
:)
<cfbolz>
abrown: where are you going to submit the paper to?
<abrown>
ha... to my advisors! hopefully they don't skewer it
<abrown>
it doesn't really seem "interesting" enough for a journal/conference
<abrown>
Tiger is sort of a toy language
<simpson>
I know that feeling. But I think that it's interesting nonetheless; we could stand to have more work done on toy languages, and this is more interesting than Yet Another Brainfuck Interpreter.
<simpson>
(Although Brainfuck's a fascinating toy for talking about certain kinds of optimizations.)
<fijal>
abrown: I don't know, it's a toy language but a small fast interpreter is something
<abrown>
well, eventually I'll post it here and you guys can see whether it's submittable
<abrown>
it's been a weird experience building this thing and writing about: half of it completely frustrating and hopeless and half of it super rewarding
jcea has quit [Remote host closed the connection]
jcea has joined #pypy
<cfbolz>
abrown: RPython in a nutshell 😜
marky1991_2 has quit [Read error: Connection reset by peer]
marky1991_2 has joined #pypy
jcea has quit [Remote host closed the connection]
marky1991_3 has joined #pypy
mjacob has quit [Ping timeout: 250 seconds]
marky1991_2 has quit [Ping timeout: 240 seconds]
mjacob has joined #pypy
antocuni has quit [Ping timeout: 252 seconds]
jcea has joined #pypy
Zaab1t has quit [Quit: bye bye friends]
jcea has quit [Remote host closed the connection]
jcea has joined #pypy
themsay has joined #pypy
themsay has quit [Ping timeout: 245 seconds]
antocuni has joined #pypy
Rhy0lite has quit [Quit: Leaving]
energizer has quit [Quit: ZNC 1.7.0+deb0+xenial1 - https://znc.in]
energizer has joined #pypy
energizer_ has joined #pypy
energizer_ has quit [Client Quit]
jamesaxl has quit [Quit: WeeChat 2.2]
themsay has joined #pypy
themsay has quit [Read error: Connection reset by peer]
themsay has joined #pypy
themsay has quit [Read error: Connection reset by peer]
themsay has joined #pypy
marky1991_3 has quit [Read error: Connection reset by peer]