<njs>
cfbolz: unfortunately I am not sure how to fix it :-)
<njs>
I feel like I must be cursed or something, if I'm counting right this is the 11th novel cpython bug I've hit in the last ~4 months, and the 4th pypy bug
<njs>
or no, I forgot; 2 of the cpython bugs turned out to be already known. but wtf.
<njs>
I wonder if we can make f_locals/locals() be a magic dict whose __setitem__ writes back to the fast locals array, and get rid of locals2fast altogether
<njs>
well, aside from initialization for eval/exec with a given locals dict
asmeurer__ has joined #pypy
rokujyouhitoma has joined #pypy
rokujyouhitoma has quit [Ping timeout: 268 seconds]
<njs>
I'm not actually sure why the trace function machinery even does this. It appears to have been added by Guido in 1993 without any relevant comment.
<njs>
maybe pdb wants to mutate f_locals from inside its trace function or something?
<arigato>
yes
<arigato>
it's for the case where you want to change a variable's value from inside pdb
<arigato>
it doesn't really make sense in the LocalsToFrame direction, though
rokujyouhitoma has joined #pypy
<arigato>
at least in python 2 you can't change a value of an outer scope variable at all, so why you'd want to do that from pdb, I don't know
<njs>
arigato: I guess the reasoning is, if you want pdb's changes to stick then you need to do FastToLocals, and if you want to do FastToLocals then you need to make sure that you're not using some stale values as input, so you have to do LocalsToFast first?
<njs>
oh, yeah, including closure variables in this is also dubious
<arigato>
I see why you'd include them in FastToLocals, otherwise the closure variables are not visible from pdb
<njs>
but especially in python 3 with nonlocal, I guess you don't have many options. It would be very weird if a local variable disappeared from function's locals() because a *nested* function closed over it
<njs>
arigato: what do you think of my suggestion on the bpo bug
rokujyouhitoma has quit [Read error: Connection reset by peer]
<njs>
oh huh
* njs
goes searching through his tabs
rokujyouhitoma has joined #pypy
<njs>
weird, it seems to have disappeared. Well, I will try again :-)
<njs>
(the suggestion is to get rid of LocalsToFast, and make locals() and frame.f_locals return a dict proxy whose __setitem__ writes back to the fast array)
<arigato>
I guess you got a conflict because I posted the generator example at the same time, or something
<arigato>
that can't easily be back-ported... there might be code out there that assumes frame.locals() returns a real dict
<arigato>
or even, more precisely, a *copy* of the frame's locals, at least for functions
<njs>
it's true, it fixes some semantically-visible bugs
<njs>
blah
<njs>
who cares about python 2 anyway
rokujyouhitoma has quit [Remote host closed the connection]
<arigato>
we're in the annoying position where, yes, it looks like a clean-up for 3.7 but it can't be applied because ideally we need a fix for the previous versions too
<arigato>
the problem also applies to 3.6, 3.5, etc
<njs>
bracket trace function calls in PyGILState_AcquireAndRefuseToLetGo()
<arigato>
:-)
<arigato>
it doesn't help my generator example
<njs>
though that doesn't fix your generator example
<njs>
but the bug has been around for at least 16 years and the generator version requires some very careful hoop-jumping to run into, so maybe it doesn't matter if a backport doesn't fix it
<njs>
not that PyGILState_AcquireAndRefuseToLetGo() is a piece of API that should ever exist anyway though
<njs>
actually, no, I guess the generator version has only been around 15.5 years, since that's when generators were added :-)
<njs>
technically locals() and f_locals are already allowed to return any random mapping object, but I guess that doesn't really help for backport purposes because in practice they rarely do
<arigato>
yes, PyGILState_AcquireAndRefuseToLetGo() would be a completely disruptive change: it would mean that when you're at a pdb prompt, other threads no longer run
<njs>
side effect of this bug: I'm suddenly horrified to realize how pdb.set_trace() must actually work, though OTOH now the name actually makes sense for the first time
<njs>
okay, more seriously: for backport purposes I can't think of anything that would work better than to have call_trampoline make a shadow copy of the locals, and then do PyFrame_LocalsToFastButOnlyIfModified
<njs>
this is a bit obnoxious because it slows down trace calls, but I guess you could optimize it down to basically a strdup of the fast array
<arigato>
but the point is to check the variables that are *not* in the fast array
<njs>
oh right :-)
<arigato>
so the shallow copy could include only the closed-over variables
<arigato>
then I wouldn't worry about slow-downs
<njs>
so yeah, you'd have to unpack the cells, but right, most of the time there aren't any
<arigato>
anyway, sys.settrace() is badly broken, performance-wise
<arigato>
it behaves in a way that is slower and slower the larger the function is
<arigato>
i.e. it doesn't have constant complexity per bytecode evaluated, but linear in the total number of bytecodes in the function
<njs>
ugh, because of lnotab decoding or something?
<arigato>
yes
<cfbolz>
This is all intensely frustrating
<arigato>
(I had to fix that in the revdb branch of pypy by introducing my own decoding with a different internal API)
<njs>
it's unfortunate that coverage + pypy is like a perfect storm of slowdowns
<arigato>
I think that coverage suffers from the problem even on CPython where it is written in C
<arigato>
I'm not 100% sure but 99% :-)
<njs>
arigato: right, but then on pypy you lose the C accelerator and you lose the JIT, usually the one kind of balances out the other :-)
<njs>
cfbolz: ?
<cfbolz>
No
<cfbolz>
You don't lose the JIT
<arigato>
(try to time coverage running a loop inside a big function, I guess it is slower if the loop is *later* in the function)
<njs>
huh, really?
<cfbolz>
The JIT actually helps coverage a lot
<cfbolz>
But the relative performance is still a lot worse than without coverage
<njs>
I see, right, so it's still JITting the code that's actually being run, but now that code includes all these calls to coverage at awkward places, which prevents all kinds of optimizations that might apply to the code without coverage?
<arigato>
no, pypy's jit is really not easily disabled
<arigato>
it is jitting and *inlining* the coverage logic at every line
<cfbolz>
arigato: I think the coverage trace function might be too large or have a loop
<arigato>
it just sees the calls to coverage as regular python calls
<cfbolz>
But otherwise yes
<cfbolz>
What I find intensely frustrating is that we have a fundamentally and conceptually broken API that we can't fix for backwards compatibility reasons
<arigato>
yes
<arigato>
Python 4, etc.
<cfbolz>
Yes that's the thing that makes it worse. This is clearly something that should have been fixed in Python 3
<njs>
the whole fast locals thing is pretty frustrating all around
rokujyouhitoma has joined #pypy
<njs>
but I think making locals()/f_locals properly mutable is something that could reasonably be done in a 3.x release? I dunno, like most of these things there's this weird path dependence where someone have slept bad and kick up a fuss and then it becomes impossible, but...
<arigato>
from one point of view, Python should have cleaned that up long ago, by forgetting how it is implemented and designing something sane for the user and only then reimplementing that
<arigato>
yes, I'm happy not to get involved in Python "language" discussions
<njs>
(c)python's whole design philosophy is to optimize for usual case usability + straightforward implementation and nothing else, though
<arigato>
yes, not saying it's a bad idea, except maybe in some very specific cases like this one imho :-)
<arigato>
(did you know that, of course, the python 3.5 async logic gives strange results with sys.settrace, and now I bet we couldn't easily fix that without breaking backward compatibility..?)
<cfbolz>
It's one of the cases where listening to VM people early would have been very beneficial, Self had this fixed before CPython existed
<njs>
uh... what does async do to sys.settrace?
<arigato>
I think that when there is an exception, you don't see all intermediate frames being returned from
<arigato>
or maybe you do, but you don't in another case
<arigato>
something like that, I can dig again if you like
<njs>
arigato: are you thinking of the weirdness around .throw(), where stack traces are wrong?
<arigato>
yes
<njs>
cfbolz: I'm curious, is there another trick for doing local variables that has similar speed+simplicity to what cpython does?
<arigato>
(I don't remember when, but indeed stack traces are wrong in half the cases, missing either entering or leaving the frames)
<cfbolz>
njs: the trick is not in local vars but in how to allow reflective access
<arigato>
cpython has no simplicity left, by now :-)
<njs>
arigato: I actually suggested a trick to yury for fixing this and deleting a bunch of that code at the same time, perhaps it will happen
<arigato>
njs: ah, cool
rokujyouhitoma has quit [Ping timeout: 260 seconds]
<cfbolz>
njs: basically 'mirrors' is the answer. But Googling for that is impossible, so I'll give you a link in a minute
<njs>
cfbolz: is it something different than providing a duck-dict that's back by the high speed storage underneath?
<njs>
*backed
<cfbolz>
(indeed typing 'mirror self' into Google is really not going to get you anything about VMs)
<cfbolz>
njs: it's more of a principled approach to reflective API design
<njs>
'mirror self vm' seems to work pretty well though
<cfbolz>
njs: but yes, implementation wise it boils down to a dict proxy
<njs>
it's probably impossible to make fast locals totally transparent in cpython, because eval and exec take an arbitrary mapping to use as the locals object, and it will inevitably drift out of sync
<njs>
unless you use some extra clever dict strategy thing, but I can see how the cpython devs might reasonably not want to do that
<njs>
but a dict proxy sure would make all the other cases less confusing
<arigato>
njs: note that exec/eval accept dict proxies as locals
<njs>
arigato: I know, I mentioned it two lines above yours :-) what case are you thinking about?
<arigato>
...I guess I don't quite understand the three lines I tried to answer
<njs>
hmm
<njs>
what I was saying was, if you pass in some random mapping object to exec or eval, and you expect that at any moment you can compare the locals as seen inside the function to the values inside that mapping when accessing the mapping directly and they will match, then you are going to be disappointed
<njs>
but we could make locals() and f_locals be such that they always show the values of either the fast array or the mapping object (pick one...), and that assignments through them always go to both
<njs>
so even if we can't 100% maintain the illusion that the locals namespace actually is a regular mapping object, we sure could make it *more* convincing than it is currently
<njs>
was that clearer?
<arigato>
I think that ``exec("code", glob, loc)`` will compile "code" in such a way that ``loc`` will always be up-to-date and valid
<arigato>
I'm not sure what occurs with ``exec(func.__code__, glob, loc)`` though
<njs>
well, func.__code__ is weird, it has a magic flag set that makes it allocate a new locals dict!
tormoz has quit [Remote host closed the connection]
<arigato>
it's only in functions, i.e. code objects with CO_OPTIMIZED, that things get messy
<njs>
hmm, I guess you're right, you can't get the FAST bytecodes outside functions, and inside functions the locals is always a real dict that python allocates for you
<arigato>
or, more commonly, does *not* need to allocate at all
<arigato>
yes, sorry, the point was that the locals is a lazily-allocated but real dict
<njs>
... now I am nervous that that horrible hack I just linked actually breaks because CPython skips calling FastToLocals when tearing down the frame or something
<arigato>
that definitely looks like a horrible hack that depends on how exactly it works on CPython
<njs>
yeah, the previous version even used ctypes. unfortunately there really are no good ways to do what that wants to do yet (basically compile mode "single_with_async"). another 3.7 feature request.
<arigato>
ah
<arigato>
maybe a different approach: stick the code in a function, compile it, then look at co.co_varnames, add "global x" for all varnames, and re-compile?
<njs>
anyway, you're right, if we ignore mostrosities like the thing I just linked, then perhaps a dict proxy approach actually would be enough to make it look like locals was really a dict in all cases
<arigato>
(sorry, that's co.co_names)
<arigato>
(or no, that's really co.co_varnames)
<njs>
yeah, I forget why we didn't do that. (or the equivalent via walking the AST.) there were a number of weird options explored there. (my favorite: inject the ambient namespace into the function by automatically adding every defined variable as a kwarg)
<arigato>
ouch, yes
<arigato>
you can't reflect changes into the outside namespace, though
<njs>
you make the function return locals(), and then call outside_namespace.update(ridiculous_func(**outside_namespace))
<arigato>
"ah"
<njs>
but that doesn't really work because any functions defined inside the code you compile end up capturing the temporary locals
<njs>
ohhh, ah-hah, there's another part of this trick: we also inject a call to locals() at the end of the function, to trigger an implicit FastToLocals :-)
<arigato>
fwiw, I'd have tried to stay clear of this mess and use the syntactic tricks only, but maybe I'm missing what wouldn't work
<arigato>
you need "global x" for all x in co.co_varnames + co.co_cellvars
<arigato>
and no locals() at all
<njs>
arigato: I guess one (possibly minor?) downside of this is that you erase the locals/globals distinction
<arigato>
ah, sorry, I thought it was the goal
<njs>
arigato: the goal is to take a line like 'await ...' at the REPL and make it execute like any other expression at the REPL
<arigato>
yes, and regular REPL expressions don't have their own locals at all
<njs>
though eh. in ipython globals() and locals() are different, but in the built-in repl they're the same.
<njs>
so maybe the distinction isn't so important.
<arigato>
ah ok
<njs>
actually, I'm confused. Looking at the ipython source, I see that it ultimately does 'exec(code_obj, self.user_global_ns, self.user_ns)', but nonetheless if I actually type 'locals() is globals()' then it returns True.
<arigato>
yes, I see the problem now :-)
<arigato>
more generally, if you want to use exec/eval with a dict proxy as locals, you need a non-function, and a non-function cannot use "await" at all
<njs>
yeah, basically
<arigato>
yay, combining two different messy areas only creates more messes, surprise
<njs>
like I said, the obvious solution is that we need a compile mode that's like "single" but with async context, but since that's at least a year away even for cpython we have to make do...
<arigato>
I see
<njs>
anyway that project is somewhat stalled on the much messier problem of, how do you meaningfully execute some async code deep inside ipython, when everything else around it is synchronous :-)
<arigato>
actually, unsure I see what "await f()" would mean in an interactive prompt
<arigato>
ah, maybe simply "switch to the main loop that is somewhere else, do background stuff, and when we come back, then come back to the REPL"
<njs>
arigato: yeah, it needs an event loop running somewhere
<arigato>
I see the point, while continuing to fear it's just one more example of: hey I need to pass "async" through place X, can you add support in the language? for each possible X that can possibly end up calling more Python code
<njs>
yeah, that's kind of how it works...
<arigato>
(the obvious example is: I want "a + b" to call __add__ but async...)
jamesaxl has quit [Read error: Connection reset by peer]
jamesaxl has joined #pypy
rokujyouhitoma has joined #pypy
vkirilichev has joined #pypy
rokujyouhitoma has quit [Ping timeout: 246 seconds]
rokujyouhitoma has joined #pypy
<kenaan>
arigo default b9047ff141b7 /rpython/rlib/test/test_rposix.py: Finally understood why these two tests fails on 32-bit. Skip them as "not really worth worrying about".
<ghosh>
Hi ... i just pushed a package to the pypi test server.. However on running, "pip install -i https://testpypi.python.org/pypi PACKAGENAME", the setup fails and says cannot find requirements.txt. My setup file reads the requirements from requirements.txt and dumps it in install_requires. How do i add requirements.txt to my final package.
rokujyouhitoma has quit [Remote host closed the connection]