<cfbolz>
that should make linux64 on trunk green again
<arigato>
looking at some of the pypyjit/test_pypy_c failures on py3.6...
<arigato>
I think the new ones are probably just of the kind "we no longer produce the dummy getfield_gc_r which was anyway removed by the jit backend because its result is never used"
<cfbolz>
arigato: yes, but not all
<cfbolz>
arigato: eg the max one seems to be "range now returns an iterator, so stuff happens differently"
<arigato>
yes, I'm talking about the failures newer than the 11th feb
<cfbolz>
ah, ok
<cfbolz>
sorry
<arigato>
of course it doesn't mean I should ignore the other failures :-)
<cfbolz>
:-)
<cfbolz>
I can try to fix some too
<arigato>
cool
<kenaan>
arigo py3.6 1a39bb272421 /pypy/module/pypyjit/test_pypy_c/test_instance.py: We no longer get the dummy getfield_gc_r
Zaab1t has joined #pypy
<kenaan>
arigo py3.6 c2f51a1cf58c /pypy/module/pypyjit/test_pypy_c/test_string.py: Fix for test_decode_ascii after the utf8 merge
<mattip>
(it is something used to micro-profile changes by rhettinger, recently came up on the cython mailing list)
<cfbolz>
it's a bad benchmark, imo
<mattip>
what would be better? They are measuring exactly what they want to measure for their purposes
<kenaan>
arigo py3.6 53322e432860 /pypy/module/pypyjit/test_pypy_c/: The dummy_get_utf8? line is now not needed any more. Keep it around in the tests for future reference just in case w...
<cfbolz>
mattip: yes, sorry, it's useful in exactly the context of cpython
<cfbolz>
even for cython it's starting to be misleading
<arigato>
so the problem is that with the old BUILD_MAP, it would see the keys are completely constant, but not with BUILD_CONST_KEY_MAP?
<arigato>
right, the first line in the opcode is space.fixedview(self.popvalue())
<cfbolz>
yes
<cfbolz>
I am not quite sure why the immutable[*]ness of the tuple content is lost
<cfbolz>
we would fix this by using space.getitem, in any case
<cfbolz>
arigato: should I?
<arigato>
ah no
<arigato>
space.fixedview on a tuple should just be "return self.wrappeditems", seen by the jit
<cfbolz>
it does
<arigato>
ah but yes, this looses the info in the class W_TupleObject about _immutable_fields_ = ['wrappeditems[*]']
<cfbolz>
yes, because it gets passed through fixedview, which doesn't always return an immutable list
<cfbolz>
bit annoying
<arigato>
yes
<cfbolz>
arigato: so, space.getitem?
<arigato>
yes, this would fix it here
<arigato>
ideally we should fix it for all space.fixedview() calls...
<cfbolz>
arigato: I can't think of an easy general fix. we would have to track immutableness dynamically in the jit
tazle_ has quit [Quit: leaving]
tazle has joined #pypy
<arigato>
which path of fixedview() can return a mutable but fixed-size list?
<cfbolz>
arigato: fixedview works on applevel lists (makes a copy)
<arigato>
yes, which means fixedview itself returns a never-mutated rpython list, no?
<cfbolz>
hm
<cfbolz>
arigato: unless one caller of fixedview mutates stuff, which would be a problem anyway
<cfbolz>
arigato: do we have a make_sure_not_mutated thing?
<arigato>
ah right
<arigato>
but likely, _unpackiterable_known_length_jitlook() for example fills a list from [None]*length, item by item
<arigato>
this means the annotator can't figure out that the result is not mutated any more
<arigato>
I think
<cfbolz>
plausible
<arigato>
no, there is no make_sure_not_mutated()
<cfbolz>
so we are a bit stuck
<arigato>
maybe we could have such a function, which would be annotated like a "lst[:]" and return a new list object, but be rtyped as a no-op
<arigato>
so that the annotation "not mutable" only propagates to the result, not to the argument
<cfbolz>
right
<cfbolz>
anyway, this is a bit cannons and sparrows (is that a thing in english?)
<arigato>
yeah
<arigato>
it's true that we might have fixedview() used a lot but probably not often on tuples that are constant
<cfbolz>
right
<arigato>
at this point you could even rewrite the BUILD_CONST_KEY_MAP with "assert isinstance(w_tup, W_AbstractTupleObject)" and call w_tup.getitem(i) directly, maybe
<cfbolz>
since nowadays we don't care about objspace separation much anymore anyway?
<cfbolz>
(I tend to agree, fwiw)
<cfbolz>
ok, will do it
lritter has joined #pypy
Zaab1t has quit [Quit: bye bye friends]
<arigato>
cool
<cfbolz>
arigato: I guess I should use space.interp_w instead of assert, to fail nicely for the people that build weird code objects
<kenaan>
cfbolz py3.6 2e21aa817a15 /pypy/interpreter/pyopcode.py: make BUILD_CONST_KEY_MAP JIT-friendly
<arigato>
right
<kenaan>
cfbolz py3.6 9f1fd685935c /pypy/module/pypyjit/test_pypy_c/test_misc.py: this is an improvement
<cfbolz>
arigato: I am doing the cffi jit test next
<cfbolz>
it's not about cffi
<arigato>
I'm not sure I understand this test_ffi
<arigato>
bdbe0dc2c7be sounds suspicious
<arigato>
though of course the problem might be more than I was not available for comments at this time
oberstet has quit [Quit: Leaving]
<cfbolz>
arigato: ah, I see
<cfbolz>
k, so we are already substantially different than default :-(
<cfbolz>
then I am maybe not going to look at it now ;-)
xorAxAx has quit [Ping timeout: 240 seconds]
xorAxAx has joined #pypy
<cfbolz>
arigato: I see a lot of this: +4397: p339 = getfield_gc_r(ConstPtr(ptr338), descr=<FieldP pypy.objspace.std.unicodeobject.W_UnicodeObject.inst__utf8 8>)
<cfbolz>
:-(
<arigato>
still?
<arigato>
the question is if the p339 is used anywhere
<arigato>
if it isn't, then you should see that this line is +4397 and the next line also +4397
<arigato>
i.e. no code was generated
<cfbolz>
sorry, sorry, false alarme
<cfbolz>
was using the wrong executable
<cfbolz>
phew
<arigato>
:-)
<arigato>
I've tracked down the extra mess shown in bdbe0dc2c7be, to the branch memory-accounting
<arigato>
which adds a "can do anything at all" logic in backendopt/writeanalyze for gc_add_memory_pressure
<cfbolz>
ah
<cfbolz>
is that needed?
<arigato>
I'm still trying to understand the message of 4038d01ba9e9
<cfbolz>
yes
<arigato>
OK I may have understood it as meaning "the problem is that when we allocate an object, we initialize the field 'special_memory_pressure' to zero; but later we might call gc_add_memory_pressure, which would put a non-zero value; without the change in 4038d01ba9e9, the initialization to zero might be moved after the gc_add_memory_pressure"
<cfbolz>
ok, but then saying "it can mutate anything" is a bit extreme, also I would have liked a unit test really :-(
<arigato>
yes, I'm still running the jit tests to see if any fails
<arigato>
I think that's why the test_pypy_c shows all these getfield(const)/guard_value again
<arigato>
because we added a gc_add_memory_pressure in the loop
<cfbolz>
arigato: you are talking about test_ffi right?
<cfbolz>
or more generally
<arigato>
yes, test_ffi
<cfbolz>
yes, plausible
* cfbolz
is digging trough the new importing and space bootstrapping instead
<arigato>
the whole missing optimization is only to support the gc 'get_stats(TOTAL_MEMORY_PRESSURE)'
<arigato>
or the whole branch, as far as I can tell
<cfbolz>
arigato: right
<arigato>
maybe we can ask around if someone ever used one particular line of what gc.get_stats() prints?
dddddd has joined #pypy
<cfbolz>
arigato: is that only about stats?
<arigato>
also I'm not too confident about (i.e. there are no tests for) the fact that we don't put an extra field 'special_memory_pressure' on W_Root, which would be globally bad
<cfbolz>
Yes, why is there no _attrs_ on W_Root any more
<arigato>
no, I mean, it's a field added by rtyping independently of any _attrs_
<cfbolz>
Hrm
<arigato>
(there is __slots__ on W_Root)
<Ninpo>
Pleased to report no performance regressions on my tool between py3.5 7.1 alpha and py3.6 7.1 alpha...and my now native async generator and async list comprehensions work just fine
<Ninpo>
:bigthumbsup:
<cfbolz>
arigato: I forgot, does the annotator look at slots?
<arigato>
cfbolz: yes
<cfbolz>
OK
<mattip>
Ninpo: it would be nice to write up your experience in a blog post
<Ninpo>
mattip: I intend to, however I've more pressing things on my list atm. As soon as I get the opportunity I will
<Ninpo>
arigato: aww, looks like that mysql-cffi thing was intended but never done. no commits, no branches, just the initial fork
<Ninpo>
Nothing done toward the cffi thing
<mjacob>
now that default is merged directly from default into py3.6, we should close py3.5, right?
<kenaan>
stian py3-bigint-to-int 4e3d4c68bcde /pypy/objspace/std/: Reduce this to int_mod and int_and ops for now.
_stian has joined #pypy
_stian has left #pypy [#pypy]
__stian has joined #pypy
<cfbolz>
__stian: hey :-)
<ronan>
huh, 1d688af446c2 prevents -A cpyext tests from running on CPython?!
<__stian>
cfbolz: Hey, long time
<ronan>
mjacob: yes, I guess, though we might want to leave it open in case we want to backport bug fixes
siddhesh has joined #pypy
<mjacob>
ronan: shouldn’t we use the release branch for it?
<ronan>
mjacob: yes, that makes more sense
<siddhesh>
Hello, I'm looking to set up a buildbot slave on an aarch64 vm. It will run only the own and rpython builders now (I reckon these do not need a functional JIT) and once the arm64 port is complete, I'll add the JIT builder too. Who can I send the settings to add to the master config so that my slave can login? Thanks!
<siddhesh>
I'll also send out a patch to add the remaining aarch64 slave and builder settings to the buildbot.
<cfbolz>
siddhesh: talk to fijal and maybe mattip
<fijal>
siddhesh: hey, I'm doing the arm64 port, but not the buildbot, ask mattip
Zaab1t has joined #pypy
<fijal>
siddhesh: ok, I'll do it
<siddhesh>
cfbolz, thanks
<fijal>
siddhesh: I think it's a bit early to run a buildbot though (it'll mostly just fail)
<siddhesh>
fijal, I was thinking of running it off trunk, not the WIP branch. Just to get that task out of the way for me.
<fijal>
that works I guess
<fijal>
let's setup a buildbot that will not run automatically
<siddhesh>
fijal, OK, so only under the ForceScheduler; I'll update my patch.
<fijal>
siddhesh: can we deal with it tomorrow or Wednesday though?
<siddhesh>
Should I send you an email with the slavename/password?
<siddhesh>
fijal, oh sure, that works for me.
<fijal>
please, fijall@gmail.com
<siddhesh>
No rush at all; thanks!
<mattip>
siddhesh: would it be possible to set up a 32bit chroot build slave as well?
<siddhesh>
mattip, I'll have to check if the CPU I have access to can do aarch32. If it can, I'll try to set one up.
<ronan>
arigato: what's the point of spaceconfig["objspace.std.withspecialisedtuple"] = True in 1d688af446c2?
<ronan>
it seems it's not required
<fijal>
siddhesh: I think it's enough to set up 32bit chroot on arm64?
<siddhesh>
fijal, Not all aarch64 cpus support the aarch32 instructions. I think only some of the stock cpus have that support.
<siddhesh>
I believe the a53 and a72 have them. I'll just have to double check to see if the core I'm running on has the support. If we do then we can do this no problem.
<siddhesh>
A bunch of the custom cores (falkor, thunderx2, etc.) don't support the 32-bit ISA at all.
<siddhesh>
ok, it's a cortex-a72. I think I can set up a 32-bit chroot on it.
kanaka has joined #pypy
kanaka has quit [Changing host]
kanaka has joined #pypy
* siddhesh
out for the day o/
siddhesh has quit [Remote host closed the connection]
<arigato>
ronan: spaceconfig["objspace.std.withspecialisedtuple"] = True is required because by adding it a test was failing
<arigato>
and it was a real failure
<arigato>
in general it's a good idea to add it because it makes more kinds of tuples available for testing parts of tupleobject.py and sequenceobject.py
<ronan>
well, it causes all -A tests to be skipped on CPython...
<arigato>
"ah"
<arigato>
sorry about that
<arigato>
unsure what to do
<ronan>
it's probably fixable, but test support code is a mess
jacob22 has quit [Read error: Connection reset by peer]
<cfbolz>
Yes, hidehiko has also supervised the work on the RPython erlang that existed a while ago
marky1991 has joined #pypy
marky1991 has quit [Ping timeout: 255 seconds]
<ronan>
pfff... cpyext test break if more than one space object is used
<mattip>
ronan: I consider it somewhat a miracle that you can run -A tests on cpyext, but it is a very useful miracle
<mattip>
and pretty amazing that you can test untranslated mixing C
Taggnostr has quit [Remote host closed the connection]
Taggnostr has joined #pypy
<fijal>
mattip: I keep being amazed by the JIT backend tests too
Zaab1t has quit [Quit: bye bye friends]
Gremico has joined #pypy
<Gremico>
I am using compile(tmpdir="tests") to try and have cffi compile the code into a folder, however when i do this i get a file not found error. Am i missing something?
<Gremico>
I found a solution by specifying a relative path in my set_source.
fling has quit [Ping timeout: 258 seconds]
lritter has quit [Ping timeout: 268 seconds]
<Ninpo>
How do I fix cffi putting compiled libs into project_root/src/lib.so instead of project_root/src/package_name/lib.so ?
<arigato>
Ninpo: use ffibuilder.set_source('package_name.lib', ...)
<Ninpo>
oh in the script I import in setup.py?
<arigato>
wherever you use ffi{builder}.set_source() right now?
<arigato>
I hope you're not using ffi.verify() because that's deprecated since ages
<Ninpo>
bang thanks arigato
<Ninpo>
No I'm not
<Ninpo>
I meant in the script I import into setup.py :)
<Ninpo>
and yes, that was the place to do it
<Ninpo>
Thank you :)
<arigato>
welcome
agronholm has quit [Ping timeout: 240 seconds]
agronholm has joined #pypy
<kenaan>
cfbolz py3.6 f87194e4cc02 /pypy/module/pypyjit/test_pypy_c/test_globals.py: another dead getfield_gc_r
<kenaan>
cfbolz py3.6 b7b3ef736a84 /pypy/: make sure __import__ is not stored in a celldict cell, otherwise local imports are really slow