cfbolz changed the topic of #pypy to: PyPy, the flexible snake (IRC logs: https://quodlibet.duckdns.org/irc/pypy/latest.log.html#irc-end ) | use cffi for calling C | if a pep adds a mere 25-30 [C-API] functions or so, it's a drop in the ocean (cough) - Armin
<simpson>
Generally, code will install fine, if its dependencies are compatible, but only fail at runtime.
<astronavt>
ah thats too bad. i guess youd need the developer to be a good citizen and check for non-cpython in setup.py
<astronavt>
or maybe theres a way to specify in pyproject.toml
<simpson>
The most common failure mode you might not already know about is file-descriptor exhaustion, which occurs when people forget to close what they've open()'d.
<Dejan>
astronavt, that is why there are thousands of tests to run
<astronavt>
simpson that would mean the library developer isnt using 'with open'? i never want to install those packages anyway :P
<astronavt>
the weakref stuff looks more problematic
<astronavt>
and __del__ not being called reliably
<simpson>
astronavt: Such code is depressingly common IRL. And folks don't always come to that conclusion; I recall a session with some folks from Intel a few years ago, where they weren't sure whether their code was buggy, or PyPy was buggy.
<simpson>
Well, as you say, there's style, and then there's smell. .__del__() is already a risky method.
<astronavt>
good point
<astronavt>
"For example, a generator left pending in the middle is — again — garbage-collected later in PyPy than in CPython." this is where i imagine stuff like memory leaks could arise even if you arent messing around with weak references and such
<simpson>
In usual use, no, it's not a problem. You'd have to leak the generator in order to leak its frame. What's really being communicated here is a meta-pattern: PyPy doesn't have reference-counting, so unlike with CPython, sometimes objects aren't destroyed immediately upon becoming inaccessible.
<astronavt>
i see. so it will still be gc'ed if it goes out of scope, just not _immediately_
<Dejan>
I am not an expert, but i think CPython does not use RC either
<Dejan>
It uses generational GC
<simpson>
Right. It's like any other GC'd language with finalizers; the finalizers run when the objects are reaped, but that's disconnected from when the objects are dereferenced.
<astronavt>
i thought reference counting was like the heart and soul of cpython semantics
<simpson>
Dejan: CPython has a GC and also RC. The GC chiefly is used to break cyclic references.
<astronavt>
thats the only reason i know what reference counting even is
<astronavt>
"C-API Differences" <- so other than stuff that uses CPython internals, a "correct" C extension should work, albeit maybe with extra overhead
<Dejan>
with if both GC decides to free some memory referenced by RC?
<Dejan>
or vice-versa
<simpson>
Dejan: All objects are registered with the GC and also have an RC field. RC happens on every access. GC runs when memory is low.
<simpson>
GC can read the RC field.
<Dejan>
aha!
<Dejan>
thanks!
<simpson>
astronavt: "It's about time." It's really too bad that the Cython team insists on integration on their terms; it could be a lot smoother and simpler if they stopped insisting on so much C.
<astronavt>
what do you mean "insisting on so much C"
<simpson>
Cython's promise is that we can write code that is basically Python, and it goes fast. PyPy's promise is that we can write Python, and it goes fast.
<simpson>
There's another part of Cython focused on CFFI, but really folks should be using cffi for that.
<Dejan>
I do lots of D coding for my personal stuff mostly, at work I use Python predominantly... I came here as a Java programmer...
<Dejan>
to me, PyPy is the reason why I started coding Python
<Dejan>
CPython is just too slow for some use-cases, and I do not want to have to re-write pieces of code in Cython, or C/D
<Dejan>
PyPy makes me believe Python has future :)
ronan has joined #pypy
Dejan has quit [Quit: Leaving]
ronan has quit [Ping timeout: 240 seconds]
antocuni has quit [Ping timeout: 276 seconds]
<arigato>
cfbolz: compressed pointers: yay?
<arigato>
this document throws some possible plans around
<cfbolz>
arigato: yes, I agree that we don't know it will actually work in the end
<arigato>
it limits the total memory to 4GB though
<arigato>
the old branch in which I played with that compressed pointers to still allow for 32GB of memory
marky1991 has joined #pypy
<cfbolz>
arigato: right
<cfbolz>
arigato: did our branch work with the jit, do you remember?
<arigato>
no, it didn't
<arigato>
(i.e. yes I remember)
<cfbolz>
Right
<arigato>
also it did the thing that can't work on OS X, which is convincing the kernel to give us the initial 4 GB of addressable space
<arigato>
note that the stm branches we did later implement something similar to the more general indirection needed
<arigato>
basically they implement everything we'd need except they still used 64 bits to represent the modified pointers
<arigato>
("modified" as in made relative to a single global address (or thread-local in the case of stm))
<cfbolz>
arigato: right, good to know if we ever want to try again
<cfbolz>
arigato: and the jit was supporting stm, right?
<arigato>
yes
<arigato>
so we'd know exactly the places to fix, at least
<cfbolz>
Cool
<arigato>
I'm still a bit skeptical, but maybe a plain max-4-GB-for-everything-allocated-by-the-GC would work to avoid most speed overheads (and maybe be faster in the end)
<cfbolz>
arigato: why is that so much faster than 32gb?
<arigato>
it would of course consume less RAM, but because the limit is 4 GB, it's not very useful for machines with a lot more than 4 GB
<arigato>
the 32 GB version has got overhead because it needs to shift by three bits, and then also (as the V8 page discusses) it's a simplification if someone can look at 8 bytes of memory and get its value as a GC pointer whether it is "compressed" or not
<arigato>
but maybe that's not a great argument, it might only say we were lazy somewhere
<cfbolz>
arigato: the overhead was a small percentage only in any case, iirc
<arigato>
OK
<arigato>
then maybe we should (one day) try again with the 32 GB limit, but this time including all GC arrays? I think GC arrays were allowed to be anywhere
<arigato>
in theory because it makes the memory limit much higher, but in practice because our GC uses plain malloc for large arrays
<cfbolz>
arigato: right
<arigato>
(the stm branch has got that fixed, because large arrays need to be allocated together with the rest)
<arigato>
(and the GC header was still 64 bits when we could also force that inside 32 bits, etc.)
<cfbolz>
arigato: are any of the changes there generally useful and we should merge them at some point?
<arigato>
I'm not sure
<arigato>
remember that it had its own GC written in C
<Dejan>
the result is the same what i have on CPython
<Dejan>
unless with PyPy we expect different behaviour?
<arigato>
so we do gc.collect() in this test to make sure the coroutines are collected and the warnings emitted, but we also find some coroutines left around from previous tests
<cfbolz>
Dejan: fwiw, it fails on x86 too
<arigato>
yes, my quick fix was enough (see b8f000ca2554)
<kenaan>
arigo py3.6 b8f000ca2554 /pypy/interpreter/test/apptest_coroutine.py: Emit warnings from unrelated older tests before catching the warnings from this precise test
<arigato>
so there were other coroutine objects left lying around, which were waiting for the next GC to emit a warning
energizer_ is now known as energizer
<arigato>
that's why the test failed by getting 3 warnings where only 1 was expected
<cfbolz>
arigato: I've been wondering, this is the struct for the utf8 index storage:
<cfbolz>
arigato: due to alignment, every array entry will be 2*wordsize big, right?
<cfbolz>
ah, sorry. no that works out fine
<cfbolz>
it's three words per array entry on 64 bit
marky1991 has quit [Ping timeout: 265 seconds]
<arigato>
yes
<cfbolz>
arigato: I feel there are still some optimizations missing somehow. eg if we have a string of length 1 and we index it, we make such an array of size 1
<cfbolz>
(probably to not make a bridge that checks the size?)
<arigato>
maybe create_utf8_index_storage could use a cache for very small string sizes?
<cfbolz>
arigato: to share the structures?
<arigato>
yes
<cfbolz>
right
<arigato>
maybe just for utf8len==1
<cfbolz>
for size 1 there are four different ones, for size 2 16
<cfbolz>
right
<arigato>
yes
marky1991 has joined #pypy
<arigato>
or for utf8len <= 3
<cfbolz>
ah no, it's less
<cfbolz>
the ascii ones are never created
<arigato>
yes, and there is only the position of indices 1, 5, 9,...
<cfbolz>
true
<cfbolz>
arigato: I think the other optimization that would be useful is to not create a structure at all for things like s[1:] and s[:-1]. we could only do that optimization in the jit, with jit.isconstant
<arigato>
ouch, yes
<cfbolz>
ok, will look into these
<arigato>
s[:-1] could be done always, by just copying the _index_storage of the bigger string and assuming we're not really loosing memory
<arigato>
s[1:] is quite a bit harder
<arigato>
ah no, I see what you mean
<cfbolz>
arigato: the goal for both would be to never even create the index
<arigato>
yes
<cfbolz>
by just calling next_codepoint_pos
<arigato>
yes
<cfbolz>
ok, on it :-)
<arigato>
:-)
<arigato>
right now, even doing s[0] builds the index I think
<cfbolz>
of course
<cfbolz>
arigato: but again, I would only do it under an isconstant guard, right?
<cfbolz>
otherwise bridges
<arigato>
yes
marky1991 has quit [Ping timeout: 265 seconds]
<Dejan>
Time to go home, have a nice weekend
Dejan has quit [Quit: Leaving]
speeder39_ has joined #pypy
marky1991 has joined #pypy
edd[m] has quit [Write error: Connection reset by peer]
bendlas has quit [Remote host closed the connection]