cfbolz changed the topic of #pypy to: PyPy, the flexible snake (IRC logs: https://botbot.me/freenode/pypy/ ) | use cffi for calling C | the secret reason for us trying to get PyPy users: to test the JIT well enough that we're somewhat confident about it
<tsutsumi>
filal: ok, I'll take a look. One thing that's not clear to me: runicode.py is a part of rpython, and unicodehelper is a part of the pypy interpreter. Is the idea to remove that unicode functionality from rpython? or reduce what's required there?
<tsutsumi>
fijal: ^ sorry misread the name
dustinm has quit [Quit: Leaving]
<fijal>
tsutsumi: reduce what's required there to be less python specific
<kenaan>
cfbolz default 353420330f35 /pypy/objspace/std/: cleanup in dict implementation: don't use has_* flags, put fallback implementations into base methods instead
<karanlyons>
Hey all, I'm seeing some weird behavior and a possible discrepancy between SSL in asyncio loops with multiprocessing. Test case: https://dpaste.de/W0yA Output: https://dpaste.de/VwYP
<karanlyons>
This error *doesn't* occur if spawn or forkserver are used, so I'm guessing there's some global coming along for the fork ride and messing things up.
lritter has quit [Read error: Connection reset by peer]
<karanlyons>
I actually have no idea how I'd even go about debugging this. I'll keep poking at it, but the test case and information that it only happens when using fork (and not spawn or forkserver) may be all I've got.
<Rotonen>
multiprocessing pickles the globals for you unless you use a shared memory worker
<karanlyons>
Rotonen: This is actually an issue of compatibility between PyPy and CPython...I think.
<karanlyons>
It isn't me sending some globals through args or otherwise in my own code.
<karanlyons>
Unless there's something wrong with my test case?
<Rotonen>
there was someone recently trying to do something like that, to spawn pypys from cpython or so - nocando per noncompatible in-memory data structures - but do hold on for an answer from an actual pypy dev - i just hang around here
<karanlyons>
Yeah, this is PyPy forking PyPy.
<karanlyons>
My guess is that it *is* a global, but one within _cffi_ssl someplace.
<fijal>
pfff
<fijal>
karanlyons: without looking in detail (and not even denying this is pypy bug), I would strongly recomment not using multiprocessing for anything at all ever
<karanlyons>
So you'd prefer subprocess?
<karanlyons>
That's still gonna fork(), no?
<fijal>
yes, but the communication is "sane"
<fijal>
eg it does not try to pickle and unpickle and pretend it's the same
<fijal>
I would probably use execnet or some other wrapper around subprocess
<karanlyons>
Oh, yeah, but again: not passing arguments that are going to have pickling problems.
<fijal>
(or just pipes0
<fijal>
right, so that's the problem with implicit pickling
<fijal>
if you understand your communication model, it's a lot less of a problem
<karanlyons>
This is—I think—specifically an issue with forked processes and something in the SSL libs.
<fijal>
are you sure it has nothing to do with multiprocessing itself?
<karanlyons>
Using forkserver or spawn with multiprocessing prevents the issue. So...kinda? I mean, it's possible that the issue is multiprocessing pickling some global state up weird and starting from a "clean" proc sidesteps the problem.
<karanlyons>
I can try to whip up an example just using fork to see.
<fijal>
would be cool
<fijal>
getting multiprocessing out of the equation would make bug hunting a lot easier too :-)
<karanlyons>
Though that might be kinda tricky since I'm also spawning the child process from inside of an asyncio loop.
<karanlyons>
Hm, let's first see if I can get the error without asyncio at all.
<Rotonen>
i might be boring here, but just use threads?
<Rotonen>
(instead of asyncio)
<karanlyons>
I already have a fix for *my* problem.
<karanlyons>
And no, I want asyncio for this.
<karanlyons>
But I'm just working now to figure out exactly what this problem is, and if it's PyPy's, a possible fix.
<karanlyons>
Yeah, I can't figure this out but my best current guess is that it's something to do with calls to the cffi openssl, likely that some context gets set up and copied over when it shouldn't be.