cfbolz changed the topic of #pypy to: PyPy, the flexible snake (IRC logs: https://botbot.me/freenode/pypy/ ) | use cffi for calling C | the secret reason for us trying to get PyPy users: to test the JIT well enough that we're somewhat confident about it
<windy>
Hi folks, I tried to use RPly to do the parsing. The language is pretty verbose so there are hundreds of grammar rules. After using RPython to translate the parser alone, it took several seconds to process a 120k line program. Is it supposed to be this slow? I haven't attached any JIT to it. Also, are there any large project that uses RPly?
<dstufft>
Dunno if y'all care or not, but linehaul (the stats daemon that turns streaming syslog lines from PyPI's CDN into download events in the BigQuery table) got rewritten and deployed today. Started out using CPython 3.6, but was pegging the CPU on it, dug into it, parsing log lines was the bulk of time, turns out pyparsing is pretty slow :( Switched it to PyPy3.5 6.0.0 atm and it seems to be handling the load much better now :P
<simpson>
Yay!
<dstufft>
instead of 100% CPU it's floating around 20-30% CPU
<dstufft>
I don't really have metrics on how much faster it is, because I don't have metrics in the rewrite :(
<dstufft>
Unless this explodes in some crazy unfixable way, will probably keep it on PyPy, switching runtime is way easier than rewriting the parsing
<njs>
"not fast enough" versus "fast enough" is a pretty significant metric
<cfbolz>
dstufft: awesome! Are you by chance up to writing a (one paragraph) blog post for the PyPy blog about this?
<dstufft>
cfbolz: sure, though let me run it for longer than an hour or so first to make sure it doesn't explode at 6h or something :)
<njs>
cfbolz: heh heh
<cfbolz>
dstufft: sure!
<njs>
cfbolz: I have already started drafting a case study myself :-)
<dstufft>
holding rock steady at 205M RSS and 20-40% CPU
<dstufft>
so far
<njs>
(this is the version of the daemon that got rewritten to use trio, since the asyncio one was crashing constantly)
<cfbolz>
Heh
<dstufft>
(the old linehaul was asyncio based, and it was getting OOM killed constantly, and asyncio hurt my head enough that I found it easier to rewrite it in trio than remember how the asyncio bits worked)
<dstufft>
then I deployed the new one in CPython, and it was pegging the CPU cuz pyparsing
<cfbolz>
dstufft: if there's no c extension and you're closing files I wouldn't be that worried. But indeed I don't know about too many pypy3 deployments
<njs>
It'll be nice to have like a week of data or so, then we can put in pictures showing the download stats before vs. after
<cfbolz>
Yes, sounds great
<dstufft>
pretty sure the only c ext is cryptography, which is cffi so doesn't really count
<cfbolz>
Indeed
<njs>
cfbolz: IIUC the biggest blocker to testing pypy3 was that he was using type annotations on variables (with the attrs package), and had to go through and remove those to make it 3.5-friendly
<njs>
I know you all are working on 3.6 as fast as you can, but maybe it's useful to hear about these things :-)
<graingert>
Is there one for any calls to __del__?
<graingert>
By the GC
forgottenone has quit [Ping timeout: 256 seconds]
jamesaxl has joined #pypy
forgottenone has joined #pypy
<atomizer>
sounds like you want a memory profiler
<atomizer>
there are a lot
ronan has quit [Ping timeout: 244 seconds]
stevenja_ has joined #pypy
tayfun26 has quit [Quit: tayfun26]
stevenja_ has quit [Remote host closed the connection]
stevenja_ has joined #pypy
stevenja_ has quit [Remote host closed the connection]
stevenja_ has joined #pypy
ronan has joined #pypy
stevenja_ has quit [Ping timeout: 240 seconds]
ronan has quit [Ping timeout: 244 seconds]
<marky1991>
stupid question
<marky1991>
interpreters written that use rpython are still limited by the gil, right?
<marky1991>
e.g. if i wanted every function call in my language to take place in its own thread, could I do that and have the threads all execute simultaneously?
<marky1991>
hmm, i found a paper about this exact subject for pypy, I'll review that
Gonsor has quit [Read error: Connection reset by peer]
<wondiws>
simpson, I'm using debian, and I can use concurrent.futures in python3, even though I have not that package installed as far as I know. I suppose concurrent.futures is available too in pypy, but how do I install this?
<wondiws>
and what is ensurepip by the way?
adamholmberg has quit [Remote host closed the connection]
<njs>
wondiws: if you use pypy3, that does have concurrent.futures in the standard library, just like cpython 3
<simpson>
I don't have a PyPy for Python 3 available to me right now, so I can't confirm, but concurrent.futures should be in the stdlib.
<wondiws>
njs, I still try to compile it myself first, that's what I always do, binaries don't quite appeal to me
<njs>
ok. compiling pypy is not for the faint of heart, but if you're having fun then go for it :-)
<wondiws>
njs, oh, is it that difficult?
<simpson>
wondiws: But you're on Debian?~ If you happen to have Nix available with nixpkgs, you could build `pypy`.
<simpson>
It's not hard, but it takes a beefy machine and a bit of time.
<wondiws>
simpson, Nix? nixpkgs? I don't know what you're talking about :P
<simpson>
No worries. Just trying to lay out options.
adamholmberg has quit [Ping timeout: 240 seconds]
<wondiws>
simpson, yes, thank you very much
<njs>
wondiws: it's certainly doable, people do it all the time. but it's not just './configure && make && make install' and 30 seconds later you're done.
Wulf has joined #pypy
adamholmberg has joined #pypy
stevenja_ has joined #pypy
stevenja_ has quit [Ping timeout: 240 seconds]
<wondiws>
njs, still compiling, but 1 hour in, I suddenly get an error that I need gdm.h...