<njs>
I'm not entirely sure if I recommend this or not
<njs>
but on Linux, the pypy interpreter could provide its own malloc() that increases memory pressure and then calls the real malloc, like an built-in LD_PRELOAD
<njs>
(of course this doesn't work on windows or macos)
Taggnostr has joined #pypy
yuyichao has joined #pypy
yuyichao_ has quit [Ping timeout: 255 seconds]
adamholmberg has joined #pypy
adamholmberg has quit [Ping timeout: 255 seconds]
rokujyouhitoma has joined #pypy
rokujyouhitoma has quit [Ping timeout: 268 seconds]
mattip has joined #pypy
<mattip>
njs: ping
<njs>
mattip: pong
<mattip>
njs: there is a PyPy related pull request on NumPy, it doesn't seem to be getting any traction
Guest59401 has quit [Remote host closed the connection]
rokujyouhitoma has quit [Ping timeout: 268 seconds]
antocuni has joined #pypy
black_ant has joined #pypy
jamescampbell has joined #pypy
girish946 has joined #pypy
kenaan has joined #pypy
<kenaan>
arigo default 4b2c32aaaf36 /: update to cffi/f06010593f74
arigato has joined #pypy
jamescampbell has quit [Ping timeout: 260 seconds]
adamholmberg has joined #pypy
jamescampbell has joined #pypy
jamescam_ has joined #pypy
jamescam_ has quit [Remote host closed the connection]
marr has joined #pypy
jamescam_ has joined #pypy
jamescampbell has quit [Read error: Connection reset by peer]
adamholmberg has quit [Ping timeout: 246 seconds]
rokujyouhitoma has joined #pypy
jamescampbell has joined #pypy
jamesca__ has joined #pypy
rokujyouhitoma has quit [Ping timeout: 260 seconds]
jamescam_ has quit [Ping timeout: 268 seconds]
tilgovi has quit [Ping timeout: 246 seconds]
jamescampbell has quit [Ping timeout: 240 seconds]
jamesca__ has quit [Ping timeout: 240 seconds]
arigato has quit [Ping timeout: 240 seconds]
jamescampbell has joined #pypy
palecsan has joined #pypy
nimaje1 has joined #pypy
nimaje1 is now known as nimaje
nimaje has quit [Killed (orwell.freenode.net (Nickname regained by services))]
jamesaxl has joined #pypy
jamescampbell has quit [Ping timeout: 240 seconds]
<palecsan>
hello, I tried translating a PyPy with oprofile support, but had no success without commenting out the analyze() method from gilanalysis.py. Is this oprofile still maintained? Or do I need other parameters from the command line? I used: pypy ../../rpython/bin/rpython --opt=jit --lldebug0 --translation-jit_profiler=oprofile --make-jobs=16 targetpypy
<palecsan>
standalone.py
jamescampbell has joined #pypy
<palecsan>
I pasted here an output from oprofile: https://pastebin.com/NwchfXM7 I see a "Loop # 2: (pypyjit: get_printable_location disabled, no debug_print)" message
<fijal>
palecsan: it's not really used
<fijal>
I use google perf tools
jamesaxl has quit [Read error: Connection reset by peer]
jamescampbell has quit [Ping timeout: 240 seconds]
rokujyouhitoma has joined #pypy
rokujyouhitoma has quit [Ping timeout: 260 seconds]
jamescam_ has joined #pypy
jamescam_ has quit [Ping timeout: 276 seconds]
John has joined #pypy
John has quit [Changing host]
John has joined #pypy
John has joined #pypy
<gabriel-m>
antocuni: I saw your response for me from westerday
<gabriel-m>
yesterday*
<gabriel-m>
and I have some questions
jamescampbell has joined #pypy
<John>
hi all :)
<antocuni>
gabriel-m: yes?
egregius313 has joined #pypy
egregius313 has quit [Remote host closed the connection]
egregius313 has joined #pypy
jamescampbell has quit [Ping timeout: 240 seconds]
amaury_ has joined #pypy
egregius_ has joined #pypy
tbodt has joined #pypy
egregius313 has quit [Ping timeout: 240 seconds]
adamholmberg has joined #pypy
oberstet has quit [Ping timeout: 240 seconds]
<John>
Hi all
<John>
I know PyPy team are working on Py3 copmatible PyPy - is anyone able to provide a realistic speed comparison between the two as it currently stands?
<John>
I'd like to use asyncio from Py3, but i cannot afford to run Python at the speed of CPython
<dstufft>
okay, I def think I have a memory leak in my PyPy/Warehouse code base, because eve after bumping it up to 2.5GB of ram it's growing to swapping out. Now to figure out what that memory leak is
egregius_ has quit [Remote host closed the connection]
cstratak has quit [Ping timeout: 255 seconds]
egregius313 has joined #pypy
<John>
Are you by any chance creating subprocesses?
<LarstiQ>
John: maybe njs has some relevant comment re asyncio there
<Alex_Gaynor>
dstufft: do you have a leak, or is it just using a lot of memory?
<John>
LarstiQ: well, i don't know anything about asyncio, so there aren't any features i'm dying for. It's more like, i'm waiting for PyPy py3 to get to where PyPy py2 is, so i can make the switch
<John>
across all tools i write and maintain
<LarstiQ>
John: my point was more that asyncio has some drawbacks, and njs is working on an alternative
<John>
So that depends on when pypy for py3 becomes equivalent to pypy for py2 i guess
<John>
ooooh
<LarstiQ>
John: do you have a particular reason to switch to 3?
<dstufft>
Alex_Gaynor: Well I dunno, on CPython I stayed under 512MB easily, switching to PyPy it grew above that, so I switched to a 1GB dyno, it grew above that, so I switched to a 2.5GB dyno and it's going above that. The next size dyno is too expensive for me to try
<John>
LarstiQ: The only thing i can't do in py2 is real hands-on multiprocessing flow control. As a result i know very little about it.
cstratak has joined #pypy
Taggnostr2 has joined #pypy
egregius313 has quit [Ping timeout: 246 seconds]
<John>
i get pools, and i get waiting for all threads to reconvene before continuing (barriers), but since i've always used the subprocess module to handle my multi-processing, this is a big black hole in my knowledge
<Alex_Gaynor>
dstufft: Can you stick a gc.collect() somewhere that runs semi-regularly and see if that changes it?
<John>
So the only reason to move to py3 is to become a better programmer
Taggnostr has quit [Ping timeout: 260 seconds]
<dstufft>
Alex_Gaynor: sure, I have a view that Fastly hits for health checks
<simpson>
John: You can get as hands-on and low-level in Python 2 as you like. `import select` is a bit of a gateway drug. When I recommend Twisted, I recommend it out of a desire to not force you to rediscover these hidden corners of UNIX.
<Alex_Gaynor>
dstufft: every 5 seconds or something?
<John>
hehe, i've used select before but, hehehe, wasn't fun times.
<gabriel-m>
antonuci: sorry, I'm back. The question was about pypy-config. What output should I get for pypy-config --libs and --ldflags ?
<John>
in fact i spent most my time not reading the select man page but reading the unix man page :P
tbodt has quit [Quit: My Mac has gone to sleep. ZZZzzz…]
<gabriel-m>
antocuni *
<John>
(pages)
<John>
And in the end i think i turned away from all forms of multiprocessing
<dstufft>
Alex_Gaynor: raum, it's currently 1 request every minute per Fastly DC, so I think it would work out to something like that yea
<John>
Twisted wasn't a whole bunch better though - there i had to learn all of the Twisted jargon, and it's all specific to twisted and it feels very ineffient learning the specifics of something that may or may not be useful in 5 years time
<John>
I've forgotten it all know. all i remember is that someone has probably made a module for every protocol you can think of in twisted.
<LarstiQ>
John: the names don't matter as much as the concepts
<John>
Hm. I got hung up on the names and gave up. Perhaps i should have stuck with it...
<Alex_Gaynor>
dstufft: 👍 let me know what affect it has, and we can see where that takes us
yuyichao_ has joined #pypy
amaury_ has quit [Quit: Konversation terminated!]
<dstufft>
John: FWIW I had a much easier time learning asyncio then coming back to Twisted than going the other way around, Twisted has a lot more abstracts in place which can be great, but I struggled to get beyod whatever the abstractions didn't provide to really grok how it all worked. asyncio was lower level and made the whole thing click for me
<dstufft>
John: though Twisted is probably a better production grade stack than asyncio
marky1991 has joined #pypy
yuyichao has quit [Ping timeout: 240 seconds]
<John>
that makes total sense to me dstufft, and i think will likely be my experience when i get around to looking at asyncio
<John>
The stdlib gets a lot of flack, but the beauty of it is they always solve the general problem
tbodt has joined #pypy
tbodt has quit [Client Quit]
<John>
And so you learn the fundamentals with the stdlib
<John>
Anyway, thank you all for your time - particularly simpson who helped me out over on #python too :)
<John>
I'll go learn asyncio so i can properly evaluate whatever the pypy team come up with :)
<simpson>
John: Well, I've gone through all this already. Might as well make sure that the next person doesn't have to.
<antocuni>
gabriel-m: it's "__dict__" (bug tracker mistakenly converted the __ into underlines)
<gabriel-m>
thank you for warning me
<gabriel-m>
!
<antocuni>
you're welcome
tbodt has joined #pypy
egregius313 has quit [Ping timeout: 255 seconds]
tbodt has quit [Quit: My Mac has gone to sleep. ZZZzzz…]
black_ant has quit [Ping timeout: 276 seconds]
tbodt has joined #pypy
jamescam_ has joined #pypy
tbodt has quit [Client Quit]
tbodt has joined #pypy
amaury_ has joined #pypy
jamescampbell has quit [Ping timeout: 240 seconds]
girish946 has quit [Quit: Leaving]
tbodt has quit [Client Quit]
cstratak has quit [Ping timeout: 255 seconds]
Rhy0lite has quit [Quit: Leaving]
black_ant has joined #pypy
jamescam_ has quit [Ping timeout: 268 seconds]
Rhy0lite has joined #pypy
rokujyouhitoma has joined #pypy
rokujyouhitoma has quit [Ping timeout: 268 seconds]
jamescampbell has joined #pypy
black_ant has quit [Ping timeout: 246 seconds]
Rhy0lite has quit [Ping timeout: 276 seconds]
Rhy0lite has joined #pypy
cstratak has joined #pypy
black_ant has joined #pypy
egregius313 has joined #pypy
egregius313 has quit [Ping timeout: 240 seconds]
oberstet has joined #pypy
amaury_ has quit [Ping timeout: 246 seconds]
jamescampbell has quit [Ping timeout: 260 seconds]
amaury_ has joined #pypy
mattip has joined #pypy
vkirilichev has quit [Remote host closed the connection]
mattip has left #pypy ["bye"]
amaury_ has quit [Ping timeout: 260 seconds]
rokujyouhitoma has joined #pypy
arigato has joined #pypy
arigato has quit [Client Quit]
girish946 has joined #pypy
rokujyouhitoma has quit [Ping timeout: 240 seconds]
realitix has quit [Ping timeout: 276 seconds]
rmesta has joined #pypy
antocuni has quit [Ping timeout: 268 seconds]
oberstet has quit [Ping timeout: 240 seconds]
girish946 is now known as girishjoshi
jamescampbell has joined #pypy
girishjoshi has quit [Quit: Leaving]
jamescam_ has joined #pypy
jamescampbell has quit [Ping timeout: 240 seconds]
pilne has joined #pypy
oberstet has joined #pypy
jamescam_ has quit [Ping timeout: 240 seconds]
amaury_ has joined #pypy
kipras`away is now known as kipras
jamescampbell has joined #pypy
jamescam_ has joined #pypy
rokujyouhitoma has joined #pypy
jamescampbell has quit [Ping timeout: 240 seconds]
amaury_ has quit [Ping timeout: 246 seconds]
black_ant has quit [Ping timeout: 260 seconds]
rokujyouhitoma has quit [Ping timeout: 268 seconds]
jamescam_ has quit [Ping timeout: 240 seconds]
<dstufft>
Alex_Gaynor: don't have enough data to say for sure yet, but since adding gc.collect() Warehouse memory use grew for a bit, and has now flatlined at some level for the first time since I switched to PyPy. Will wait and see if that trend holds for more than 30 minutes or so
<Alex_Gaynor>
dstufft: where'd it flatten?
<dstufft>
just over 1GB of RAM
<dstufft>
previously it was getting killed at Heroku beyond 2.5GB of ram
<dstufft>
killed by heroku*
black_ant has joined #pypy
jamescampbell has joined #pypy
egregius313 has joined #pypy
cstratak has quit [Ping timeout: 240 seconds]
jamescam_ has joined #pypy
jamescampbell has quit [Ping timeout: 240 seconds]
<dstufft>
gsnedders: just to an view function that fastly uses to check the health of the WArehouse service (e.g. something that will reliably get called every few seconds)
<Alex_Gaynor>
dstufft: ok, I assume it's staying flat?
<dstufft>
Alex_Gaynor: let me check again
<dstufft>
Alex_Gaynor: looks like right after I said it was staying flat, it jumped up a few times again, up to 1.3GB now
<dstufft>
shoulda kept my mouth shut
mattip has joined #pypy
<Alex_Gaynor>
dstufft: are there any heavily used cffi extensions in warehouse? pscyopg2cffi I'm guessing?
<mattip>
gabriel-m: you might want to check out this cffi-wxpython mix from amaury
<Alex_Gaynor>
dstufft: why are you using both bcrypt and argon2_cffi?
<dstufft>
Alex_Gaynor: some passwords are hashed with one, some are hashed with the other
<Alex_Gaynor>
dstufft: how hard would it be to switch from pscyopg2cffi to pg8000?
jamescampbell has joined #pypy
<dstufft>
Alex_Gaynor: Um, unless there's some specific compatability issue, probably not hard, it's all going through SQLAlchemy so my code shouldn't change
<dstufft>
I can try that
<dstufft>
looks like there are a few pure python options
<Alex_Gaynor>
dstufft: probably any pure python one will do
<tos9>
Alex_Gaynor: Are you aware of specific issues with psycopg2cffi? Or just a guess?
<tos9>
(We also run psycopg2cffi in prod... I wonder what our memory usage looks like over longer term, can have a look)
jamescampbell has quit [Ping timeout: 240 seconds]
<Alex_Gaynor>
tos9: Testing whether it's the same issue pjenvey hit yesterday with externally allocated memory allocations dwarfing gc allocated ones, leading to infrequent GC runs, leading to memory bloat
* tos9
nods
<tos9>
Probably should read the scrollback, but first /me pulls up our memory usage graph.
sophiya has joined #pypy
<Alex_Gaynor>
Just using psycopg2cffi wouldn't guarantee you'd it, you'd need to also not allocate a lot of GC memory.
<Alex_Gaynor>
dstufft: (Remove the gc.collect() calls when testing with pg8000!)
<tos9>
Alex_Gaynor: Which I assume means "don't create lots of objects" right?
<tos9>
That we probably... do not fall into, depending on what level triggers it. For every psycopg2cffi response we assemble a Python object.
<Alex_Gaynor>
tos9: That's probably not a sufficient description. Objects which never actually turn into gc allocations because the JIT removes them wouldn't count, nor would (I think) objects which don't outlive the nursery.
<tos9>
A ha.
<dstufft>
pg8000 is the most popular one, so gonna use that
<Alex_Gaynor>
pjenvey, cfbolz: Actually, I just realized that's another detail in our cffi+gc troubles, you specifically hit this if very little of your garbage outlives the nursery. I wonder if there's some cleverness with ffi.gc() (or __del__ for that matter) and nursery we could do
<tos9>
OK, well, either way, I'm looking at our memory usage graphs and they are impressively flat over a 1 month period, so I'm guessing whatever it is, we're not hitting it
rokujyouhitoma has joined #pypy
jamesaxl has quit [Ping timeout: 268 seconds]
apotap has joined #pypy
iko_ has quit [Ping timeout: 276 seconds]
rokujyouhitoma has quit [Ping timeout: 260 seconds]
<Alex_Gaynor>
cfbolz, pjenvey, fijal: So in a local test, as far as I can tell, if you use ffi.gc() you're basically guaranteed to have higher steady state memory usage then with non-destructor objects, because they bypass the nursery and are only collected on major collections the following program uses more memory than the equivilant allocation using just GC memory
<Alex_Gaynor>
cfbolz: the approach of adding a size arg to ffi.gc(), but no default if you don't provide it?
<cfbolz>
yes
<cfbolz>
about the default we need to think. I agree that somethings seems not ideal
<Alex_Gaynor>
I don't think that's sufficient for cryptography -- we often have no idea how much memory openssl malloc'd behind teh scenes. Although I suppose that API is enough for us to set our own default with a wrapper function
<Alex_Gaynor>
I'll leave a comment tonight, not logged in to bitbucket on my work computer
<Alex_Gaynor>
alternatively, reaperhulk ^ want to leave a comment? (if not I'll do it when I'm home)
<dstufft>
hm
<dstufft>
pg8000 is borken on Warehouse, UUID type nonsense. Wonder if py-postgresql is any better
Ryanar has joined #pypy
rokujyouhitoma has joined #pypy
rokujyouhitoma has quit [Ping timeout: 260 seconds]
antocuni has joined #pypy
<dstufft>
Alex_Gaynor: Ok, turns out the answer is No, I can't switch to a pure python thing, because they all suck too hard to actually work :(
<Alex_Gaynor>
dstufft: this makes sciencing difficult. mmmm, so you can leave the gc thing in I guess, if it's not hurting perf too much
<dstufft>
(it's still growing with gc.collect(), but it hits periods of flat growth for awhile, and it seems to be generally growing at about half the rate)
<nanonyme>
I thought txpostgres was good
<dstufft>
nanonyme: well for one it's a twisted library and this isn't twisted ;) but afaik it just uses psycopg2 under the covers
<nanonyme>
Maybe I remember wrong then. I'm pretty sure there was an actual implementation of the Postgres protocol
tbodt has quit [Quit: My Mac has gone to sleep. ZZZzzz…]
<nanonyme>
Anyway, yeah, I doubt there are good and battle-tested pure-Python alternatives
<nanonyme>
Oh, pgasync, maybe?
<nanonyme>
Except that's not pure-Python, apparently
<dstufft>
Alex_Gaynor: Hmm, that bug mgiht be a red-herring here, psycopg2cffi doesn't use ffi.gc anywhere
<tos9>
(FWIW I evaluated [quickly] all the postgres libraries at the time and yeah I settled on psycopg2cffi precisely because all the others looked bad)
<Alex_Gaynor>
dstufft: It uses __del__, same dealio
<dstufft>
Alex_Gaynor: ah okay
<tos9>
If there's a way I can help btw by sending more details about our own app that might find the problem obv lemme know.
<Alex_Gaynor>
dstufft: ffi.gc is just wrapping a cffi ptr with something that has an __del__ (and acts like the underying object for all other purposes)
<dstufft>
maybe I'll just switch back to CPython for now until https://bitbucket.org/cffi/cffi/issues/320/improve-memory_pressure-management gets solved and then try again. Even if it steady states where it's at now, it'll "cost" like 3x the amount of credits for the 2.5GB dyno instead of the 512MB dyno I was using with CPython
<Alex_Gaynor>
cfbolz, pjenvey: I think this demo makes the argument that the problem isn't cffi at all, since I can make the issue with only GC memory.
Rhy0lite has quit [Quit: Leaving]
egregius313 has joined #pypy
jamesaxl has quit [Quit: WeeChat 1.7.1]
<Alex_Gaynor>
Very easily gets up to 4GB on this computer.
egregius313 has quit [Ping timeout: 246 seconds]
vkirilichev has joined #pypy
<pjenvey>
Alex_Gaynor: ah, nice
mattip has left #pypy ["bye"]
<tos9>
Uh, apologies for the dumb question, but does that reproducer not just say "objects with __del__ are not being collected"?
<tos9>
Or er, that, but "inside lists" maybe?
<Alex_Gaynor>
tos9: It's more complex than not collected; it's about the ratio of various types of allocations
rokujyouhitoma has quit [Ping timeout: 260 seconds]
linkmauve1 has left #pypy [#pypy]
<tos9>
Does gc.set_debug(gc.DEBUG_STATS) do anything on PyPy (completely unrelated to the above)
<tos9>
OK, looks like no.
<antocuni>
tos9: I think that pypy does not even expose that method
<tos9>
antocuni: Yeah seems to be the case
<antocuni>
s/method/function
<tos9>
Can I get PyPy to tell me when the GC is running?
<tos9>
And what kind of collection it's doing?
<tos9>
PYPYLOG?
<antocuni>
I don't think you can know from within the python program
<antocuni>
but yes, PYPYLOG tells you
<tos9>
Will PYPYLOG have "reasonable-ish" performance impact?
<tos9>
Looks like quite a bit of output for a simple program
<antocuni>
there are two ways to use PYPYLOG
<tos9>
ah, just found the docs, ok, it's not a filename :P
<antocuni>
the first is PYPYLOG=filename: this emits only the start/end markers, and it should be very fast
<pjenvey>
why does malloc's add_memory_pressure default to False
<antocuni>
sorry, "PYPYLOG=filename" without the colon
<tos9>
antocuni: a ha
<antocuni>
the second is "PYPYLOG=:filename", with a colon at the start: it also prints the info *inside* the sections
<antocuni>
this is supposedly slower, although you should check
<antocuni>
if you use the colon, you can also filter only the sections you are interested in
<antocuni>
"PYPYLOG=gc:filename" only emits info about the gc* sections
<tos9>
antocuni: should PYPYLOG=-: do anything?
<tos9>
(it doesn't seem to print anything for me)
<antocuni>
tos9: this means "select all the sections whose name starts with '-'"
<tos9>
antocuni: Ah, OK, so it doesn't mean the "write to stderr" that it means at the end then
<antocuni>
"PYPYLOG=:-" is probably what you whant
<tos9>
OK, /me tries with a filename
<tos9>
antocuni: Yeah I was trying to follow your first filename: example
<antocuni>
note that there are various tools to analyze the PYPYLOG output
<antocuni>
tos9: look at the difference between "PYPYLOG=-" and "PYPYLOG=:-"
<antocuni>
also, pypy/rpython/tool/gcanalyze.py
<antocuni>
or pypy/pypy/tool/memusage/memusage.py
<pjenvey>
er, just shouldn't be needed in most cases (answering my own q)
egregius313 has joined #pypy
<tos9>
antocuni: A ha, cool, will check those out -- I'm seeing long pauses in an app and just want to 100% confirm it's not the gc.
<antocuni>
ah, then PYPYLOG=gc:- should help
iko has joined #pypy
adamholmberg has quit [Remote host closed the connection]
adamholmberg has joined #pypy
adamholmberg has quit [Remote host closed the connection]
adamholmberg has joined #pypy
amaury_ has joined #pypy
<tos9>
antocuni: Can one of those 2 tools tell me how long major collections are taking?
<tos9>
(Is timestamp even recoverable from whatever this value in the brackets here means)
egregius313 has quit [Ping timeout: 240 seconds]
<antocuni>
I think that the hex value is some internal cpu counter; i.e., it's not helpful as an absolute value but you can measure relative times (not sure about the unit of measure, though)
<tos9>
K, will run gcanalyze.py and see what it spits out
<antocuni>
ah, rpython/tool/logparser.py is also interesting: IIRC, it draws a graph of how much time is spent in each section
<antocuni>
I don't remember exactly what all these tools do, but you should be able to easily write your own analyzer if you want
* tos9
nods
<tos9>
antocuni: Thanks for the tips for sure.
<tos9>
Think one of these will either work or tell me enough about what I should write
jamescampbell has quit [Remote host closed the connection]
<antocuni>
logparser.py print-summary seems like a good start
jamescampbell has joined #pypy
hihi has quit [Ping timeout: 260 seconds]
hhhhh has quit [Ping timeout: 260 seconds]
<tos9>
Bleh... PYPYLOG=gc-collect:/foo is still writing to stderr
<pjenvey>
Alex_Gaynor: that 'dump_rpy_arena' I suggested yesterday would probably be useful for looking at this latest leak script of your's
jamescampbell has quit [Ping timeout: 260 seconds]
vkirilichev has quit [Remote host closed the connection]
oberstet has quit [Ping timeout: 246 seconds]
rokujyouhitoma has joined #pypy
egregius313 has quit [Remote host closed the connection]
egregius313 has joined #pypy
rokujyouhitoma has quit [Ping timeout: 240 seconds]
egregius_ has joined #pypy
egregius313 has quit [Ping timeout: 246 seconds]
egregius_ has quit [Remote host closed the connection]
egregius313 has joined #pypy
jamescampbell has quit [Remote host closed the connection]
egregius313 has quit [Ping timeout: 246 seconds]
marky1991 has quit [Ping timeout: 260 seconds]
amaury_ has quit [Ping timeout: 260 seconds]
egregius313 has joined #pypy
antocuni has quit [Ping timeout: 246 seconds]
<dstufft>
Alex_Gaynor: we may have finally reached a steady state on PyPy + Warehouse + psycopg2cffi + gc.collect() in /_health/ at 2.2GB of RAM (it's held to roughly that for 2 hours now). That's higher than I feel comfortable with ATM so I'm still going to go back to CPython for now, but once there's a fix for the __del__ thing in PyPy I'll try it out again
Taggnostr2 has quit [Remote host closed the connection]
tbodt has joined #pypy
rokujyouhitoma has joined #pypy
egregius313 has quit [Remote host closed the connection]
egregius313 has joined #pypy
rokujyouhitoma has quit [Ping timeout: 255 seconds]
<pjenvey>
dstufft: you can double check it's not something stupid at the python level leaking by taking a look at pypy's gc.dump_rpy_heap, you have to pass it through pypy's gcdump script for readable output
<pjenvey>
(I actually embedded gcdump inside our app to spit it out, pypy should probably include something similar out of the box to make that easier)
<pjenvey>
dstufft: but with cpython so much lower.. maybe unlikely it's something stupid
rmesta has quit [Quit: Leaving.]
<dstufft>
pjenvey: you'd expect CPython to have the same leak wouldn't you? PyPy and CPython should have the same I thought, just PyPy might clean up much later because it's not refcounting? I could be completely off base though :)
<dstufft>
(could be a leak in one of the libraries I used on PyPy and didn't use on CPython though ofc)
<dstufft>
Digging into it real deep is more effort than I feel like spending right now though, I don't particularly have a problem with CPython in this app, I mostly wanted PyPy because I think PyPy is great in general and I wanted to see if PyPy3 was good stuff yet or not :)
<pjenvey>
dstufft: yea, but pretty unlikely it's something like that