cfbolz changed the topic of #pypy to: PyPy, the flexible snake (IRC logs: https://botbot.me/freenode/pypy/ ) | use cffi for calling C | "the modern world where network packets and compiler optimizations are effectively hostile"
<yuvipanda>
is there a way for me to tell from vmprof total 'wall time' spent on a function?
<yuvipanda>
than just relative time?
<fijal>
nope
<fijal>
(it does not have notiont of time FYI)
<fijal>
you could guess, but not
<fijal>
no
<yuvipanda>
right
demonimin has quit [Ping timeout: 240 seconds]
realitix has joined #pypy
demonimin has joined #pypy
demonimin has quit [Changing host]
demonimin has joined #pypy
forgottenone has quit [Ping timeout: 240 seconds]
forgottenone has joined #pypy
rokujyouhitoma has joined #pypy
<njs>
yuvipanda: cprofile will at least tell you how many times the rollback code is called, so you know whether it's one giant call or 1000000 little calls
<yuvipanda>
njs: right. I know it gets called at the end of every request tho...
<yuvipanda>
njs: I'm now unsure if it is necessary or not.
<yuvipanda>
not sure if that is necessary ofc
<njs>
oh odd.
<yuvipanda>
it looks like possible defensive programming, in case a commit was forgotten earlier.
<njs>
of course "wait we're doing this thing that takes a ton of time and we don't need to" is the ideal outcome from profiling, so hopefully it's that :-)
<njs>
one might hope that the db library is clever enough to realize that if the very last command on a connection as a commit then the rollback is unnecessary
<njs>
but perhaps not...
<yuvipanda>
njs: plus the fact that it's tornado complicates things - idk if we have one session per connection or not...
rokujyouhitoma has quit [Ping timeout: 260 seconds]
<njs>
yeah I can't help you there
<yuvipanda>
njs: am also not sure if cpython had the same profile.
<yuvipanda>
njs: so it's possible this is something that's cheaper with cpython and we didn't notice...
<njs>
or everything else is more expensive :-)
<fijal>
maze of passages all alike....
<yuvipanda>
right :D
<njs>
(but like fijal noted yesterday, ORMs sometimes do weird "clever" things that cause pypy to break out of JIT-mode and get slow, it's not impossible you're hitting one of those)
<yuvipanda>
yeah
<yuvipanda>
I should tweak my dockerfiles so I can easily switch between pypy and cpython
<fijal>
njs: did you look through my pycon ru slides by any chance?
<njs>
fijal: didn't I chat with you about them the other day?
<fijal>
maybe :)
<fijal>
conferences are weird - they erase my ability to commit to long term memory
<njs>
I think the conclusion was "yep those problems sure are hard problems" :-)
<fijal>
click
<fijal>
njs: ok, now I remember :)
<njs>
(and I whined that mutating generator locals is totally a legitimate use case)
<fijal>
yes, yes, you remind me exactly about the conversation :)
<njs>
:-)
<yuvipanda>
njs: also, not sure if you saw - the patch fixed the SSL corruption :) I still can't comment on bitbucket tho
<njs>
fijal: so what's the status of no-GIL pypy?
<fijal>
njs: we need to write the blog post what we want to do, I think
<realitix>
Hello fijal and arigato. During his keynote, Armin Ronacher said that pypy copy the mistakes of CPython. Do you know what he was talking about ?
<fijal>
bug-to-bug compatibility
<fijal>
C API
<fijal>
such stuff
<realitix>
yes but is there others choices ?
<realitix>
to keep cope compatible
<realitix>
code*
<LarstiQ>
realitix: the difference is in philosophy, wether to aim for best compatibility, or to fork the Python language for a "better" one
<fijal>
we tried not supporting C API
ronan has joined #pypy
<njs>
realitix: I'm not sure he even meant this as a criticism of pypy
<realitix>
i agree
<arigato>
yuvipanda: thanks for confirming that the SSL corruption is now fixed :-)
<yuvipanda>
np, arigato! thanks for fixing it!
ronan has quit [Quit: Ex-Chat]
ronan has joined #pypy
ronan has quit [Read error: Connection reset by peer]
vkirilichev has quit [Remote host closed the connection]
nimaje1 has joined #pypy
nimaje1 is now known as nimaje
nimaje has quit [Killed (verne.freenode.net (Nickname regained by services))]
inad922 has joined #pypy
ronan has quit [Quit: Ex-Chat]
ronan__ has joined #pypy
ronan__ is now known as ronan
vkirilichev has joined #pypy
vkirilichev has quit [Remote host closed the connection]
vkirilichev has joined #pypy
rokujyouhitoma has joined #pypy
rokujyouhitoma has quit [Ping timeout: 240 seconds]
rmariano has quit [Ping timeout: 260 seconds]
mihneas has joined #pypy
<yuvipanda>
njs: fijal to close the loop, we used vmprof to figure out that sqlalchemy was hitting us, and it was a bug in our code that was making us do far more noop DB rollbacks than necessary.
<yuvipanda>
njs: fijal we fixed it, and now everything is much faster :)
<fijal>
yuvipanda: cool, what are the raw numbers?
<yuvipanda>
fijal: don't have good comparison numbers between pypy and cpython yet.
<yuvipanda>
I'll probably have something by end of the week :)
<yuvipanda>
fijal: do you know if the cpext overhead (vs cffi, say) is like 10x or more like 1.5x? just trying to guess order of magnitude :)
<yuvipanda>
fijal: anyway, am going to sleep now - it's 5AM! It would be very useful for us if you could find time to raise the vmprof.com limit :D we've sorted out our problems rn for upto 1.5k users, and beyond that the profile is > 100M limit :D
<fijal>
more like 5x
Rhy0lite has joined #pypy
<yuvipanda>
fijal: hmm, that shouldn't be too bad.
<fijal>
plan_rich: can you look into that?
<yuvipanda>
we can probably live with that for now
rubdos has joined #pypy
magniff has joined #pypy
antocuni__ has joined #pypy
rokujyouhitoma has joined #pypy
rokujyouhitoma has quit [Ping timeout: 268 seconds]
<arigato>
is anyone responsible for the "contrib/virtualenv-pypy" subrepos? that seem to have appeared recently
<fijal>
on BB?
<arigato>
yes, in pypy/pypy
<fijal>
Our apologies, but the repository "virtualenv-pypy" has been deleted.
<fijal>
or where do you find it?
<fijal>
plan_rich: ping
vkirilichev has quit [Remote host closed the connection]
<kenaan>
arigo default 3c4fb99e0c59 /pypy/module/gc/: Issue #2612 Simplify gc.get_referrers(). Before, it wasn't guaranteed to return the opposite result as gc.get_ref...
<arigato>
yes, I've got a different "hg verify" result on different repos...
<arigato>
but always something along the same lines
<arigato>
and on one repo, no error at all
<arigato>
pjenvey: I didn't investigate so far
<arigato>
yes, I think your last comment is invalid, because W_Roots should use only the new finalizer queues, not the old-style rpzthon __del__, indeed
antocuni has quit [Ping timeout: 255 seconds]
* arigato
is on Windows and Windows likes to switch my keyboard to the US style at *entirely random points in time*: that's why that previous line contains "rpzthon" but "finalizer". it switched in the middle
<pjenvey>
=]
<TheAdversary>
pjenvey: Error: Spurious "]". You may want to quote your arguments with double quotes in order to prevent extra brackets from being evaluated as nested commands.
<benbangert>
lol
<mattip>
+]
<mattip>
=}
<mattip>
=]
<TheAdversary>
mattip: Error: Spurious "]". You may want to quote your arguments with double quotes in order to prevent extra brackets from being evaluated as nested commands.
<arigato>
what exactly is this bot?
realitix has quit [Quit: Leaving]
<fijal>
good question
<fijal>
TheAdversary: help
<TheAdversary>
fijal: (help [<plugin>] [<command>]) -- This command gives a useful description of what <command> does. <plugin> is only necessary if the command is in more than one plugin. You may also want to use the 'list' command to list all available plugins and commands.
<TheAdversary>
fijal: Error: There is no command "owner". However, "Owner" is the name of a loaded plugin, and you may be able to find its provided commands using 'list Owner'.
rokujyouhitoma has quit [Ping timeout: 258 seconds]
yuyichao has quit [Ping timeout: 248 seconds]
yuyichao has joined #pypy
asmeurer_ has joined #pypy
antocuni has joined #pypy
asmeurer_ has quit [Quit: asmeurer_]
<mattip>
if a PyObject has a ob_pypy_link, is there an easy way to find out if that pypy_link is not referenced anywhere and collectable?
<mattip>
then we could directly release it in decref when the PyObject's refcount reaches rawrefcount.REFCNT_FROM_PYPY
rokujyouhitoma has joined #pypy
forgottenone has quit [Remote host closed the connection]
rokujyouhitoma has quit [Ping timeout: 255 seconds]
Rhy0lite has quit [Quit: Leaving]
asmeurer_ has joined #pypy
vkirilichev has joined #pypy
<arigato>
mattip: no
<arigato>
it's as hard as the problem of releasing unclosed file objects
<mattip>
ok, it should just be an optimization anyway, the gc should be collecting correctly
* mattip
am optimistic I will find the problem
<mattip>
by comparing when refcnt reaches 0 on cpython vs pypy
<mattip>
s/0/whatever passes for 0/
<mattip>
setting a watchpoint I see cpython decrefs the callargs in call_function, it seems we do not?
rokujyouhitoma has joined #pypy
rokujyouhitoma has quit [Ping timeout: 268 seconds]
<mattip>
false alarm. The args tuple in the function call in PyPy has an effective refcnt of 0, but is kept alive by its pypy_link, all is seemingly OK
rmariano has quit [Ping timeout: 240 seconds]
Ryanar has quit [Ping timeout: 246 seconds]
rokujyouhitoma has joined #pypy
rokujyouhitoma has quit [Ping timeout: 240 seconds]