cfbolz changed the topic of #pypy to: PyPy, the flexible snake (IRC logs: https://botbot.me/freenode/pypy/ ) | use cffi for calling C | "nothing compares to the timeshifter, my personal polar expedition in software" - pedronis
TheAdversary has quit [Remote host closed the connection]
TheAdversary has joined #pypy
Hasimir has joined #pypy
asmeurer_ has quit [Quit: asmeurer_]
rokujyouhitoma has joined #pypy
rokujyouhitoma has quit [Ping timeout: 248 seconds]
<DRMacIver>
How good is pypy at specialising integer arithmetic which never overflows a 64-bit word? (And does the answer change if it's inside a trace function?)
<DRMacIver>
(I'm currently fleshing out some plans for coverage performance and wondering whether they're worth porting over to the pure python version for pypy's sake)
<LarstiQ>
hmm, I remember some integer bounds work, but don't remember if that's specialized
kipras`away is now known as kipras
rokujyouhitoma has joined #pypy
rokujyouhitoma has quit [Ping timeout: 248 seconds]
forgottenone has quit [Quit: Konversation terminated!]
riddle has joined #pypy
riddle has quit [Quit: leaving]
riddle has joined #pypy
<fijal>
DRMacIver: but also it mostly does not matter, as in you can squeeze extra few % of performance while being 100x faster than cpython already
<DRMacIver>
fijal: In this case it might, because it's pretty hot path code, and if the integer ops end up allocating (which I can control in the C code) it's probably not really worth making the change.
<fijal>
so neither of them allocates
<DRMacIver>
?
<fijal>
the difference of checking overflow is whether or not you have an overflow checking operation
<fijal>
I think there is a misunderstanding - most times you don't see integers allocated in pypy
<DRMacIver>
Ah, right.
<fijal>
*even* if you can't prove they don
<fijal>
t overflow
<DRMacIver>
Cool. That's plenty for me then
<DRMacIver>
Thanks.
<fijal>
I think storing it on an instance is an exception
<fijal>
(or in something like a heterogenous list)
<DRMacIver>
Ah. That was going to be my next question. Or rather, not storing it on an instance but using it as a key in a dictionary
<fijal>
if you just use integers, then it's optimized
<fijal>
I think keys too, not only value
<fijal>
cfbolz would remember the details
<DRMacIver>
So there are two cases here that I'm looking to optimze.
<DRMacIver>
In line mode, the storage is a dictionary mapping int objects to None. In branch mode, it's tuples of int objects to None.
<fijal>
right, so all of them shuld be fine
<fijal>
I think
<DRMacIver>
The problem that I am going to be trying to fix for CPython is that fresh int and tuple objects are allocated on every line event.
<DRMacIver>
And I can fix that C side pretty straightforwardly, but I'm wondering whether there is some analagous optimisation that would be worth performing on pypy
<DRMacIver>
(the other optimisation I am considering is having a custom storage class for those tuples of ints in CPython. Is there something built-in I can use in pypy to do typed storage?)
<DRMacIver>
All of this is super speculative right now BTW. I'm just feeling out the shape of the space, and I plan to prioritise CPython because I have the feature I need this for turned off by default on pypy anyway :-)
raynold has quit [Quit: Connection closed for inactivity]
arigato has joined #pypy
<cfbolz>
DRMacIver: int to None is an int set?
<DRMacIver>
cfbolz: It is an actual dict due to reasons (I don't actually know what reasons)
<DRMacIver>
But yes it is being used logically as if it were a set
<cfbolz>
OK. It's cheapisn
<cfbolz>
Cheapish
<cfbolz>
DRMacIver: the slowdown will always be a bit huge in pypy because the individual lines in pypy are usually very fast, so an extra dict lookup for every line is comparatively expensive
<DRMacIver>
cfbolz: Yeah one of the things I'm considering for pypy is providing an optimized data structure that acts as a replacement for the dict.
<kenaan>
arigo py3.5 a600b93a7578 /lib_pypy/_ctypes/pointer.py: Fix for ctypes.cast(bytes)
<cfbolz>
It doesn't matter. It will always be much much slower than eg an ADD, JO sequence that is generated for an int addition
rokujyouhitoma has joined #pypy
<cfbolz>
DRMacIver: what is the information you actually want to get out of this,. In den end?
<DRMacIver>
cfbolz: Where this == my line of questioning or coverage itself?
<cfbolz>
Yours
rokujyouhitoma has quit [Ping timeout: 248 seconds]
<arigato>
if the goal is only to have yes/no coverage information for every line,
<DRMacIver>
cfbolz: I'm trying to determine whether there's enough low hanging fruit that I can reasonably expect to make enough of an impact on coverage performance on pypy to make it viable for pypy users to run the new Hypothesis coverage features, without my acquiring a deep understanding of pypy itself and doing some heavy duty work on tracing
<arigato>
it's particularly pointless to JIT-compile any code at all
<Alex_Gaynor>
arigato: That's hard though, because it's possible tracing is the first time you hit a branch, I don't think we have a thing for "trace but immediately delete the resops"?
<arigato>
Alex_Gaynor: it's a trace that does a residual call, and the residual call is not recorded
<arigato>
it should be easy to add, but it's unclear if it solves the problem completely
<arigato>
I guess in Hypothesis's case the question is more subtle, like "is this line of code executed between these two points in time"
<cfbolz>
arigato: we nicely inline into the trace function, that's not the problem at all. No residual calls
<cfbolz>
It's just that whatever tiny thing the trace function does, it will always produce a significant slowdown, if you do it *everywhere*
<arigato>
I'm just saying that simply not inlining the trace function at all and not emitting anything looks like a good potential fix
<Alex_Gaynor>
I think the easiest "correct in all situation" is to provide a hook like "here's my coverage buffer" and then we emit just the "set 1 bit" code every time the tracing function would be called. Then the caller can easily reset or change the buffer whenever they want.
<cfbolz>
arigato: for coverage only, right?
<arigato>
cfbolz: yes, only for a specific situation
<Alex_Gaynor>
Maybe coverage should just be a core VM feature, instead of making people write complex tracing hooks.
<cfbolz>
OK, but I think DRMacIver really wants something very different, anyway
<arigato>
that's my point. the tracing hooks end up complex, implementation- and version-specific, and almost impossible to optimize as well
forgottenone has joined #pypy
arigato has quit [Quit: Leaving]
<DRMacIver>
So I'm specifically interested in building on top of actual coverage.py these days, and in particular tracking arcs is useful, but if you gave me a way to just go "start()/stop()" and just gather every (file, line number) pair that executed in that time I could make use of it.
<DRMacIver>
It would be less good than with branch coverage, but it would be much better than without
<DRMacIver>
err. without any that is
<DRMacIver>
Anyway, I have to disappear for now
<Alex_Gaynor>
DRMacIver: you want the ability to start/stop regularly, correct?
<DRMacIver>
But will be back later.
<DRMacIver>
Alex_Gaynor: Yes
<DRMacIver>
Alex_Gaynor: But if it's low enough overhead the ability to just reset the list at a given point and gather later would be fine.
<Alex_Gaynor>
Ok, so we can't elide stuff from being emitted into JIT'd code.
<DRMacIver>
My specific use case is "Run a test, see what bits were covered by that test"
<DRMacIver>
But I am also interested more generically in performance of coverage
<Alex_Gaynor>
DRMacIver: do you need the exact bits, or would a hash like afl/libFuzzer use be enough?
<DRMacIver>
Alex_Gaynor: I could get by with hashing but it would result in significant loss of functionality
<DRMacIver>
At least in the long run
<Alex_Gaynor>
I don't have a particular conclusion; just thinking about the design space.
<DRMacIver>
Yeah that's fine, I get that :-)
<DRMacIver>
Basically my optimal scenario here is for coverage.py to become magically fast, but can make use of essentially anything
<DRMacIver>
My two categories of use cases are being able to meaningfully distinguish test cases based on execution and to provide better diagnostics to users
<DRMacIver>
Hashing is totally sufficient for the former if it includes branches but provides little help with the latter
forgottenone has quit [Ping timeout: 240 seconds]
arigato has joined #pypy
<kenaan>
arigo default 167b802baf3b /lib-python/2.7/: Issue #2628 Take the 'fake_frame' objects returned by sys._current_frames() on PyPy, and make inspect.isframe() re...
rokujyouhitoma has quit [Ping timeout: 240 seconds]
<ronan>
mattip: I mentioned cpyext finalizers, not tp_finalize
<ronan>
mattip: calling gc.collect() from C code is nearly useless, because it doesn't run the finalizers
<mattip>
ronan: sorry, not really following. Where is this an issue? also - how can we document/avoid the np.void(b'test') crash?
<ronan>
well, the np.void() thing is really a numpy bug, we can't do much about it
<ronan>
except that we could try not to crash, but I have no clue how to identify the issue
<ronan>
mattip: as for gc.collect(), I've run into it in cython tests, but didn't you mention a numpy issue where they call it from C to clear ref cycles?