#pypy on 2017-05-01 — irc logs at freenode.irclog.whitequark.org

05:25 <njs> is that expected? (I noticed because I'm trying to be careful to make sure that I explicitly call .release() on my memoryviews to avoid issues with pypy's gc, and now I'm confused because the test I wrote for this isn't failing after all...)

05:25 <mattip> njs: "which version of pypy", and "might be a bug"

05:25 <njs> mattip: this is pypy3 5.7.0

05:28 <njs> mattip: test case is just 'b = bytearray(10); m = memoryview(b); b += bytes(100)' -- on cpython this raises BufferError, on that pypy it succeeds

05:28 Garen_ has joined #pypy

05:29 <mattip> jsut a wild guess, from my refcheck branch on pandas, perhaps cpython uses refcount semantics

05:29 <mattip> to track other references to the bytearray, and if ob_refcnt > 1 it raises

05:29 <mattip> let's see if I can verify that

05:29 <njs> on cpython, I believe mutable buffer exporting objects like bytearray have a special number-of-buffer-exports count

05:30 <njs> it's not ob_refcnt, you can have two variables pointing to the same bytearray object and still mutate it

05:30 Garen has quit [Ping timeout: 252 seconds]

05:32 * mattip cloning cpython from github, which feels wrong somehow

05:35 <bbot2> Failure: http://buildbot.pypy.org/builders/own-win-x86-32/builds/1385 [default]

05:37 <njs> huh, it looks like the memoryview is actually following the bytearray's internal buffer around when it gets reallocated, at least if I'm correctly interpreting this ._pypy_raw_address method I found

05:43 <mattip> AFAICT we are missing cpython _canresize() function from Objects/bytearrayobject.c, we don't bother to track-and-check anythin like ob_exports

05:44 <njs> clever

05:44 <bbot2> Success: http://buildbot.pypy.org/builders/pypy-c-jit-linux-x86-64/builds/4578 [matti: force build, cpyext-obj-stealing]

05:45 <njs> but... what about this code, which also fails in bytearray _canresize()?

05:45 <njs> b = bytearray(10); a = np.asarray(b); b += bytes(100)

05:46 <njs> (I mean, it fails on cpython)

05:48 <mattip> my cursory glance at pypy.objspace.std.bytearrayobject.py, like in descr_add,

05:48 <njs> of course np.ndarray doesn't have anything like the memoryview.release method, so once you start doing this thing you're kind of screwed in any case without a refcounting gc

05:48 <mattip> seems to me we don't bother checking anything, we just reallocate

05:49 <njs> but yeah, it sounds like that leaves you with an ndarray pointing into the middle of nowhere?

05:50 <mattip> can you check that in your original example, m and b have different buffers, or do they overlap?

05:50 <njs> what do you mean by "have different buffers"?

05:50 <mattip> i.e. if you assign to m, the values in b do not change

05:50 <mattip> (b after the appending operation)

05:51 <njs> ah, no, in pypy3 it seems to be very clever – when I change the length of b, then m._pypy_raw_address() changes, and writes to either b or m are still visible in the other

05:52 DragonSA has joined #pypy

05:52 DragonSA has quit [Changing host]

05:52 <njs> I assume this is do with the mechanism

05:52 <njs> ugh

05:52 <njs> https://gist.github.com/njsmith/0ffd38ec05ad8e34004f34a7dc492227

05:53 <njs> I assume this is related to the mechanism the gc uses to keep pointers correct when moving objects

05:54 <mattip> so I guess everything is OK and we should simply document that difference,

05:55 <mattip> unless there is an explicit place in python documentation where cpython behaviour is required

05:57 <njs> heh, and if I shrink the size of the bytearray to be smaller than the memoryview, the memoryview starts raising IndexError!

05:58 <mattip> the documentation https://docs.python.org/2/library/stdtypes.html#memoryview is very terse about modifying the underlying buffer

05:59 <njs> the docs do mention this: https://docs.python.org/3/library/stdtypes.html#memoryview.release

05:59 <njs> "Many objects take special actions when a view is held on them (for example, a bytearray would temporarily forbid resizing)"

06:00 <njs> but I dunno that anyone cares about this. I only care because I wanted to make sure it wasn't happening :-)

06:00 <mattip> ok, so 3.2+ seems to need that

06:01 <njs> I guess you may run into trouble with cpyext stuff eventually though

06:02 <mattip> cpyext uses a "clever" PyBuffer to expose the buffer as a c-pointer

06:03 <mattip> and you are correct, there are probably still issues with it

06:03 <njs> mattip: so what does that np.asarray line I pasted above do on pypy?

06:07 <mattip> it seems to be messed up

06:07 <mattip> b = bytearray(10); a = np.asarray(b); b += bytes(100); print a; print a[0]; print a

06:08 <mattip> prints two different values for contents of a

06:10 <njs> well, uh, you're welcome I guess :-)

06:10 <mattip> :)

06:11 <mattip> very strange

06:27 altendky has quit [Quit: Connection closed for inactivity]

06:53 amaury has joined #pypy

06:57 <bbot2> Failure: http://buildbot.pypy.org/builders/jitbackendonly-own-linux-armhf/builds/1453

06:57 <bbot2> Started: http://buildbot.pypy.org/builders/pypy-c-jit-linux-armhf-raspbian/builds/1295

07:18 <bbot2> Failure: http://buildbot.pypy.org/builders/pypy-c-jit-macosx-x86-64/builds/2997 [py3.5]

07:37 arigato has joined #pypy

07:42 amaury has quit [Ping timeout: 240 seconds]

08:06 Tiberium has joined #pypy

08:07 ielectric has joined #pypy

08:08 ielectric is now known as domenkozar

08:19 arigato has quit [Read error: Connection reset by peer]

08:25 amaury has joined #pypy

08:28 <bbot2> Success: http://buildbot.pypy.org/builders/jit-benchmark-linux-x86-32/builds/2986

08:33 oberstet has joined #pypy

08:43 Taggnostr has joined #pypy

08:56 antocuni has joined #pypy

08:57 FASDFSF has joined #pypy

09:05 Taggnostr has quit [Ping timeout: 260 seconds]

09:15 asmeurer_ has quit [Quit: asmeurer_]

09:18 Taggnostr has joined #pypy

09:26 arigato has joined #pypy

10:03 ramonvg has joined #pypy

10:09 antocuni has quit [Ping timeout: 255 seconds]

10:26 _whitelogger has joined #pypy

10:27 forgottenone has quit [Ping timeout: 260 seconds]

10:47 arigato has quit [Read error: Connection reset by peer]

10:52 mattip has left #pypy ["bye"]

11:07 <bbot2> Failure: http://buildbot.pypy.org/builders/pypy-c-jit-win-x86-32/builds/3077 [default]

11:07 <bbot2> Started: http://buildbot.pypy.org/builders/pypy-c-jit-win-x86-32/builds/3078 [py3.5]

11:16 amaury has quit [Ping timeout: 240 seconds]

11:23 jamesaxl has quit [Read error: Connection reset by peer]

11:25 jamesaxl has joined #pypy

11:27 kipras`away is now known as kipras

11:38 kipras is now known as kipras`away

11:46 antocuni has joined #pypy

12:10 forgottenone has joined #pypy

12:15 kipras`away is now known as kipras

12:24 amaury has joined #pypy

12:36 jacob22_ has joined #pypy

12:45 Rhy0lite has joined #pypy

12:48 <bbot2> Success: http://buildbot.pypy.org/builders/jit-benchmark-linux-x86-64/builds/2194

13:05 marky1991 has quit [Remote host closed the connection]

13:05 marky1991 has joined #pypy

13:23 amaury has quit [Ping timeout: 260 seconds]

13:44 rmesta has joined #pypy

13:44 rmesta1 has joined #pypy

13:45 antocuni has quit [Ping timeout: 255 seconds]

13:48 rmesta has quit [Ping timeout: 240 seconds]

13:50 <kenaan> arigo default fa2b7ecca667 /lib_pypy/_ctypes/structure.py: Issue #1213: in ctypes, complain instead of silently ignoring _swappedbytes_

13:57 oberstet2 has joined #pypy

14:00 oberstet has quit [Ping timeout: 240 seconds]

14:04 <bbot2> Failure: http://buildbot.pypy.org/builders/pypy-c-jit-win-x86-32/builds/3078 [py3.5]

14:07 FASDFSF has quit [Ping timeout: 255 seconds]

14:13 Tiberium has quit [Ping timeout: 240 seconds]

14:15 Tiberium has joined #pypy

14:15 ronny has joined #pypy

14:22 <tos9> ^ arigo: thanks!

14:31 yuyichao has quit [Ping timeout: 260 seconds]

14:34 lritter_ has quit [Ping timeout: 260 seconds]

14:46 tbodt has joined #pypy

14:47 yuyichao has joined #pypy