cfbolz changed the topic of #pypy to: PyPy, the flexible snake (IRC logs: https://quodlibet.duckdns.org/irc/pypy/latest.log.html#irc-end ) | use cffi for calling C | if a pep adds a mere 25-30 [C-API] functions or so, it's a drop in the ocean (cough) - Armin
speeder39_ has quit [Quit: Connection closed for inactivity]
lritter has quit [Ping timeout: 240 seconds]
forgottenone has joined #pypy
Rhy0lite has quit [Quit: This computer has gone to sleep]
andi- has quit [Remote host closed the connection]
andi- has joined #pypy
jcea has quit [Quit: jcea]
_whitelogger has joined #pypy
_whitelogger has joined #pypy
_whitelogger has joined #pypy
oberstet has joined #pypy
forgottenone has quit [Ping timeout: 240 seconds]
dansan has joined #pypy
dddddd has quit [Ping timeout: 240 seconds]
forgottenone has joined #pypy
Rhy0lite has joined #pypy
Rhy0lite has quit [Quit: Leaving]
Rhy0lite has joined #pypy
lritter has joined #pypy
forgottenone has quit [Quit: Konversation terminated!]
forgottenone has joined #pypy
Techcable has quit [Quit: ZNC - http://znc.in]
Techcable has joined #pypy
speeder39_ has joined #pypy
viblo has joined #pypy
viblo has left #pypy [#pypy]
viblo has joined #pypy
<viblo> hi
<viblo> Im trying to convert a project using CFFI from ABI to API mode
<viblo> it works fine on linux (or, at least all my tests pass), but on windows there's one that fail which involves some callbacks, and the result is that python exits directly.
<viblo> Anyone have any idea what I can do?
<viblo> I have tried both with cpython 3.8 and latest pypy
<tos9> I can test changes to rlib by running the modules in rlib/test right? Or do I need to translate to see the changes?
<tos9> viblo: you mean your test case crashes python and you're trying to get more info on what happened?
<viblo> yes, exactly
<viblo> python crashes (silently exits) when it reaches a specific place
<viblo> which uses some callbacks from the c-side into python
<tos9> I am not a cffi expert, but 80% of the time this happens to me it's me forgetting to keep around a reference to an object from ffi.new etc.
<tos9> make sure you're not doing that inline in the functions you're calling
<tos9> e.g. do_something_with(ffi.new(...))
<tos9> beyond that, stepping through with a debugger is a thing, did you try that?
<viblo> the python code is quite old, and has been running in ABI-mode for a long time on linux, windows and osx
<viblo> so I think it should be correct
<viblo> but will double check :)
<viblo> I tried compiling my code with debug=True
<tos9> I don't think that will help, it will just show you stuff during compilation IIRC
<viblo> after that I cant import the module at all
<tos9> you need to find where exactly it's segfaulting or whatever
<tos9> it's unfortunately probably also possible that there's a bug in cffi windows support, not sure how likely that is but probably nonzero -- if you paste a reproducible example someone who *is* a cffi expert may have better advice
<tos9> (otherwise probably either way stick around and someone with better knowledge will chime up)
<tos9> ok answering my own question it looks like yeah running rlib tests does the translation itself, cool, I think I never remember that
<tos9> hmmmmm and some tests fail, ok guess I hunt down buildbot
<viblo> ok. I will try to make a example, but its difficult to make something small and selfcontained but Ill do some more testing
<bbot2> Started: http://buildbot.pypy.org/builders/pypy-c-jit-win-x86-32/builds/5386 [mattip: force build, py3.7]
<viblo> I ran it through windbg but Im not very experienced in debugging these things
<mattip> viblo: our mac buildbot is offline, it should be back in a week or two
<viblo> from the looks I think the stack that shows are completely wrong
<mattip> for windows, debug=True means you need a whole environment with a debug-built python
<mattip> the dll links to python37_d.dll
<mattip> sorry, the pyd you build links to python37_d.dll
<viblo> if I use the windows installer, is it enought to just pick the debug symbols_
<mattip> dunno
<viblo> ah, wait you mean I should run python_d instaed of python
<mattip> if you have it, yup
<mattip> this should pick up your debug=True module
<viblo> right, thx
<mattip> can you isolate the code that prepares the callback arguments? In all likelyhood one of them is not being kept alive
<mattip> this happens to me often when I assign to a field in a cdef struct
<mattip> buf = ffi.new(MyType[1]); buf.field = ffi.new(char[10])
<mattip> I had a cffi branch to detect this situation on CPython:
<mattip> but it is just a proof-of-concept
<mattip> this is the actual change
<viblo> when I try to run python_d
<viblo> it says it cant find my module
<viblo> no module named ...
<viblo> after I compiled with debug=True
<viblo> do I need to do somethign else?
<viblo> sorry if this is a basic question :)
<tos9> mattip: the mac buildbot info was to me I assume
<mattip> yup, sorry
<tos9> mattip: is there a way I can answer "did these tests fail for the released pypy3"? for like historical builds instead
<mattip> ... and now that I read your message more carefully: correct, we do not run rpython on mac
<tos9> aha interesting, k, I'm going to assume they always failed then maybe
<tos9> the tests are test_rposix_stat
<tos9> I think I'm slowly getting towards making them pass if I dust off enough cobwebs on how to understand these errors
<mattip> for historical builds you can go to the nightly build for that branch, for instance http://buildbot.pypy.org/nightly/release-pypy3.6-v7.x/
<mattip> but I think old results are purged after a month or two
<tos9> mattip: that's for not-rpython tests though right
<mattip> correct
<tos9> so if nothing outside of rpython failed it won't tell me anything
<mattip> feel free to ask for help, you can use a gist or a pastebin for logs
<tos9> right now I'm at... https://bpa.st/Y2EQ
<tos9> where the bug I'm trying to fix is that pypy is using the wrong types for *time fields on macos, so it's losing precision
<tos9> but macos names the fields differently, so just going one by one to the places that have that `if TIMESPEC` check
<tos9> (it just calls them st_atime, with the e at the end)
<mattip> you might need to change the actual definitions of some of the fields in ALL_STATS_FIELDS ?
<tos9> (figured it out)
<tos9> 5 tests down, 1 still failing
forgottenone has quit [Quit: Konversation terminated!]
forgottenone has joined #pypy
<tos9> mattip: just to make sure I don't do dumb mercurial things, the correct workflow is `hg branch whatever` then `hg push --new-branch -b whatever` right?
<mattip> that will create a permanent branch, which I like
<viblo> mattip: so I have found the specif case that cause the issue now. One callback is defined to return a struct. If I instead change the c-code to return a pointer to struct instead, and then update python code accordingly it works
<mattip> so basically correct, just add a hg commit -m"fix something"
<viblo> so something happening when returning a plain struct...
<mattip> ahh. that seems to remind me of an ffi bug
<viblo> since it was just a struct of 4 floats I return a tuple (1.0,2.0,3.0,4.0) from python.. (which I make sure to keep around for the duration)
<viblo> ah
<viblo> In my case the struct is defined
<viblo> typedef struct cpSpaceDebugColor {
<mattip> please use a pastebin. What is sizeof(cpSpaceDebugColor) ?
<mattip> arigo: any thoughts if cffi is senstive to that bug? The fix is in https://github.com/python/cpython/pull/9258
<viblo> ah sorry
<viblo> is it enough to check from python side? there it says it owns 16 bytes
<viblo> if I create with ffi.new(...)
<mattip> ffi.sizeof(ffi.new('cpSpaceDebugColor[1]'))
<viblo> yes exactly
<mattip> so it is not that bug, since 16 is an allowed size
<viblo> (just a note, I use cffi 1.14.0)
<mattip> what version of cpython/pypy?
<viblo> cpython 3.8.4 / pypy 7.3.1
<viblo> but Ive tried with cpython 3.7 also
<viblo> I will have to leave for bed now, but big thanks for the help so far. At least I have a (quite ugly) workaround of changing c-code to use pointer :) Ill contine to test things tomorrow.
speeder39_ has quit [Quit: Connection closed for inactivity]
<mattip> thanks for the report. Might be worthwhile filing an issue at https://foss.heptapod.net/pypy/cffi
viblo has quit [Remote host closed the connection]
forgottenone has quit [Quit: Konversation terminated!]
<arigo> mattip: a quick note, I think that this cpython pull/9258 is about function *calls*, and not about *callbacks*
<mattip> ahh. There was an older issue about ffi and windows, something about calling conventions but I am pretty sure that one is fixed
<arigo> digging into cffi's own copy of libffi_msvc, though, I see the same bug they're talking about: what about structs < 8 bytes but != 1,2,4,8
* mattip zzz
<arigo> ...indeed, writing a test for that fails on windows
Rhy0lite has quit [Quit: This computer has gone to sleep]
oberstet has quit [Remote host closed the connection]
<bbot2> Exception: http://buildbot.pypy.org/builders/pypy-c-jit-win-x86-32/builds/5386 [mattip: force build, py3.7]