antocuni changed the topic of #pypy to: PyPy, the flexible snake (IRC logs: https://botbot.me/freenode/pypy/ ) | use cffi for calling C | "PyPy: the Gradual Reduction of Magic (tm)"
<ciphergoth>
I'm trying to figure out how to use CPython 3.6 + cffi to test my C code, but I'm getting confused. Is this the right place to ask for advice?
<simpson>
This is the right place for cffi questions.
<ciphergoth>
thanks
Hotpot33 has joined #pypy
<ciphergoth>
Assuming I do as the docs advise and use API mode, what's the second argument to set_source? Not the actual C file under test, right?
<ciphergoth>
ffi compiles my modile when I give set_source the library_dirs directive, but then when I call importlib.import_module, it fails with "Library not loaded" and it's looking in the wrong place.
jacob22_ has quit [Quit: Konversation terminated!]
<arigato>
no cluuuue translate.py runs with the GC feeling responsible for 3GB of arenas and 300MB of raw-mallocs, both in default and in mmap-for-arenas
larsivi_ has joined #pypy
<arigato>
but the default RSS grows to 5GB, and the mmap-for-arenas RSS grows to 6GB
<arigato>
fijal: would be cool! ...if we manage to avoid an overlap as this looks like a busy period for us
antocuni has joined #pypy
<fijal>
arigato: right
Arfrever has quit [Ping timeout: 240 seconds]
<nedbat>
Using PyPy on the mac, it seems like the builds aren't entirely endorsed by the dev team yet? Am I misreading that?
<fijal>
nedbat: the builds are a bit "meh"
<fijal>
they require stuff from homebrew
awkwardpenguin has joined #pypy
<nedbat>
fijal: i'd like to be able to use them with confidence, but I can't offer personal effort. If I wanted to put the word out on twitter looking for volunteers to help, would that help?
<fijal>
nedbat: they're allright, just require libraries from homebrew
<nedbat>
fijal: so we just need some more instructions about how to use them?
<fijal>
nedbat: the only difference would be to bundle libraries with the builds, as opposed to requiring them installed
<fijal>
yes and no, bundled build has its value and should probably be there, for which volunteer would help for sure
<fijal>
we generally recommend pypy from homebrew anyway (since then they deal with openssl being up to date for example)
<fijal>
we have noone who tracks openssl releases so we can do security releases of bundled stuff for example (which is a problem)
<Rotonen>
you also need different libs per different macos version (and that keeps slowly changing over time) so most projects just staticly link in their deps instead of trying to educate their userbase to take a stance on userspace library management
<nedbat>
fijal: it took a really long time for me to load that page too!
awkwardpenguin has quit [Ping timeout: 264 seconds]
<fijal>
nedbat: so yeah that's a bug, we should fix it
<fijal>
but it has nothing to do with the builds
<nedbat>
fijal: oh, it seems like it's mac only?
<fijal>
it's a combo of some sort (but yes, only visible on mac)
<fijal>
of host unicode width and target unicode width
<fijal>
note that we have a plan to drop completely unicode width in 6.0 and be always wide
<fijal>
since we'll use utf8 internally
<nedbat>
fijal: when i try pypy2-v5.10.0, I get "dyld: Library not loaded: /usr/local/opt/ncurses/lib/libncursesw.6.dylib", is that a brew thing I need to install? the older versions run fine.
<Rotonen>
oh, you assume homebrew is in /usr/local ?
<fijal>
yeah that's the brew thing
<fijal>
nedbat: so yes, that part would definitely welcome a volunteer to clean it up
<nedbat>
fijal: ok, what kind of skills should I be looking for? "Mac build guru
<nedbat>
"?
<fijal>
I think (but I might be wrong) it's enough to redo cpython hacks
<fijal>
so "guru" might be a bit too much ;-)
<fijal>
how about "someone with round tuits"?
Rhy0lite has joined #pypy
<Rotonen>
https://pastebin.com/xv8iZPWR <- for reference here's what i'm doing for building a python on 10.13 which can run in sandboxed mac apps up to 10.9 - if building on a newer system than the target, you have to take care not to use syscalls not available in previous versions
<nedbat>
fijal: ok, but is it "Mac build"? (yes, guru will scare everyone away!)
<fijal>
yeah
<nedbat>
Rotonen: are you working on the official Mac builds?
<fijal>
nedbat: if you don't mind I'll go have a nap, didn't sleep much last night :-)
<nedbat>
fijal: sleep is good. do it! :)
<fijal>
I'll read the log
<Rotonen>
no, i do closed source mac and win32 python stuff at $WORK and i've been given a release form to share that script of mine to pypy people tackling mac releases
<nedbat>
Rotonen: do you want to help with the official builds? :)
<Rotonen>
i'm here to 1) complain about zope things not working with pypy 2) looking at tackling 64bit windows builds
<fijal>
Rotonen: both worthy and annoying problems :)
<Rotonen>
i might pop by leysin as well as i live in .ch
<Rotonen>
as for the mac builds i can at least provide you that scaffold of some pitfalls i've found - i do recommend building on the latest tools and hacking in support for older versions like that
<arigato>
Crys: I'm unsure that the problem is general; I'd rather think it is specific to sha3
<arigato>
e.g. on cpython there's the C implementation and on pypy it uses the pure python version
<arigato>
(maybe)
<Crys>
arigato: It's Python / pypy 2.7. They don't have sha3.
<arigato>
as far as I know, we fully emulate CPython's behaviour in cpyext
<arigato>
so where does this sha3 come from?
<Crys>
arigato: I'm the author of sha3 package. The actual implementation is a C implementation without tp_dict, tp_dictoffset and without getter for __dict__
<arigato>
ah, there's details in the issue
<Crys>
arigato: <module '_pysha3' from './_pysha3.pypy-41.so'>
<arigato>
can you reproduce with a 10-lines C extension?
<Crys>
compiled with pypy setup.py
<Crys>
I could try
<arigato>
we could try ourselves, too :-)
<Crys>
but 10 lines is going to be hard.
<arigato>
ok, 20
<Crys>
25? :p
<arigato>
you're right, it seems to reproduce
<arigato>
I just hooked in a test making a custom type
<arigato>
and the instance has a __dict__
exarkun has quit [Ping timeout: 240 seconds]
exarkun has joined #pypy
<Crys>
good
<Crys>
I'm not crazy :)
<arigato>
surprizing that nobody complained before :-)
<Crys>
Tell that the three other persons in my head!
<Crys>
:p
tormoz has quit [Remote host closed the connection]
tormoz has joined #pypy
igitoor has quit [Ping timeout: 252 seconds]
igitoor has joined #pypy
danieljabailey has quit [Quit: ZNC 1.6.5+deb2build2 - http://znc.in]
danieljabailey has joined #pypy
igitoor has quit [Changing host]
igitoor has joined #pypy
drolando_ has quit [Remote host closed the connection]
<arigato>
a current state dump about the mmap-for-arenas branch
<arigato>
I may have a theory
<arigato>
small GC objects are stored in arenas, and larger ones raw-mallocated
<arigato>
consider the raw-mallocated objects only: if during translate.py their total size grows to 1.5GB at one point but is usually more 500MB
<arigato>
then that would explain the different behavior in mmap-for-arenas:
<arigato>
in "default" both the raw-malloced and the arenas are obtained with the libc malloc(),
<arigato>
so it's fine to imagine that after the size grew to 1.5GB and shrank again, at least the same memory can be reused for arenas
<arigato>
in mmap-for-arenas it cannot occur, and thus there is 1GB of unused memory that sticks around
<xorAxAx>
ah, sounds legit
<arigato>
yes, I see the size grew to 1.32GB
<arigato>
earlier in the process when there was not so many arenas
<arigato>
the GC triggers the next major GC based on the sum of the raw-malloced and the used-in-arenas size
<arigato>
so we just happened to create a lot of big objects and not so many small objects during that GC cycle
<xorAxAx>
fragmentation in the non-movable space ... how to avoid
<arigato>
some Java VMs have a GC that is similar to PyPy's but also has the ability to compactify the whole heap very occasionally
<arigato>
it's something I'm not sure I want to consider
<arigato>
it would involve adding logic to the GC that is rarely run at all, and rarely breaking the invariant that old objects don't move
<xorAxAx>
and they can compact their equivalent of raw-malloc allocations?
<xorAxAx>
ah, its not the raw-malloc space?
<arigato>
yes
<arigato>
for Java VMs it's all handled by the GC and not delegated to the libc malloc() but yes
<xorAxAx>
well, sure. but if you have an allocation of native memory or however they call it, then they would incur fragmentation of that space as well potentially
exarkun has quit [Ping timeout: 240 seconds]
exarkun has joined #pypy
<arigato>
no, the java VMs get one big chunk of memory at the start and that's it, often
<arigato>
I imagine that if they even have the ability to grow the heap, they ask in very big chunks
awkwardpenguin has joined #pypy
<xorAxAx>
yes
nunatak has quit [Quit: Leaving]
antocuni has joined #pypy
marr has joined #pypy
<kenaan>
arigo default e21bb2a19d05 /rpython/memory/gc/: Print some more stats at the end of a major GC
jacob22__ has quit [Quit: Konversation terminated!]