antocuni changed the topic of #pypy to: PyPy, the flexible snake (IRC logs: https://botbot.me/freenode/pypy/ ) | use cffi for calling C | "PyPy: the Gradual Reduction of Magic (tm)"
drolando has quit [Remote host closed the connection]
drolando has joined #pypy
<arigato>
solarjoe4: I added an answer yesterday, are you looking for something else?
<arigato>
as for "why does the padding happen", cffi tries as best as possible to emulate the C compiler for your platform, so on Windows (only) it aligns "double" on 8 bytes
<solarjoe4>
arigato, oh, that's you :) I responded earlier, the structs are not defined within my code, but a thrird party dll, so I can't pack them. basically I am looking for no-worry hassle free way to handle this, because there are a lot of structs...
<arigato>
I guess either numpy doesn't add any alignment, or they have their own rules that says that "doublre" is only 4 bytes-aligned
<arigato>
ah I see
<arigato>
then it's really a question for numpy
<solarjoe4>
bascically I need either numpy or cffi to be smart enough to handle the padding
<arigato>
how to have numpy accept the particular extra padding that you don't really control
<arigato>
how is that a cffi issue?
<arigato>
well I guess, you could have two structures declared in the cdef
<arigato>
one with packed=True and the other without (so two calls to ffi.cdef())
<arigato>
and copy the fields manually from one to the other
<solarjoe4>
didn't mean that, I am just not that familiar in detail with C
<solarjoe4>
and cffi...
<arigato>
so, the C compiler can add padding and depends a bit on which exact C compiler is used
<solarjoe4>
or is there a completely different way to do this? instead of cffi.buffer and np.frombuffer? or is this generally the right way?
<arigato>
if your input is really a C structure with such padding, then using cffi like you do is probably the sanest option
<arigato>
but then if you want to pass it to numpy, you can't use the same structure directly
<arigato>
so, either you copy the fields one by one into the numpy array (slow unless you're on pypy)
<arigato>
or you convert into packed data, like a struct declared with packed=True
<arigato>
or, indeed, you add padding in the numpy array declaration
<arigato>
if you want an automated way to know the padding, you can do that:
<solarjoe4>
yes, but I need to know where to insert the paddings
<arigato>
ok
<arigato>
you can use for example ffi.offsetof("mystruct", "a")
<arigato>
and ffi.sizeof("mystruct")
<arigato>
and use this info to build the dtype list with padding
<arigato>
(painfully)
<solarjoe4>
hm ok, thanks, I have to read on the functions
<arigato>
(you don't have to add a 'f4' for padding; it's more general to add a number of bytes ('i1') instead)
<solarjoe4>
I will probably run into problems :)
<solarjoe4>
because the effect is the same?
<arigato>
yes
<arigato>
well if your structure really has got only one place where it could need padding,
<arigato>
you can also go for the special solution
<arigato>
and insert an 'f4' only if ffi.offsetof("mystruct", "d") > ffi.offsetof("mystruct", "c") + 4, for example
<solarjoe4>
hm there are many structs, probably no special cases, they are not even sorted by size. would I use ffi.typeof to find the fields of the struct, then get their type and offset and check from there?
<arigato>
yes
<solarjoe4>
would ffi.alignof be of any use?
<arigato>
it looks harder to use, there
<arigato>
it's used inside cffi itself to know which offsetof should be applied
<arigato>
but in this case, you're really interested in these offsetofs