<amirmc>
Aggregating them into a stream of posts would also be useful. So they can be injected into the RSS feed. That can come later though. :)
<avsm>
thomasga: agreed -- it's too big to have the dev reports for mirage cover the 90+ repos, so a storage-focussed one in the datakit repo is convenient enough to follow
<avsm>
+1 amirmc
<avsm>
Alright; and also feel free to PR changes against historical reports -- they are code like any other in the git repo :-)
<yomimono>
bi-weekly mirage ones on the off-week for calls would be great; would give us something to seed call agenda too maybe
<yomimono>
(and remind that the call is the week following)
<avsm>
yomimono: +1 -- it might be slightly off (the monday instead of wednesday) due to calendaring, but i'll work it out and mail the list with details
<avsm>
and if you have anything you'd like included i miss, just email me or whatever in the first instance and i'll include
<avsm>
now, there is something utterly amazing about jbuilder samoht and i discovered today
<avsm>
you can clone multiple repos into a subdirectory tree
<avsm>
type in "jbuilder build" at the toplevel, and it builds them all in one go
<avsm>
referring to opam-installed packages if it can't find them locally
<avsm>
so this is effectively an "automatic pin with incremental build"
<avsm>
it's _amazing_ for day-to-day dev as there's no more need for `opam update -u` all the time
<avsm>
and builds seem to be 4-5x faster for larger source trees
<avsm>
so all in all, it's really making progress and showing concrete improvements
<yomimono>
that also goes pretty far toward addressing sgrove's problems with nondeterministic packaging, right?
<avsm>
yes, big time, since we can vendor all the source
<yomimono>
er, nondeterministic solutions for package installation
<avsm>
opam basically gets out of the way of day-to-day dev
<avsm>
but remains working entirely for end users and publishing
<thomasga>
I've been using the "build all the world" feature today to bisect a weird error in a dependency, that's pretty great
<avsm>
we still need to do some work on hannes' point about building more esoteric repositories (like ctypes or nocrypto), but this model supports those ones being installed by opam
<talex5>
I used it today to test datakit while modifying the 9p library :-)
<thomasga>
also the topkg-builder integration works great
<avsm>
yay thomasga and talex5 -- i used it to build mirage-ci while modifying datakit :-)
<yomimono>
avsm: thanks for updating on that point, it would have been my next question
<thomasga>
watermarking + almost full release cycle almost on par
<yomimono>
this topkg integration looks cool though :)
<avsm>
so I think we can consider jbuilder a cautious green light to do more porting -- but if anyone has time to look at "complex" repositories that would be helpful
<thomasga>
the only missing bits is doc generation
<avsm>
right: docgen is on Jeremie's list
<thomasga>
and we need to check how this works for C stubs
<thomasga>
(with xen-specific flags)
<Drup>
I still kind of agree with bunzli, it's weird to use topkg like that, and the feature might be better of bundled inside jbuilder directly
<avsm>
yep -- so now is the time to try all of this while diml has time to make our requested changes
<thomasga>
e.g. there are still a few unknown to use it as the base for the mirage tool
<avsm>
Drup: that seems in scope as well; the bridge was very straightforward though
<thomasga>
I think the integration can go further
<thomasga>
but the current state is already great :-)
<Drup>
(and it just might be abstract enough so that we could finally split the generic build part from the mirage part, and get all the ad-hoc stuff in functoria)
<apache2>
is it possible to include Ctypes in a mirageos unikernel?
<mato>
avsm thanks
<Drup>
(but that's a side point)
<apache2>
I get an error complaining about lack of the "Str" module
<avsm>
apache2: yes if you do stubgen, but it needs a bit of effort to cross compile
<apache2>
ok
<Drup>
apache2: use re, get rid of str :p
<avsm>
hm, definietly shouldn't need Str. that's a very deprecated module
<Drup>
avsm: it's not deprecated, unfortunatly
<apache2>
avsm: so it's easier if I extend the Cstruct stuff to allow me to leak the pointer to a cstruct buffer?
<avsm>
Drup: morally, if not officially :-)
<apache2>
(all I need is to be able to allocate a page, write to it, and get the pointer to it
<avsm>
apache2: yeah for the moment
<avsm>
ctypes is missing some build glue to be totally automatic
<apache2>
ok, thanks!
<thomasga>
would be great to have a ctype-jbuilder bridge btw :p
<avsm>
do file an issue on cstruct if you have any problems
Bluerise has joined #mirage
<avsm>
thomasga: i think there is one...
<thomasga>
ha..
<apache2>
avsm: thanks, I will try to have a go at it.
<avsm>
on the release front, cstruct, cohttp, github, will all have jbuilder releases with their opam packages split up into explicit ones
<avsm>
thanks to rgrinberg for vast amounts of work on this
<thomasga>
datakit had a release for jbuilder too, I am currently porting irmin
<Drup>
there should also be decent support for incremental jsoo compilation
<thomasga>
(next is ocaml-git)
<apache2>
avsm: do you know if there's a reason the ppx doesn't expose a method to cast a cstruct.t to the record defined inside [%%cstruct ]? It does generate convenience functions for [%%cenm ] whic is super nice
<apache2>
[%%cenum ] **
<avsm>
apache2: since there is no type definition generates for a cstruct.t
<avsm>
but there is one for cenum
<apache2>
oh, never heard about jbuilder. is there a gap between jbuilder and topkg or are they equivalent?
<avsm>
the original cstruct was just meant to be a bunch of views over a Cstruct.t
<avsm>
apache2: jbuilder is a build tool -- it replaces ocamlbuild and oasis
<apache2>
ah alright
<avsm>
topkg is a release management tool, so now there is a bridge between it and jbuilder
<yomimono>
yes, seconding #ocaml - very friendly there and lots of knowledgeable folks
<thomasga>
(I need to run — I'll catch up the minutes later)
<yomimono>
later thomasga!
<dudelson>
functors and GADTs are giving me a bit of a hard time right now
<dudelson>
again, hopefully next week I will be able to dig into those
<yomimono>
it'll all still be there next week ;)
<avsm>
yeah, start with some really simple apps and work your way up
<avsm>
and you'll have Gobi ported to Irmin and Git in no time
<dudelson>
amirmc: hope you like it :) It's very much a work in progress but is a lot of fun to hack on
<avsm>
next topic: Solo5 multiarch status update. We have access to POWER hardware now thanks to IBM, which I need to configure as well, and am working on ARM64 right now (when not on this call)
<reynir>
Does irming use gadts?
thomasga has quit [Quit: Leaving.]
<avsm>
mato! what's happening in the land of multiple architecture support in solo5? can we shed our chains of x86?
<reynir>
sorry irmin
<avsm>
reynir: sort of, the depyt library does (to do type descriptions)
<avsm>
various places elsewhere do as well
<reynir>
Interesting, thanks
<dudelson>
avsm: one last thing: I sent you, yomimono, and thomas a logistics email, but I didn't get any replies. Did you receive it?
<Drup>
depty is probably a bit violent as an introduction to GADTs
<avsm>
dudelson: oops, probably still in my backlog. will look after this call...
<mato>
avsm: I was waiting for an ack on the multiarch PR, got that from ricarkol today so will merge tomorrow.
<dudelson>
avsm: great, thank you!
<mato>
avsm: Then I need to work with the ARM folks to help them rebase (?) their work onto the new order of things.
<mato>
avsm: I'm not sure what the best strategy is there -- they have a long lived branch off ancient master, but the new structure of things on the guest and host sides is entirely different.
<avsm>
the nice folks at packet.net have given us several 96 core arm64 boxes (!)
<avsm>
mato: is there a lot of variance between arm64 boards, or can we have a fairly unified codebase?
<mato>
avsm: There's also armyofdockerness at the lab to which mort has given me access, so I can test on that.
<avsm>
great. my main worry with arm64 is that we'll have lots of weird compile-time configuration flags for different hardware
<avsm>
and am hoping arm64 has made some of that complexity go away unlike arm32
<mato>
avsm: I have no idea, but it's a port to KVM on ARM64, so I don't expect much variance.
<mato>
avsm: but then everything about arm is weird that way, so i'm expecting to be surprised.
<avsm>
well, i'm hopeful we can hold the single-binary line for a while :-)
<avsm>
thanks for all the hard work on this; i'll run some aarch64 ocaml mirage bulk builds by end of week
<avsm>
I am working on building the base containers
<avsm>
for those with arm64, you can now do "docker run -it avsm/multiarch-experiment ocaml" and automatically obtain either an arm64 or x86_64 container depending on where you run it
ricarkol has joined #mirage
<mato>
what are you using to push the muliarch manifests to the hub?
<avsm>
so no need for anything special on arm anymore -- i'm expanding that out so that we can CI automatically on arm64 and ppc64le as well
<avsm>
mato: a custom datakit-ci plugin that builds the multi arch containers and then pushes a unified manifest using the registry v2 spec
<avsm>
i'll have it published on github by end of week -- it needs some modifications to datakit-ci that i'm working with talex5 on
<avsm>
its also easy to add other arches, so i have ppc64le (from ibm), arm32 and then x86_32
<mato>
avsm: do you know of any self-contained tool to push multiarch images?
<avsm>
just use the --from-args option and you dont even need an intermediate file; it's great
<mato>
ricarkol: hey. we were just discussing solo5, I got your OK on the multiarch PR, will merge later today/tomorrow AM.
<avsm>
any other questions about solo5 anyone?
<avsm>
(and hi ricarkol!)
<ricarkol>
Yes mato, all good from my side. Ah, and im working on the gdb PR.
<ricarkol>
Hi avsm
dudelson has quit [Ping timeout: 272 seconds]
<mato>
awesome! getting a better gdb implementation will be great
<avsm>
that'll be useful indeed...
<mato>
(gdb server, that is)
<amirmc>
avsm: I have a side question about arm
<yomimono>
ricarkol mato: do the instructions for gdb on ukvm on the solo5 readme still work?
<yomimono>
(I can try them myself, just thought I'd ask while you're both here)
thomasga has joined #mirage
<ricarkol>
It needs some non-documented changes to the makefile,,
<mato>
yomimono: almost, but I'm not sure what the current hook to get ukvm built with the GDB module via mirage is
<yomimono>
ricarkol mato: is UKVM_MODULES no more?
<ricarkol>
Yes, just add the module
<yomimono>
thanks :)
<avsm>
and finally! OSCON is on! and amirmc and yomimono are speaking!
<avsm>
what's happening in Austin?
<yomimono>
furious computering in the speakers lounge
<mato>
yomimono: I'm not sure.... I changed a bunch of stuff in ukvm-configure on master, need to check.
<mato>
yomimono: however for the version in opam the instructions should work
<yomimono>
mato: great, thank you! :)
<avsm>
yomimono: dont forget to raid the sticker collection
<yomimono>
I'm going to try to squeeze it in to demo time, if time permits
<avsm>
good idea -- debugging is so much easier in ukvm than with xen backend
<avsm>
assuming gdb works, of course :-)
<amirmc>
yomimono and I met someone from Netflix who wrote something about dealing with multiple repos. I'm hoping to chat to her later on.
<avsm>
amirmc: ah yes, the interesting blog post -- most interesting.
<avsm>
my laptop battery is dying, so i need to gently fade into the dark offline night before i am abruptly terminated
<hannes>
I'd like to raise three more things (sorry if this is not a good time) -- what happened to the issue day? can someone tell what to do with https://github.com/mirage/mirage-tcpip/pull/309 ?
<mato>
yomimono: Uh... (just checked), actually those instructions are bogus. They were valid back when we put all of the generated build runes into the Makefile.
<mato>
yomimono: I.e. pre the reintroduction of "mirage build"
<yomimono>
hannes: I'm not convinced that your fix is correct which is why I insist on tests
<yomimono>
if you want to PR reversion of the feature set and change the opam depends and release a 3.1.1 that's fine with me
<yomimono>
it's not a full reversion, we don't have a full test case to test it against, and it's clear that we don't understand what the code is doing in the fuller context
<hannes>
yomimono: well, I'm sure it breaks something with jumbo frames, but I just don't know how to test these... and no, sorry.
<hannes>
nvm then
<yomimono>
we have several maintainers and core team members for this project and there's no good reason this should be blocked on me
avsm has quit [Ping timeout: 260 seconds]
<yomimono>
if you aren't satisfied with my level of service I might suggest that you bring this problem to djs55, the author of the original patchset, and a participant in the thread
brson has joined #mirage
<hannes>
yomimono: thanks for taking care of this issue!
gemmag has quit [Quit: Page closed]
amirmc has quit [Quit: Leaving.]
<yomimono>
hannes: thanks for bringing it up, it is definitely something we need to fix and it shows that we need to do better about testing *before* merge, not after
<mato>
yomimono: So... unfortunately the GDB instructions will not work for Mirage. This is because in the lead up to Mirage 3, we changed to not generating a Makefile and calling out to the Mirage tool itself.
<mato>
yomimono: And in the process of that I forgot about the UKVM_MODULES thing.
<yomimono>
mato: aw, OK. thanks for looking into it.
<mato>
yomimono: there is a workaround, but it's not really suited to a demo...
<yomimono>
it already looked a bit fiddly :(
<yomimono>
so I'll plan to mention that it's possible but not include it, I think
<mato>
yomimono: Yeah, I'm not sure what a good fix would be, in any case would involve some changes in the mirage tool, so not something we can do right now. will think about it.
<yomimono>
mato: thank you :)
<yomimono>
I need to run as well, talk starting very shortly
<kensan>
mato: Regarding the multi-arch work: is it ok for me to update the Muen PR once it is merged or do you prefer to postpone it until after the ARM support has landed?
mort___ has quit [Quit: Leaving.]
strykerkkd has quit [Quit: Leaving]
kensan has quit [Ping timeout: 240 seconds]
reet has quit [Ping timeout: 240 seconds]
insitu has joined #mirage
argent_smith has quit [Quit: Leaving.]
philtor has joined #mirage
insitu has quit [Quit: My MacBook has gone to sleep. ZZZzzz…]