hannes changed the topic of #mirage to: bug cleaning day on friday, 17th November, UTC afternoon until late -- good news everyone, MirageOS 3.0 was released in February 2017! hack on
balduin has left #mirage [#mirage]
littleli has quit [Quit: Connection closed for inactivity]
olle has joined #mirage
argent_smith has joined #mirage
thomasga has joined #mirage
olle has quit [Quit: olle]
olle has joined #mirage
mort___ has joined #mirage
djs55 has joined #mirage
olle has quit [Quit: olle]
thomasga has quit [Quit: Leaving.]
thomasga has joined #mirage
<hannes>
I'm curious what people think about OCaml versions
<hannes>
by now, we require 4.03, mirage-xen works only on 4.04.2 atm
<hannes>
should we move our travis CI to use 4.03, 4.04, 4.05, and 4.06? or is there a reason to drop any in between (4 jobs sound a bit too extensive for lots of the projects)
<hannes>
[looking into the OCaml changes for 4.04 I can't find a good reason to drop 4.03, but I may miss something]
<thomasga>
I don't think we should support so many versions
<djs55>
I've been adding 4.06 mainly to check `-safe-string`. Maybe I should add `-safe-string` to the existing versions instead of adding another entry to the matrix
<hannes>
thomasga: yes, I agree. I think 4.06+ is the future :D
<thomasga>
maybe we should bump the minimal version every 6 months :p
<hannes>
so, travis: 4.04 and 4.06 should be sufficient (until someone invests time and energy to port mirage-xen to 4.06)!?
<thomasga>
and we should aim to be always support the latest one (eg. thanks for all the work on 4.06!)
<thomasga>
so are you saying we should try to support versions n-2 and n? Why not simply n-1 and n?
<hannes>
thomasga: since mirage on xen is 4.04.2 only atm
<thomasga>
ha that's a good reason :-)
<thomasga>
we should probably invest in making that upgrade cost a bit less costly
<hannes>
which is a separate issue someone should actually fix
<hannes>
there's a plan... to reuse the ocaml-freestanding work.. but so far no volunteer to do it afaict
<thomasga>
yup I saw that
<djs55>
if the minimum version we test is 4.04.2, should we increase the ocaml version in the opam file to match?
<ansiwen[m]>
hannes: no opinion, my project is stuck on mirage 3.0.4 anyway, because of webmachine... :-/
<hannes>
djs55: I'm unsure about that... otoh, it would be honest (since we don't really care about older OCaml releases), otoh if we want to do some performance regression analysis, it is a pain to update all the opam files
<hannes>
ansiwen[m]: but that's mainly a "simple matter of programming"... when someone comes along and writes a HTTP date serialiser/deserialiser without a Unix dependency (as outlined by spiros)
<thomasga>
is there a ticket somewhere on GH with that issue?
<thomasga>
(just curious to see what needs to be implemented exactly)
<ansiwen[m]>
hannes: ha, aren't most of the problems a "simple matter of programming"? ;-)
<hannes>
djs55: i tend towards we should bump to available: >= 4.04.2 to avoid strange issues with wrong versions
<ansiwen[m]>
hannes: and the biggest problem here I think is a bootstrapping problem. For a newcomer like me it's very tough to get into that stuff, very high initial costs. If it would be easier for new-comers to get started I'm sure there would soon be a couple of people who could help releasing the "coding pressure"
<djs55>
hannes: ok, so to summarise we're going to limit the travis matrix to 4.04.2 and 4.06, and bump the available to >= 4.04.2
<djs55>
(is that right?)
<ansiwen[m]>
hannes: I'm currently trying to undersand Irmin, and for a functional-newbie like me it's fucking hard to get into that stuff, because documentation is very technical (not so many examples) and many existing code-snippets are out of date. so a lot of implicit knowledge is necessary. my brain's totally melting.
<hannes>
djs55: that is at least my opinion. if we agree, we should announce this to the mailing list. I also saw in mirage-block-unix you test a huge amount of different linux distributions, which imho is not really needed (since mirage-block-unix is pure OCaml)
<djs55>
yeah I was going to mention that too :)
<hannes>
djs55: for tuntap or other things testing a whole variety makes sense imho, for pure OCaml code not so much
<djs55>
I agree — I think that pure OCaml means we don't need to test lots of distros. I think only repos with C stubs need that
<hannes>
ansiwen[m]: "easier for newcomers to get started" -- yes, that'd be lovely. any concept for that? mine so far is to organise hack retreats in morrocco and get more people on board
<hannes>
ansiwen[m]: yes, irmin could need some more documentation and examples, thomasga is your friend :)
<hannes>
(and yes, more tutorials and better documentation could help the MirageOS project, I feel like we've some maintainence to do, but we're in general in a pretty good shape)
<ansiwen[m]>
hannes: yeah, that's great, I was planning to come to morocco twice at least, and I'm sure I would love it and it would bring a lot value. however, it is also quite a hurdle for someone doing that as a hobbie.
<ansiwen[m]>
My concept would be: connecting newbies online, creating "easy" documentation, with lot's of examples (I know from myself, everybody hates to write that up, but it's really helpful). That's especially important, because Mirage is not exactly a very popular project, so there is not so much mirage-based code out there, that could serve as good examples.
<hannes>
thomasga: oh nice, only needs to be extracted and released!?
<hannes>
djs55: any recommendation which distro in a dockerized travis to use?
<thomasga>
hannes: yup it seems so!
<djs55>
hannes: I normally use alpine since the base image is small and the package manager (apk add foo) is very fast
<thomasga>
+1 for alpine
<hannes>
ansiwen[m]: agreed to both points. good news is i quit my current job to focus on mirageos from next year on :)
<thomasga>
ansiwen[m]: and me too :-)
<hannes>
djs55, thomasga: alpine-3.5 is the right word here, or is there sth "better"?
<ansiwen[m]>
hannes: someone who is seasoned in ocaml might be able to read the code-generated Irmin docs and feel comfortable with it, but for me, who started with ocaml just for MirageOS, this is really tough stuff. It's like learning a language purely with BNF specs. A lot's of the semantics are not clear to me.
<thomasga>
so I will have more time to write tutorials on Irmin
<ansiwen[m]>
you both quit your jobs to work on mirageos? and where is your magic money bag? I might join! :-)
<hannes>
ansiwen[m]: my magic money bag is savings from 3 years of employment at university
<thomasga>
also which docs/tutorials did you read and which parts aren't clear?
<djs55>
hannes: I think alpine-3.5 is fine. It looks like there is a 3.6(.2) but I doubt there's a significant difference for general OCaml CI purposes
<ansiwen[m]>
thomasga: there is not a real walk through example for your own content type, most examples use Irmin.Contents.String
<ansiwen[m]>
thomasga: There is this Irmin.Type stuff that you have to use to define your own type, but nowhere is explained, what it exactly is, and why you need it.
<hannes>
ansiwen[m]: about this irc channel: it is not highly populated at all time, if you don't get an answer, writing a mail to mirageos-devel is usually very helpful
<ansiwen[m]>
thomasga: of course, if everything needs to be serialized to/from JSON anyway, I could just use Irmin.Contents.String and use the JSON strings as content
<ansiwen[m]>
hannes: I did two days ago, there was also not too much answer. maybe I'm too impatient. :-)
<ansiwen[m]>
thomasga: also all the connections between the different modules is not clear to me. What combinations can I actually use in MirageOS unikernels (without unix) git storage on VFAT? git protocol? http protocol? and how do I instantiate all these variations?
<hannes>
ansiwen[m]: I use irmin only with Canopy (https://github.com/Engil/Canopy) -- which uses an in-memory store... regarding block device store there are two approaches under development (e.g. wodan, which has been discussed recently on mirageos-devel) -- I don't think anyone stores irmin/git data on a block device with MirageOS yet
<ansiwen[m]>
hannes: it doesn't have to be on blockdevice... any persistency (over network to a git or irmin server) would be totally fine (even better, from a "cloud-infrastructure" perspective)
<ansiwen[m]>
as thomasga suggested, I will open an Irmin issue, where I collect all the questions that came up during my rough journey :-)
<hannes>
ansiwen[m]: the git rewrite - https://github.com/mirage/ocaml-git/pull/227 seems to have git push working, which is great news. no, i have not used this either with MirageOS..
<ansiwen[m]>
hannes: I wonder what people are using MirageOS for, if a restart looses all the data... what applications don't need some kind of persistency? are all projects based on RO data that is compiled into the kernel? or pure consumers of external data?
<hannes>
ansiwen[m]: the unikernels I have deployed either contain read-only data (i.e. pinata, dns server, nqsb.io), or fetch from a remote git (canopy at hannes.nqsb.io).
<ansiwen[m]>
hannes: so pure fetch? they never change that data?
<hannes>
yes
<hannes>
but as said, there's wodan under development, which should already work for writing data
<hannes>
plus git which has a working git push in a PR
<ansiwen[m]>
if you compare with what containers usually do, and where I see mirageOS as an alternative: they all use a kind of database as a backend, that keeps the data persisten in a volume... if you want to open up MirageOS to all these applications, we really need that.
<ansiwen[m]>
hannes: wodan has "fixed-sized
<ansiwen[m]>
keys and bounded-size values", that doesn't sound very appealing to an application developer, I must say...
<hannes>
ansiwen[m]: there's no doubt we need it, as said earlier there is ongoing work you can try _now_. I'm not a magician and cannot "just make this work" -- it needs people to work on the code. complaining about existing potential solutions does not improve this either, sorry.
<ansiwen[m]>
hannes: I'm not complaining, sorry if you understood it like that. just pointing out. I know that's not the fault of anyone
thomasga has quit [Quit: Leaving.]
<ansiwen[m]>
and I'm happy to help improving the situation as much as I am able to with my current knowledge.
<ansiwen[m]>
sorry, if I came across too negative, wasn't my intention
<ansiwen[m]>
hannes: and I'm not expecting anyone to do any work for me, I just want to have a clear understanding of what is existing currently, so that I know, what my options are and what I have to work on.
<ansiwen[m]>
hannes: so, this http/rest Irmin protocol from a http-client side to another (Non-MirageOS-) Irmin server, like what the "Getting Started" docs show with the cli tool, wouldn't work from a unikernel either? (And until now I wasn't even aware that git push didn't work yet)
olle has quit [Quit: olle]
olle has joined #mirage
talex5 has joined #mirage
mort___ has joined #mirage
thomasga has joined #mirage
<thomasga>
to come back to some of the early discussions in this channel about, thanks to g2p we now have a wodan <-> irmin integration which works. We are polishing the patches before pushing that upstream. Also thanks to @dinosaure we have a much better Git backend (including a full re-implementation of the git GC, push/pull, for both client and server). So with these two work combined we now have a good story for persistent storage for Mira
<thomasga>
applications. It just needs a few more polishing before being of general use.
<thomasga>
also the `Irmin.Type.*` stuff is necessary to seriallise the data on disk and over the network. You could of course use strings and/or cstructs if you want to use your own serialisation format. It's just there for convenience.
<ansiwen[m]>
thomasga: well, the Irmin.Contents.S signature requires a Irmin.Type implementation for your content type... so I have to deal with it in one way or another, right?
<ansiwen[m]>
thomasga: let me resend something I sent after you left:
<ansiwen[m]>
13:11
<ansiwen[m]>
Hhannes: so, this http/rest Irmin protocol from a http-client side to another (Non-MirageOS-) Irmin server, like what the "Getting Started" docs show with the cli tool, wouldn't work from a unikernel either? (And until now I wasn't even aware that git push didn't work yet)
<thomasga>
if you use a string as contents, just use the Irmin.Type.string.
<thomasga>
for the http/rest API: it is very low level now, e.g. the cient has to do lot of work
<thomasga>
so this is not supposed to be used directly — there is work in progress to have a higher level API but this is not done yet
<thomasga>
the current state of irmin sync today is that: if you control both the client and server and they both use irmin (for instance irmin-unix on the client and irmin-mirage on the server) you are fine: it will use a simple JSON/rest API to do the sync. If you don't control the client, some directions are broken today (e.g git push has some issues — pull/fetch mostly work). We are very close to have a much better Git implementation so
<thomasga>
will be solved. Also we are working on having a capnproto API for Irmin for a more efficient/secure RPC system with Irmin.
<ansiwen[m]>
thomasga: ok, so I either use Irmin.Contents.String as content type, but then I have to serialize and deserialize for every data access by myself, right?
<ansiwen[m]>
And if want to use my own ocaml record type as content type, I have to "describe" it with Irmin.Type... and then I can implement the pp and of_string functions using the json serializers from Irmin.Type?
<thomasga>
yes
<thomasga>
and yes
<ansiwen[m]>
thomasga: ok, thanks for that valuable input. A follow up question. My record contains a field of a type from another library (nocrypto RSA key), so I don't have control on that. how can I include that into my content type in the best way?
<ansiwen[m]>
thomasga: still describing it with Irmin.Type, although it's a 3rd party structure, or using a base64-blob, or what?
<ansiwen[m]>
thomasga: oh, I see! that's a good example of a documentation which I have no idea what it's talking about after reading it. ;-)
<ansiwen[m]>
or rather: how to actually use it.
<thomasga>
yes I need to add code examples in the docs
<thomasga>
thanks for pointing this out
<thomasga>
I will probably revisit all the docs at one point anyway, some stuff are a bit old and some info is missing.
<ansiwen[m]>
thomasga: I'm happy to help, I will collect all the issues I had as a newbie, so we can improve these parts. I totally understand how difficult it is to know what a newbie would need to understand it, if you are deep into a subject already.
<thomasga>
thx
<ansiwen[m]>
thomasga: in my case, since I have already json serializers for my type for my own rest api, it would be the easiest to just use that as content type. but I'm scared by the performance impact, that the continuous parsing for every irmin read would cause. I would prefer if I could use Irmin as a KV store for the native ocaml objects. but maybe it's premature optimization?
<hannes>
djs55: should I PR CHANGES and release for ocaml-cstruct? I guess it would be good to have a release now :)
<ansiwen[m]>
thomasga: *use strings as content type (I mean)
<thomasga>
yes I guess that's a premature optimisatio — I guess what you really want is transforming your ocaml value into a tree and update that tree instead. This is something you could do with http://mirage.github.io/irmin/irmin/Irmin/module-type-S/Tree/index.html#type-concrete but there are no automatic converter from Irmin.Type to this concrete tree representation. That would be a good idea to add one.
<thomasga>
I'll open an issue to remember :-)
<ansiwen[m]>
ok, I see.. well, my type is pretty "monolitic", it's a private key, so basically write once, read often. It would be really rare, that I change only one field. you would rather import a complete new key. and even if, the data is small, would be no problem to replace the whole object, although only a single field is changed. but thanks, super interesting, that the irmin tree could expand into the object even...
<ansiwen[m]>
thomasga: ^^^
<djs55>
hannes: sorry didn't see this message! I've made a proposed CHANGES.md in #188 — let me know what you think
<djs55>
I think once your travis changes are in (and the 4.06.0 build fix in #187?) we should definitely tag a release
<hannes>
djs55: lgtm
<hannes>
I also opened at various other repositories mainly updates for testing 4.04.2 and 4.06.0 :)
<hannes>
(and cstruct I'd keep the 4.03 bound tbh, since it is used so widely)
<hannes>
djs55: local build --> warnings are errors (and a different set of warnings) iirc - travis was happy without your 187
<djs55>
ah that makes sense
<hannes>
but I think merging 187 is a good idea!
<hannes>
djs55: you have in mirage-flow and mirage-protocols (and mirage-channel) some PRs... would be nice to get them back into shape... what is the situation with keepalive, are you happy with the interface + implementation?
<djs55>
I think Balraj had a comment (which he may have only made to me in person). I'll try to bash them into shape. I had forgotten about keepalives!
<hannes>
mirage-flow and channel are also not keepalive thingies..
mort___ has quit [Quit: Leaving.]
yomimono has joined #mirage
<yomimono>
hey folks :)
<hannes>
hello mindy! :)
<hannes>
djs55: and keepalive: I did not look at your tcpip changes, but only at the protocol and stack PRs (and think they're good to go)
<hannes>
I will "destroy" https://github.com/mirage/ocaml-scry since it is a fork (of ocamllabs/ocaml-scry) and not mirage relevant. there is also development happening in ocamllabs/ocaml-scry rather than in mirage/ocaml-scry
<hannes>
unless anyone objects _now_
<hannes>
*puff* gone
<yomimono>
belated approval
<djs55>
was there an animation on github involving a recycle bin?
<olle>
reynir. yomimono: I saw your names on a tool to generate certs, at work, today. Thank you for that.
<yomimono>
olle: :D that's awesome to hear! it must be ocaml-certify, I thought nobody was using it
<reynir>
Heh, I mean what tool is that?
<reynir>
If it's certify, then thank yomimono! I barely did anything :-)
<djs55>
hannes: I'll ping avsm on another channel just in case
<olle>
yomimono: That's the one. There was some grumbling "this OPAM, it's so difficult to get installed" or "we should get rid of" and then a couple of text lines of workaround.
<hannes>
I added it to yomimono's list
<olle>
yomimono: Workaround, as in "install intructions:.
<olle>
(Please note, I'm not asking for anything - just saying yay.)
<olle>
Tha's all, my Friday has arrived, as ordered.
olle has quit [Quit: olle]
<yomimono>
:D
<yomimono>
ocaml-ipaddr needs to be adopted by someone, I think :(
<yomimono>
I think mirage-bushel-www may not be all the way abandoned, but perhaps it primarily lives in ocamllabs or somewhere
<yomimono>
but I'll let the maintainer speak for it; I don't really know its status
talex5 has quit [Quit: Saliendo]
<djs55>
I think we're ready to tag cohttp.1.0.0 (and friends) (the CHANGES.md is already merged). I propose to do this, and then merge https://github.com/mirage/mirage/pull/864 and tag mirage.3.0.6 — does this seem sensible?
<yomimono>
are there pins that can fix the build failures in the tests for 864?
<yomimono>
it looks like that's still looking to install mirage-http, which goes looking for Cohttp_lwt.Server
<hannes>
djs55: for mirage 3.0.6 -- should we go all the way and make it ocaml-version > 4.04.2!?
<hannes>
yomimono: djs55 has a mirage-skeleton branch which fixes this! :)
<yomimono>
cool, sgtm then
<djs55>
I can add the pin for mirage-skeleton to fix that… I can edit the travis config at the same time
<djs55>
hm, this is the most interesting travis matrix yet :) we've got various combinations of DISTRO={debian-testing,debian-unstable,centos-7,ubuntu-16.04} OCAML_VERSION={4.03.0, 4.04.2} and EXTRA_ENV="MODE={xen,ukvm,unix,virtio}"