03:09 <DanC_> hmm... primitive _init takes a parameter of type Env. so any module can get an Env just by declaring a primitive?

10:08 <sylvanc> ponysaurus: x.bitwidth()

10:09 <sylvanc> having a separate call for a byte size vs a bit size seems unnecessary, although of course always open to be showing it's worth it

10:10 <sylvanc> if you are doing a writebuffer, and you need .bitwidth(), it's possible you are doing it wrong :)

10:10 <sylvanc> sounds like maybe you are accepting a union type to be written to the buffer?

10:10 <sylvanc> if so, every caller will end up allocating memory

10:11 <sylvanc> DanC: yeow, that's an oversight from long ago!

10:11 <sylvanc> i think primitive _init should take no arguments

10:11 <sylvanc> thanks for the heads up

18:02 <SeanTAllen> @Praetonus i dont understand why a type for "collection of arrays" would be needed and how that is different than Seq[Array[A]] or why that is preferrable to Seq[Seq[A]], can you explain?

18:04 <ponysaurus> sylvanc: Yes, I was accepting a union type. I am going to clean that up.

18:06 <ponysaurus> @jemc, @praetonus .. : The following makes more sense, would like to hear your thoughts: Have a list of arrays as the writebuffer data structure. Keep allocating a new buffer, as we fill up (possibly exponentially increase the size?).

18:07 <ponysaurus> somehow provide an implementation to extract a sendable packet from the HEAD of the buffer. Then manage the buffer and the arrays that are getting used up at the head?

18:07 <ponysaurus> does this sound more reasonable?

18:08 <ponysaurus> *keep allocating for a new array (NOT buffer)

18:10 <Praetonus> SeanTAllen: The difference is that the collection I'm proposing provides a direct access to its elements. The collection of arrays is just a storage detail. In an array, all elements are stored contiguously. In a linked list, elements are separate. Here, the idea for the storage is to have nodes (like a linked list) of contiguous storage (like arrays). Some kind of hybrid structure

18:12 <Praetonus> But the key really is the direct Seq access to elements. The collection could then be used directly in WriteBuffer and even be retreived directly by the user since it provides a safe Seq interface

18:12 <Praetonus> The underlying arrays wouldn't be accessible directly

18:12 <ponysaurus> I was just about to type the first sentence, yes!

18:12 <ponysaurus> type your first sentence

18:13 <SeanTAllen> "the underlying arrays wouldn't be accessible directly", that sounds like "access" means a copy. am i misinterpreting that?

18:13 <SeanTAllen> i'm not sure i am going to understand without at least some pseudo code

18:13 <ponysaurus> how would you provide safe and sendable access to the first N elements without a copy?

18:14 <ponysaurus> this is the tricky part

18:14 <Praetonus> By access, I mean there won't be a array(i: USize) returning the ith array

18:15 <Praetonus> But there will be a apply(i: USize) returning the ith global element

18:15 <ponysaurus> of course not, by access I don't mean that either. But how do we extract the first few elements without a copy?

18:15 <Praetonus> e.g if there are two arrays of size 500 and you ask for the 750th element, you get the 250th of the second array

18:16 <jemc> ponysaurus: to avoid copying, my inclination is that when I want to "take" some bytes from WriteBuffer object, it gives me back a ByteSeqIter rather than a ByteSeq

18:16 <jemc> the ByteSeqIter can be used with `writev`

18:17 <jemc> so if you have the N bytes you want to return segmented over multiple arrays internally, you return those multiple arrays to avoid copying as much as possible

18:17 <Praetonus> I'd say the whole object is the ByteSeq. We just have to keep the global size to ensure all accesses to elements are in the structure bounds

18:17 <ponysaurus> hahaha .. brings me back to my issue about ByteSeqIter ;)

18:17 <jemc> see scatter/gather IO

18:17 <SeanTAllen> what issue is that ponysaurus ?

18:18 <jemc> Praetonus: TBH I'm not seeing how your proposed data structure is that helpful for either (Read)Buffer *or* WriteBuffer

18:18 <SeanTAllen> in my experience praetonus, "just" tends to hide a lot of complexity

18:19 <jemc> it sounds like your DS is just a different implementation of an Array or List - I don't see how it solves the problems we're talking about (or at least the ones *I'm* talking about)

18:19 <jemc> not that it wouldn't be a useful data structure - I just see it as a separate conversation I think

18:19 <SeanTAllen> i am very confused at this point. i think some code, pseudo or otherwise would help quite a bit

18:20 <ponysaurus> SeanTAllen: #691

18:21 <jemc> ponysaurus: yeah, that's something we need to fix - but I don't think it invalidates my point :)

18:21 <ponysaurus> @jemc, I am going to try and code that up with ByteSeqIter. What's a best way to collaborate in cases like this? github gist?

18:22 <ponysaurus> i am sure I will bother you here, with questions

18:22 <jemc> gist is fine - or you can point to a different branch on your personal repo

18:22 <ponysaurus> ok .. will do

18:22 <jemc> but basically, I would picture something like this:

18:23 <ponysaurus> go on ..

18:23 <ponysaurus> :)

18:23 <Praetonus> jemc: You're correct, ByteSeqIter could be used for this. However, getting a Seq from WriteBuffer could be useful, for example to allow some final modifications before sending

18:24 <Praetonus> SeanTAllen: I'll make a basic implementation and put the link on Github, it will be better if we have actual code to discuss

18:26 <Praetonus> SeanTAllen: On a completely unrelated subject, I can't find the invite on Github. Is it hidden in a special tab or something like that?

18:28 <SeanTAllen> Praetonus: they send an emaik

18:28 <SeanTAllen> Praetonus: they send an email

18:28 <jemc> ponysaurus: internally, I have a List of ByteSeqs, some may be Array[U8] and some may be String: ['F', 'O', 'O'], "BAR", ['B', 'A', 'Z'] - I only add elements to the tail, and only remove elements from the front - in the public API, I can `append` an entire ByteSeq (Array[U8] or String) by pushing to the tail of the list - I can `push` a U8 to the list by either pushing it onto the last ByteSeq or creating

18:28 <jemc> a new ByteSeq with one element (or varying the strategy based on the circumstances, whichever is deemed to be the most efficient algorithm)

18:29 <SeanTAllen> it also says they can be accepted at "github.com/ponylang" Praetonus

18:29 <Praetonus> SeanTAllen: Ah yes, here it is

18:30 <jemc> ponysaurus: to remove from the front, I remove whole ByteSeqs at a time - if for some reason I need to limit the number of elements to some maximum N, I may need to in some cases split a ByteSeq on the boundary and copy its contents into two resulting ByteSeqs, one kept at the head and the other as the tail of the return ByteSeqIter

18:31 <jemc> however most of the time I would think you wouldn't need to limit the number of elements, so you would just extract the entire contents as a ByteSeqIter and continue with an empty list

18:32 <jemc> I think your use case of fixed size packets is the more specialized one, compared to the generic use case of arbitrary-sized packets

18:33 <jemc> not all transports have framing, so the difference between two fixed size packets and a single arbitrary packet combining both of their contents is often a difference without a distinction

18:36 <ponysaurus> This makes sense, however, i still don't like resizing the existing ByteSeq at the tail, every time I push a number to it.

18:36 <ponysaurus> jemc

18:36 <ponysaurus> jemc:

18:37 <jemc> there's room to tweak the optimal algorithm there - however it should be noted that Array has some semi-"smart" resize logic when it resizes, so that even if you declare size 1, it starts at 8 and increases by powers of 2

18:37 <ponysaurus> jemc: And also, am I the only one who wishes for fixed sized packets? MTU in network transport is still a useful thing

18:38 <jemc> ponysaurus: you're not the only one - I just think that this is not the only use case we need to support

18:38 <ponysaurus> ok, thanks for the resize info

18:40 <ponysaurus> yes, and I completely understand the don't care about packet size case. But, you should take care of people who care about packet sizes. Because, once you make that choice, you may lose them :(

18:40 <ponysaurus> jemc

18:41 <jemc> this is one reason why I'm always harping about third-party packages over stdlib ones - it's often the shortest path to usefulness if we each solve our own specific problems and share the solutions with others who want to use and contribute to them - rather than trying to make something specialized into something generic and waiting/working for community consensus on that generic solution, when everyone has

18:41 <ponysaurus> @jemc: all in all, thanks a lot for your feedback

18:41 <jemc> different needs and priorities

18:42 <jemc> ponysaurus: yes, I'm not saying it can't support fixed size packets - just saying we probably want to support arbitrary packet sizes if it's going to be in stdlib

18:42 <ponysaurus> I agree, this isn't a compiler issue. And can be made into a 3rd party package first. I will do that.

18:42 <jemc> the problem right now with 3rd party packages is that we don't have a package manager yet :D

18:43 <jemc> but yes, you can certainly move faster when you're not trying to get everyone in a diverse group of people to agree on something

18:43 <ponysaurus> and then the stdlib can take it, if it makes sense. This looks like a much more logical route.

19:02 <ponysaurus> jemc: Just to summarize my thoughts: Imagine a WriteBuffer as a list of fixed size ByteSeqs. Maintain a HEAD and TAIL. Tail to where the user keeps writing. Head from where the user demands a ByteSeqIter of a certain length. Now I am not sure if pony has a way to provide this ByteSeqIter without copying overhead. But I believe this will provide what’s necessary for both parties who care about the packet size and to those who d

19:05 <ponysaurus> I would like to have people's opinion on this. And this will definitely fit my use case, once issue 691 is fixed. I am going to focus on trying to get something like this implemented for myself.

19:14 <jemc> ponysaurus: using a List[ByteSeq], if your packet size happens to fall on a boundary between ByteSeqs, it should be possible without copying to sever the link of the linked list at that boundary

19:15 <shepheb> I made ponyc segfault. pulling and rebuilding, and I'll try with the latest.

19:15 <jemc> if your packet size doesn't happen to fall on a boundary, I don't think there's any way to avoid copying - which is one big reason why one might not want to use an exact packet size

19:15 <jemc> you could instead try for a packet size *near* some given size - rounding to the nearest Array boundary - if your concern is MTU

19:16 <jemc> errr.. ByteSeq boundary

19:18 <shepheb> no segfault on master. never mind then.

19:28 <ponysaurus> @jemc : Ok, thanks!

20:11 <DanC> re package manager... I wonder if it would help to use nix until you manage to do something better

20:16 <jemc> DanC: I think the main point of contention in the package manager issue is not about the actual management of packages, but about defining how the user experience looks - where do you define dependencies and how does it integrate (or not integrate) with the compiler

20:18 <jemc> there are some great ideas in that thread, some of them conflicting - I have some ideas about how to try to support a variety of use patterns via plugins to a generic `pony` binary, but haven't gotten around to fleshing these out yet

20:19 <jemc> but I think the answer is yes - we should support things like nix and gx that already work well for managment of packages

20:20 <jemc> but have a nice user experience and some flexibility to that experience so users can do the advanced things they need for their use case

20:21 <jemc> in my own personal roadmap for pony work, I'm planning to try to take up this mantle and start some work on this as soon as the `process` package is done

20:22 <jemc> ideally we can have a flexible-enough plugin system that everyone can be happy with how their dependencies are defined and fetched

20:28 <DanC> "nice user experience" might be a bit generous for nix. it took me quite a while to learn, and I'm still only comfortable with a small part of it.

20:39 <ponysaurus> jemc: A ByteSeq == (String | Array[U8] val). So appending to an existing ByteSeq is impossible since it's immutable if it were an Array[U8] val. Am I missing something?

20:43 <jemc> DanC: I'm talking mainly about the tooling/integration with the Pony environment - but I don't know hardly anything about nix workings at the moment

20:44 <jemc> which is why my first integration would probably use gx

20:45 <doublec> Is the process package being developed in a github repo somewhere?

20:50 <jemc> ponysaurus: a type always declares a "default" cap, but that doesn't stop you from using it with another cap

20:50 <jemc> for example: `let x: ByteSeq iso = recover iso [as U8: 71, 72, 73] end`

20:51 <ponysaurus> ahhhh .. ok! Didn't know that!!!

20:51 <jemc> that said, for ease of implementation, you might want to keep everything but the final ByteSeq as a val, and use iso only for the final one

20:52 <jemc> so once you start a new tail, `consume` the old one into a `val` in your list - should be no need to mutate it after that

20:52 <jemc> that's up to you though - I just mention because often `val` is easier to handle than `iso`

20:53 <ponysaurus> yes, this is exactly what I was hoping to do. I admit, code is a lot cleaner with `val` than `iso`

20:54 <jemc> doublec: as far as I know, the code for the process package isn't yet public