gmcabrita has quit [Quit: Connection closed for inactivity]
_whitelogger has joined #ponylang
Matthias247 has joined #ponylang
virtual_lark has joined #ponylang
virtual_lark has quit [Remote host closed the connection]
gmcabrita has joined #ponylang
obadz has quit [Ping timeout: 260 seconds]
obadz has joined #ponylang
_andre has joined #ponylang
staticassert has joined #ponylang
<staticassert>
does pony have a default pattern? Like _ in rust
kr1shnak has joined #ponylang
jemc has joined #ponylang
<jemc>
staticassert: we do have `_` which can be used in some places, and which we refer to as "don't care"
<jemc>
what are you trying to do?
<lisael>
in matchings, the else clause may be what you're looking for, staticassert
<staticassert>
I was thinking about the typed exceptions and 'elsematch' - it seems like that would just be better suited to _
<staticassert>
but maybe elsematch is the way to do it
<lisael>
mmmh sorry, typed exceptions is the new feature i didn't try yet. Just read the (cool) RFC. I fear I can't help right now
virtual_lark has joined #ponylang
cousin_luigi has left #ponylang [#ponylang]
amclain has joined #ponylang
nonick has joined #ponylang
nonick is now known as mabi
mabi has quit [Changing host]
mabi has joined #ponylang
<staticassert>
Yeah, I like the RFC. I was planning to comment on it suggesting some stuff from Duffy's blog but I Think those should be their own RFC.
Xyliton has joined #ponylang
<Xyliton>
Could someone help me with a small design problem? How should I model Websockets in Pony? I I implemented a subset of the RFC in pony already (no fragmentation, only textframes, and no masking or extensions) so I _can_ connect to a ws, but my code isn't really reusable and I intend to release it to the public ASAP because I saw some people discussing about websockets on reddit, but there were no results (yet)
<jemc>
Xyliton: can you ask a more specific question? (like, maybe calling out a particular design pattern in your library or in aturley's that you want some feedback about?)
staticassert has quit [Quit: Page closed]
<Xyliton>
What would be the most optimal (pony-) way of defining a ws lib? Should I do it the way tcp and http connections work and have a WebsocketConnectionNotify interface/class which is given to a WebsocketConnection, which handles the underlying tcp connection?
<Xyliton>
Or is there some other, better way of doing this?
<SeanTAllen>
Are websockets always over TCP Xyliton ?
<Xyliton>
I think so
<Xyliton>
I never did them over anything else, tbh
<Xyliton>
is http always over tcp? If so, then websockets should be too iirc
<Xyliton>
because at first they send a HTTP header telling the server that it want's to upgrade to websockets, and it wouldn't make sense if ws and http wouldn't share the same protocols otherwise
<SeanTAllen>
in that case, i dont think you needa WebSocketConnection
<Xyliton>
what should I do then?
<SeanTAllen>
I think what you probably want is to implement a TCPCnnectionNotify to handle the standard websocket parts
<SeanTAllen>
and have it have callback hooks for user specific code
<SeanTAllen>
int he same way TCPConnection has TCPConnectionNotify for user specific callback hooks
<Xyliton>
So I only add something like WebsocketNotify?
<Xyliton>
which is added to a TCPConnectionNotifyß
<Xyliton>
*?
<SeanTAllen>
Im not sure what the scope of "only add" is there
<Xyliton>
only the notify, instead of a "full" WebsocketConnection
<SeanTAllen>
you would have something like (ignore naming) class WebsocketHandler is TCPConnectionNotify
<Xyliton>
yes
<SeanTAllen>
and the websocket handler takes a "notify" for callbacks for user specific web socket handling code
_whitelogger has joined #ponylang
<Xyliton>
is it possible to get the host from a tcp connection?
<Xyliton>
but how do I handle sending data back? when I only write a tcp notify which takes a callback "object", wouldn't I have to pass down the raw tcp connection to the custom notify?
<Xyliton>
people would have to build frames all the time
<jemc>
Xyliton: you'd have to use the `sent` and `sentv` methods of the `TCPConnectionNotify` to encode the websocket frames as raw byte sequences (through whatever transformation process websockets use - I don't recall right now)
<jemc>
this is how `net/ssl/SSLConnection` works
<Xyliton>
ohhh, that's a handy function. Thanks!
<Xyliton>
So I can make the custom notify send strings only, and the tcp notify can wrap them in a frame?
<jemc>
in `pony-zmq` I ended up using an intermediate actor, but if you could do it all in a `TCPConnectionNotify` object, it would be preferable
<jemc>
Xyliton: yeah, that would be the idea
<Xyliton>
I got a problem though
<Xyliton>
`sent` takes a ByteSeq
<Xyliton>
but my frames are `Array[U8]`
<Xyliton>
wait, nevermind
<Xyliton>
I derped
<Xyliton>
How do I go from ByteSeq to string?
<jemc>
Xyliton: `type ByteSeq is (String | Array[U8] val)`
<Xyliton>
ohh
<Xyliton>
so I just have to match against that?
<Xyliton>
so either the byteseq itself or the String.from_array of it?
<jemc>
eventually, we'll be able to get rid of the `else` case, when the compiler is smart enough to recognize simple cases of exhaustive matches
<SeanTAllen>
I think before diving into too much design, it would make sense for you to try implementing using existing TCPConnection related stuff and then where you need to hide things etc should become a lot more clear.
<Xyliton>
I already got a simple ws connection working
<Xyliton>
but now I want to abstract some of the sending away
<Xyliton>
so I can more easily implement fragmented messages
<Xyliton>
if I want to convert from a sent to a sentv I have to return an empty string and call conn.writev inside the callback instead, right?
<SeanTAllen>
return an empty string?
<jemc>
Xyliton: yeah, that sounds right
<Xyliton>
according to the tcp connection docs returning an empty string in `sent` "blocks" the message
<SeanTAllen>
do you want to swallow data?
<Xyliton>
I have to swallow data
<Xyliton>
because it turns from a simple string into a frame
<SeanTAllen>
if you expose the tcp connection to users
<SeanTAllen>
they might call write
<SeanTAllen>
and if you return "" from sent
<SeanTAllen>
their data will be thrown away
<SeanTAllen>
silently
<jemc>
SeanTAllen: he's going to call `writev` within the callback
<Xyliton>
^
<SeanTAllen>
i understand
<SeanTAllen>
but we were previously discussing whether to expose TCPConnection to the user
<SeanTAllen>
if you do "" from sent
<SeanTAllen>
you should not expose TCPConnection to the user
<Xyliton>
that data swallowing is part of my TCPCOnnectionNotify
<Xyliton>
which implements the websocket RFC (kinda)
<SeanTAllen>
in which case you should not expose the user to TCPConnection
<Xyliton>
why?
<Xyliton>
isn't the write from the connection calling sent from my callbacks?
<SeanTAllen>
because if they call "write" on the TCPConnection, you are throwing their data away
<Xyliton>
I thought it calls my callback first?
<jemc>
SeanTAllen: the data is not getting thrown away, it's getting transformed, just as SSLConnection does
<SeanTAllen>
perhaps i need to see the code
<SeanTAllen>
I thought Xyliton was using TCPConnection and TCPConnectionNotify
<Xyliton>
yes
<SeanTAllen>
and exposing TCPConnection to folks
<SeanTAllen>
so what happens if I call write on TCPConnection?
<jemc>
SeanTAllen: he is, just as `net/ssl` does
<Xyliton>
I believe it would call the `sent` of my websocket tcp notify
<SeanTAllen>
websocket tcp notify is a TCPConnectionNotify?
<Xyliton>
yes
<SeanTAllen>
and what happens in the sent method of that notify?
<Xyliton>
it transforms the ByteSeq into a frame, writev's that to the connection and then swallows the original data
<SeanTAllen>
ok
<SeanTAllen>
i misunderstood what you were saying
<SeanTAllen>
got it
<Xyliton>
no problem. I'm talking confusing stuff most of the time
<Xyliton>
lol
<Xyliton>
I guess you could call me a noob when it comes to programming too. I only have some experience with Lua, Java, Ruby and Haskell (and some Lisp, but that isn't really "learning"). I would like to learn more, but school is taking up most of my time
<SeanTAllen>
Nothing wrong with being inexperienced
<jemc>
Xyliton: possible snafoo you might run into:
<jemc>
if your `sent` method transforms, and calls `writev` with the transformed payload, then your `sentv` method should *not* transform, since it would mean data is getting transformed twice. However, this means if a user calls writev directly, their frames won't get transformed as needed
staticassert has joined #ponylang
<Xyliton>
How do I append an array of strings, with an unknown length? Do I use a for loop and an aggregator var for this?
<Xyliton>
I intend to share my ws code with the pony community, so if anyone needs to writev to the connection directly, instead of sending strings, they may fix that
<jemc>
it would seem the way out would be to delay the transformation until the `sentv` method, so your `sent` method would simply call `writev([orig_byteseq])`
<SeanTAllen>
jemc: thats a nasty area of that API that we need to address in some fashion.
<Xyliton>
maybe an optional bool which which allows you to send "raw" messages?
<Xyliton>
which won't get transformed
<SeanTAllen>
re: How do I append an array of strings, with an unknown length? Do I use a for loop and an aggregator var for this?
<jemc>
ah, actually I'm wrong, the API already exists to do this
<SeanTAllen>
what is of unknown length?
<jemc>
Xyliton: you can use `write_final` inside your notifier to avoid hitting any other notifier methods
<jemc>
however, we probably need a `writev_final` for folks who want to take advantage of the `writev` syscall...
<Xyliton>
As long as I receive frames with the `final` bit set to 0 they count as fragments which I have to append together to get the full content. This happens when the content is too big for one frame, or the server you connect to just likes fragments
<Xyliton>
the content should be appended to the first frame with `final` set to 1
<Xyliton>
should I just store a string in my notify which get's appended to when I encounter a final=0 frame, and notifies the "higher-level" notify once it receives one with final=1?
<SeanTAllen>
im assuming that is connected to your earlier question but i'm having a hard time connecting them
<Xyliton>
I want to notify the user only when I encounter "complete" frames
<Xyliton>
fragmented ones should be buffered until I get a "finish" frame
<SeanTAllen>
i understood that much
<SeanTAllen>
ok
<Xyliton>
and I have to append the content of every fragmented frame, which is then sent to the user
<SeanTAllen>
have you looked at Writer in the buffered package?
<Xyliton>
so appending to a string isn't a good/correct way of doing this?
<SeanTAllen>
and also Reader
<Xyliton>
I'm using Reader to parse every frame already
<SeanTAllen>
if you keep appending to a string
<SeanTAllen>
then unless you have allocated enough memory
<SeanTAllen>
it will keep allocating memory on each append
<SeanTAllen>
and that isnt good for performance
<Xyliton>
I would have to increase the size of the writer too, wouldn't I?
<Xyliton>
or atleast the writer has to increase the size of it's underlying array
<SeanTAllen>
that is going to happen less often
<Xyliton>
why?
<SeanTAllen>
thats a longer explanation
<jemc>
Xyliton: it's a common performance idiom in such situations to store as an array/list of byte sequences, rather than a single byte sequence that keeps getting expanded
<Xyliton>
okay
<SeanTAllen>
if you want to learn more about it
<SeanTAllen>
i would invent a few scenarios for strings that arrive
<SeanTAllen>
and see what happens with memory allocations for both writer and for appending to a String
<SeanTAllen>
just mentally walk through the algo and see the difference in how they work
<jemc>
incidentally, this performance idiom is the reason why writev exists
<Xyliton>
would I store the writer itself as replacement for _current_content?
<jemc>
Xyliton: using the `buffered.Writer` would be one option, or using an `Array[ByteSeq]` or `List[ByteSeq]` would be another
<jemc>
(that's what `buffered.Writer` uses internally)
<Xyliton>
and String.from_array turns such an array into the concatenated "version" of it?
<jemc>
then you could pass that array/list directly to the user in the `WebSocketNotify` method, or you could concatenate as a final step, instead of concatenating at every step along the way
<jemc>
the former would be a true "zero copy" system, but arguably a little less convenient for some users than the latter
<Xyliton>
I prefer sending the data as Strings to the user
<Xyliton>
"zero copy"?
<jemc>
either would be more performant than concatenating continuously at every step along the way
<SeanTAllen>
zero copy == copies memory zero times
<Xyliton>
but I have to increase the size of the array too, don't I?
<SeanTAllen>
allocating memory and copying things from one region of memory to another is slow
<SeanTAllen>
unless you allocate enough memory for a string ahead of time
<Xyliton>
oh
<SeanTAllen>
then appending to it results in new memory being allocated
<SeanTAllen>
and the old memory being copied to it
Matthias247 has quit [Read error: Connection reset by peer]
<jemc>
the approach of concatenating as a final step would be "single copy", if you implement it in such a way that you make sure to allocate the right size String buffer at the start before starting your concatenation
<jemc>
that is, you should create a `String(total_size)`, where `total_size` is the sum of the size of all frames
<Xyliton>
I can't concatenate it with String.from_array?
Matthias247 has joined #ponylang
<SeanTAllen>
`from_array(data: Array[U8] val)`
<SeanTAllen>
note
<Xyliton>
would I create that string first and then append with "str = str + data" in a loop?
<jemc>
`String.from_array` doesn't do concatenationg - it "moves" a buffer from an `Array[U8]` object into a new `String` object - it can't concatenate multiple buffers
<SeanTAllen>
from_array takes an array of U8
<SeanTAllen>
what jemc said, better than i
<SeanTAllen>
str = str + data in a loop is many copies
<jemc>
Xyliton: close - you'd use `str.append(data)` in a loop
<jemc>
Xyliton: what you probably want to do is change your `new create` to `new iso create` - all params will need to be sendable so you'll want to change `WebsocketNotify` to `WebsocketNotify iso` as well
<SeanTAllen>
your WebsocketNotify shouldbe an iso
<SeanTAllen>
new create(host': String, target': String, notify': WebsocketNotify iso) =>
<Xyliton>
why should it be iso though?
<SeanTAllen>
_notify = consume notify'
<SeanTAllen>
im guessing
<SeanTAllen>
there's not enough code here for me to compile and get your error
<jemc>
Xyliton: it needs to be iso because it needs to be sendable, and needs to be mutable - iso is the only option that meets those requirements
<Xyliton>
do I have to make the functions in websocket_handler iso too?
<Xyliton>
wait, I can't. they are ref already
<SeanTAllen>
can you paste your code where you are creating your WebsocketHandler?
<SeanTAllen>
will default to creating an iso by default
<SeanTAllen>
your websocket handler is being passed to another actor so it isnt safe to be a `ref`
<SeanTAllen>
sorry
<SeanTAllen>
i see you arent using the test one
<SeanTAllen>
but you want the same change in websocket handler
<SeanTAllen>
new create(host': String, target': String, notify': WebsocketNotify) =>
<SeanTAllen>
should be
<SeanTAllen>
new iso create(host': String, target': String, notify': WebsocketNotify) =>
<SeanTAllen>
the important part of this though is to understand why you need an iso and not a ref
<Xyliton>
I got "new iso create(host': String, target': String, notify': WebsocketNotify iso) =>" there right now
<SeanTAllen>
its not safe to share a ref across actors so pony won't allow you to do it
<SeanTAllen>
if you need a mutable that you can send from one actor to another it needs to be an iso
<SeanTAllen>
which means you have to consume it before sending to another actor
<SeanTAllen>
and by consuming, you are giving up access to it and it would be a compile error to try to use it after consuming it
<SeanTAllen>
make sense?
<SeanTAllen>
so for example you will see with TCPConnection that the TCPConnectionNotify is an iso
<SeanTAllen>
because it would be created in one actor and then given to the TCPConnection which is another actor
<Xyliton>
oh
<SeanTAllen>
make sense?
<SeanTAllen>
so your error was "hey, you are trying to send a shared mutable thing where you need a unshared mutable thing"
<staticassert>
Are there timeouts in pony? If I send a message from actor A to B and actor A expects some data back from B, is there a way to handle actor B not sending the message back?
<SeanTAllen>
why would actor B not send a message back?
<jemc>
staticassert: check out `Timer`, `Timers`, & `TimerNotify` in the `time` package
<staticassert>
If it's on another computer, and that computer lost power, I guess
<SeanTAllen>
ah
<SeanTAllen>
ok
<SeanTAllen>
thats very different than in process
<staticassert>
I'll check out Timer
<staticassert>
SeanTAllen: totally
<SeanTAllen>
so there's no distributed pony right now
<SeanTAllen>
so really, this is a network actor sending to another network actor
<staticassert>
yeah, is that a defined concept in pony yet?
<SeanTAllen>
in another process
<SeanTAllen>
there are many ack strategies from that
<staticassert>
like in Erlang actors model systems pretty well, so handling a computer crashing is the same as handling any local actor
<SeanTAllen>
are you looking for "the message was received on the other machine" or "the message was processed"?
<staticassert>
SeanTAllen: processed
<SeanTAllen>
local actors don't fail like that in pony
<staticassert>
SeanTAllen: I know
<SeanTAllen>
for processed, you'd need your own system to handle that
<SeanTAllen>
we built one for Wallaroo
<staticassert>
But like let's say I send a mesasge to B from A, and I want to get a message back with a result - not just a reply, but the result of a computation
<jemc>
staticassert: the fundamental difference from Erlang is that Pony actors aren't allowed to wait within a behaviour
<SeanTAllen>
that meets the given constratints of our system
<staticassert>
jemc: right, which I like actually
<staticassert>
I'm not an expert in erlang but nesting receives felt icky
<SeanTAllen>
so within the same process staticassert, you'd use a Promise or a callback
<SeanTAllen>
there's no set way for handling that across processes at the moment
<staticassert>
I see, ok
<staticassert>
It feels like Erlang models the real world a bit more - in Pony an actor can never fail, so it can't really model a real system because all real systems can fail
<SeanTAllen>
we ended up writing our own network actors based on tcp connection (but not actually tcp connection) to handle cross process acknowledgemetns
<staticassert>
Which isn't to say that the tradeoff isn't worth it
<staticassert>
interesting
<SeanTAllen>
i would disagree with you erlang models the real world a bit more statement
<staticassert>
How so? I'm not an expert in Erlang or Pony - and I don't mean to knock one or the other to be clear :P I'm just curious
<SeanTAllen>
different choices were made and they have different consequences. i dont think it has anything to do with modeling the real world.
<staticassert>
That's fair. I think a big part of Armstrong's model was inspired by the real world though - he talks a lot about causality and emulating physics with the actor model.
<SeanTAllen>
i think a reasonable statement would be, erlang allows for failures in a distributed system and has established patterns for dealing with them
<staticassert>
That's definitely true, yes.
<SeanTAllen>
back to time outs though
<SeanTAllen>
if you want to do something after X amount of time
<SeanTAllen>
then yes, you want a Timer like jemc said.
<SeanTAllen>
be wary using them, starting up tons of them will eventually have a cost
<SeanTAllen>
its take a simple case
<SeanTAllen>
lets say you sent 50 messages from socket A to socket B
<SeanTAllen>
you could do a timer for each of those messages that will fire at the appropriate time for the timeout
<SeanTAllen>
but that's a lot of timers
Xyliton has quit [Ping timeout: 256 seconds]
<SeanTAllen>
perhaps what you would rather do is on each send ( or a subset of sends ), check for "time outs"
<SeanTAllen>
and augment it with a timer that fires from time to time in case of a lack of sends
<SeanTAllen>
that could also check waiting work
<staticassert>
hm, yeah seems like I could isolate that sort of work - like send a message from A to B and then send one to C, and C tracks my timeouts and, unless it gets an ack, executes some code
<SeanTAllen>
that would create extra work
<SeanTAllen>
which you may or may not want
<SeanTAllen>
ou have to send a number of additional messages in that case
<staticassert>
yeah
<SeanTAllen>
A can track its own time outs
<SeanTAllen>
but
<SeanTAllen>
you cant do that with TCPConnection
<SeanTAllen>
your idea is quite fine if you arent trying to push performance really hard
<SeanTAllen>
we have 4 or 5 different kinds of "TCPConnection" like things in Wallaroo to squeeze out best performance by having everything needed for resilience etc in that single network actor
<jemc>
just a quick note - the reason you have to put a `Timer` inside a `Timers` is because you can save some performance by putting multiple timers into the same `Timers`, instead of spinning up a new `Timers` for every `Timer`
<SeanTAllen>
excellent point Jemc
<staticassert>
For the kind of work I do performance is just way less of an issue I think
<SeanTAllen>
I'm on the extreme end of pony performance so... ya most things i end up thinking about in these cases probably dont matter to other folks
<staticassert>
:P
<SeanTAllen>
its hard for me to remember that
<staticassert>
I've had to sort of just give up on performance for work, so I'm more drawn to software reliability, which is much more of an issue that I see every day
<SeanTAllen>
I like to think of performance work as reliability at the planet energy usage level
<SeanTAllen>
im half joking
<staticassert>
yeah, that's fair
<staticassert>
I hate burning performance
<SeanTAllen>
at this point though, pony is a concurrent language, not a distributed one, so in terms of erlang and distributed relaiability, its head and shoulders above pony
<SeanTAllen>
or rather in terms of having strategies for dealing with errors
<SeanTAllen>
i appreciate that type systems can save me from a large class of errors
inara has joined #ponylang
<staticassert>
Yeah, I want to do an RFC for the abandonment pattern mentioned in Joe Duffy's blog... *or* keepers, but I'm not sure yet
<staticassert>
I liked typed exceptions as a way to handle expected failures, but I think that div by zero and other such unexpected things (assertions) should be handled differently
<jemc>
staticassert: did you listen to this past week's sync call?
<jemc>
I made an argument that its hard in a library to know what is an abandonment-level failure for an application
<jemc>
and advocated for shifting abandonment as a choice for the caller, and by convention libraries typically would not use abandonment, only applications would - with an option for the person doing the compiling to whitelist which packages can use abandonment, to enforce this (mush like we do for FFI)
<SeanTAllen>
i wish i hadn't missed that sync jemc. I agree with your basic points.
<jemc>
syntax idea I floated was using `!` in the invocation, similar to the current open RFC to require `?` in invocations of methods that could raise an error: https://github.com/ponylang/rfcs/pull/82
<jemc>
that is, the idea was that you could use `!` instead of `?` in the invocation, to call the method "confidently", effectively asserting that an error would not be raised, so it would not be a partial call in the type system, and would abandon at runtime on failure
<SeanTAllen>
thanks for the retweet staticassert
<SeanTAllen>
jemc: im very much on board with the general principle. do not like libraries making those kinds of decisions and i think its important for the language to make it easy to return that level of control to the application.
m6w6_ has joined #ponylang
<SeanTAllen>
that application level of control is one of the things i love in Smalltalk and CLOS's condition system
<staticassert>
jemc: Yes, I did listen. I agree, by convention a library should never raise an abandonment level failure. I still think that there are some cases where that wouldn't be the case, and it should be expected that libraries will, sometimes, raise that level of error.
<staticassert>
SeanTAllen: happy to
<SeanTAllen>
neither of which is directly applicable to Pony but, i love that you are advocating for something in that spirit
<staticassert>
Rust has a similar 'libraries should never panic' rule, but libraries *do* panic sometimes. I don't think it's worth trying to solve that problem. it's better to make handling abandonment very natural.
<staticassert>
Rust can't do this, I think pony can, because actors are so ubiquitous and are such natural bounds for abandonment
<staticassert>
Duffy makes a similar argument I think
<SeanTAllen>
I love her music
<SeanTAllen>
well some of it
<SeanTAllen>
ill show myself out now
Xyliton has joined #ponylang
<staticassert>
if only I knew who that duffy was lol
<jemc>
staticassert: there are always "exceptions to the rule" when it comes to prudent reasons to break a convention - however, I think the convention still has value
<staticassert>
jemc: 100% agree that the convention should be 'library code should never raise an abandonment error'
<Xyliton>
I just tried to use the json package in my code but it tells me that there are clashes with the types in the package and my code. I'm 99% certain that I'm not defining any of these types though.
<jemc>
that is, rust isn't wrong to have a convention like that, even though libraries may sometimes have good reasons to break it
<jemc>
yeah
<staticassert>
totally
inara` has quit [*.net *.split]
m6w6 has quit [*.net *.split]
m6w6_ is now known as m6w6
<jemc>
I do think that the user who is developing/compiling the application should have the tools to have the final say in enforcing that convention, if they choose, though
<jemc>
which is why I suggested a whitelist of packages, the tactic we use to empower users to limit FFI (which is similarly dangerous in that it can crash the program)
<SeanTAllen>
Xyliton: can you gist the error. i'm not sure what you mean by "clashes"
<staticassert>
jemc: this is something that they've talked about in the rust ecosystem as well - determining if a library can panic, and whitelisting based on that. I think it's really hard to do
<staticassert>
I think FFI is much more dangerous since it impacts stability in a far more wide reachign way, as well as security
<jemc>
staticassert: what do you mean specifically by "hard to do"?
<Xyliton>
FFI is the "easy way out" sometimes though
<staticassert>
jemc: determining that a library will never raise an abandonment error without introducing false positives where that abandonment could never be reached
<SeanTAllen>
Xyliton: can you gist the current "testy.pony"?
virtual_lark has quit [Remote host closed the connection]
<jemc>
staticassert: well, if we had the kind of static analysis that could prove the "unreachables" were truly unreachable, we wouldn't need the abandonment errors for those to begin with
<staticassert>
Totally, but then what do you base the whitelisting off of? The presence of any abandonment in a library?
<SeanTAllen>
Xyliton: try deleting the second use "json". if that works, can you open an issue for the confusing error messge?
<Xyliton>
oh, damn. I had two use "json" in there
<jemc>
whitelisting a package for abandonment would effectively mean that "I trust the package author to use abandonment appropriately/safely", just as it means for FFI - or in other words, it's an expression of trust that the unreachables are truly unreachable
<Xyliton>
it compiles now
<Xyliton>
thanks for spotting that <3
<staticassert>
Yeah, I ran into that yesterday - the error message around double imports is rough
<SeanTAllen>
Xyliton: your welcome, please open an issue
<Xyliton>
is it an issue though?
<SeanTAllen>
staticassert: open an issue for those
<SeanTAllen>
yes
<SeanTAllen>
all confusing error messages are to be considered bugs
<jemc>
as SeanTAllen said some time ago (paraphrasing), "you'd be surprised how often those unreachables end up being reached in practice"
<SeanTAllen>
haha
<SeanTAllen>
i do say variations on that
<SeanTAllen>
quite a bit
<staticassert>
jemc: what's the purpose exactly? If abandonments can come from anywhere (your application code or a trusted library or an untrusted library) and are all handled the same way (div by 0 should be handled the same way as an assertion error, etc)
<staticassert>
like why say you trust a package not to raise an abandonment if anyone writing an application must accept that abandonments may be raised
<staticassert>
hopefully I'm making sense :P
<jemc>
well, I guess I disagree fundamentally that "anyone writing an application must accept that abandonments may be raised"
<jemc>
I also disagree that things like div-by-zero or array-out-of-bounds are *always* grounds for abandonment, as Duffy suggests in the post
<staticassert>
I think div-by-zero is kind of an ideal example, since it's so relevant to pony
<staticassert>
the fact that it returns an arguably incorrect result due to the fac that it would be overly verbose not to indicates that there should be a way to handle bugs differently
<jemc>
let's say I create a calculator application, that accepts input via stdin and prints results to stdout
<jemc>
a user types in "42 / 0 # lol"
<staticassert>
and they get back '0' - an incorrect result
<staticassert>
ideally you'd check the divisor first and handle that gracefully with 'you cant div by zero' - otherwise, you should respond with some generic error "invalid expression" in the case of an abandonment error
<staticassert>
42/0 not being handled is a bug, bugs shouldn't lead to incorrect results, bugs ideally should not impact primary operation - the calculator should be usable after that expression is evaluated
<staticassert>
*in my opinion* of course, I'm stating these things like facts when they are not :P
Xyliton has quit [Ping timeout: 258 seconds]
<jemc>
it's pretty interesting to me that you pick divide-by-zero as your ideal example for abandonment, when in fact it is an ideal example for a "partial function" (in mathematical terms)
<staticassert>
I guess I don't think whether it's partial or not is relevant to whether it is a bug or not
<staticassert>
with my limited understanding of a partial function being a function on a set that does not map every element to another ?
<jemc>
that is, zero is not in the valid domain for the divisor of the division function - so the function is partial, it does not have a result for all valid inputs
<jemc>
staticassert: right, that's the basic concept
<staticassert>
But we can agree that making division a partial function in pony (where partial functions must be handled with try) would be incredibly unergonomic
<staticassert>
so there must not be a direct 1 to 1 in terms of whether being partial maps to how we handle errors
<jemc>
it would be unergonomic, it's true - but accepting that bad math can crash your application (or actor or whatever) isn't the only option
<staticassert>
I think it's worth noting that actor/ application would be really different - the isolation is totally key
<staticassert>
iny our calculator app the program would not shut off, it might just clear the screen and say "Invalid expression"
<staticassert>
In the case of *good* code that checks for 0 before dividing, it could say "Invalid expression: 42 / 0. Can not divide by 0"
<staticassert>
jemc: but before I interrupt you further, what would the other option be?
<jemc>
one option is to adjust the semantics of the function so that it's no longer partial - for example, saying that "this function returns the result of dividing the first operand by the second operand, unless the result of division would be undefined, in which case zero will be returned"
<jemc>
which is the option that was chosen for pony so far
<staticassert>
but then you're conflating 0, a number with meaning of its own, with errors
<staticassert>
Now I don't know if my 0 actually goes into 42 0 times or if I got an error
<jemc>
well, as you said before "good code" would check first if it matters to them
<jemc>
under your proposal (which I'm making assumptions about because I haven't seen it yet), I don't know whether some math logic deeply nested in a method I'm calling is going to panic my actor and make my handle an error I have no idea about how to handle at my actor level
<staticassert>
so what should bad code do? If it returns an incorrect result in some cases, how would I ever trust that code?
<jemc>
the non-locality of it is what troubles me the most
<jemc>
staticassert: it's not an incorrect result, its a correct result of a function with different semantics from pure mathematical division
<staticassert>
I guess
<staticassert>
That's really unexpected behavior though
<jemc>
`div` could probably use a docstring that explains the semantics, that's for sure
<staticassert>
In terms of non-locality, the issue is 'how do I handle errors I can't reason about', fundamentally, right? They're coming from somewhere unknown, and you don't know what they are.
<staticassert>
So I would say you handle them all the same way. Or, if they're typed, the keeper pattern (with a default case) would be interesting.
<staticassert>
idk, I think there's a lot to discuss here
<jemc>
quick question - do you propose using the same treatment for array-out-of-bounds - I think that's actually a better example for this discussion, since it's currently partial in pony instead of using the change-the-semantics workaround that div uses
<staticassert>
jemc: yes
<jemc>
so let's talk about that, then
<staticassert>
Sure
<jemc>
I don't agree that array-out-of-bounds in general falls into the category of "errors I can't reason about"
<jemc>
if you're disconnected from the call site, then yes, it definitely appears to be that way, in that you can't reason about what happened or why
<staticassert>
Well, so I think it sort of goes back to ergonomics. Can I do foo[1] in pony? Or do I need to use a .get()
<jemc>
but the same could be said of any error, the more disconnected you are from the call site, the less able you are to reason about it, so there's not really anything meaningful you can do to recover
<staticassert>
jemc: you can roll back a transaction, you can reset your state, you can send it to a logger, etc
<jemc>
but at the call site, you (as a programmer) have the best shot at handling the error (or at least knowing that you don't have enough info, and passing it off to some kind of "keeper" to handle)
<staticassert>
true
<jemc>
this concept/convention of handling errors as locally as possible is very central to the design of Pony right now - the greater awkwardness of "passing the buck" up trumps the lesser awkwardness of handling it locally, so we force programmers to handle their problems locally
<jemc>
err, not really force in an absolute sense, but "incentivize" I should say
<jemc>
however, at an application level, I've totally had the practical need to make an assertion that "this error will never be raised"
<jemc>
I can do this at the application level, because I'm the application programmer, and I trust myself
<staticassert>
What happens when you're wrong?
<jemc>
well, some form of abandonment is what we're proposing to happen, right?
<staticassert>
right
<staticassert>
hm
<jemc>
so, I implicitly trust myself and my application to make assertions like that, but in my whitelist proposal, I could choose to enforce that no other packages make such assertions, except ones that I explicitly trust to be correct
<staticassert>
so let's go back to you writing your application
<staticassert>
and you write your assertion, right? I think in the sync meeting the compiler was mentioned as an example.
<staticassert>
How would you handle an assertion failure?
<staticassert>
assertion failure that leads to abandonment*
<jemc>
it's very analogous to FFI in that both cannot be "proved" statically by the compiler to be correct/safe, so if I'm paranoid I need a mechanism to grant explicit trust to packages that I believe took the proper precauations to be safe
<jemc>
staticassert: so far, I've been making it crash my entire program, and print info that would be useful in tracking down the issue
<SeanTAllen>
same
<jemc>
but I personally would recoil in horror if a library package that I didn't trust was doing the same
<staticassert>
What do you think about abandonment being bounded by actors?
<SeanTAllen>
same
<staticassert>
at the actor level*
<staticassert>
since actors share no state, one actor having a bug shouldn't really impact another actor too badly
<SeanTAllen>
i dont see how that would work in Pony.
<SeanTAllen>
because does this actor go away?
<SeanTAllen>
or does it magically reset it state?
<SeanTAllen>
it cant go away
<SeanTAllen>
there could be references to it
<staticassert>
I think those are sort of separate questions - we can agree it can't go away
<staticassert>
like what it does exactly we can ignore for now - just talking about the premise of actors being isolated
<staticassert>
they are already isolated in terms of state, so shouldn't fault isolation sort of follow?
<SeanTAllen>
im not sure i want code other than what i write resetting the state of the actor
<staticassert>
I'm not saying it would reset automatically, or that it would have to be handled in one way (or at all necessarily)
<staticassert>
but like, there is the possibility that you could bound it at the actor level, without worrying about your entire program state being screwed up
<SeanTAllen>
in theory i would agree with that
<SeanTAllen>
but
<staticassert>
that's kind of the premise of abandonment, to me, that it works when faults are isolated
<jemc>
in my opinion, we're basically talking about two extreme ends of the spectrum here - errors are handled as locally as possible, and abandonment is saying that isn't possible to do because your program wouldn't know how to proceed, so you crash the entire program
<jemc>
so stopping at the actor level is a "compromise" that doesn't really satisfy either concern
<staticassert>
In Duffy's post abandonment only kills the actor
<jemc>
that is, in general, it doesn't seem local enough to meaningfully handle the error, but I'm also not thinking that in general we can just expect an actor to trudge along happily after a "this should never happen" scenario
<jemc>
of course, I'm interested to see your actual proposal to talk more specifically about it
<jemc>
just talking in generalities at the moment
<SeanTAllen>
and for me
<SeanTAllen>
after any proposal, i dont think i'd have any real response without thnking through a variety of scenarios
<staticassert>
jemc: If the error is transient - like just that one message messed it up - why crash the program?
<staticassert>
Yeah, I'm enjoying the discussion - I would want to come up with multiple motivating examples and really dig into this.
<SeanTAllen>
in the Sendence case of why we crash the program, because hitting those scenarios mean we have hit a "this can not happen" segement of code, which means there is a bug
<staticassert>
But abandonment in the 'crash the entire program' would never work for a service that actually has to serve multiple users. One user sending malformed data should not crash the other user's session.
<SeanTAllen>
and we want to make the bug very obvious
<SeanTAllen>
in our case, we should never end up in a "this can not happen" scenario for malformed input.
<SeanTAllen>
thats normal error handling
<staticassert>
I mean, let's say you have a server that's got ttwo sessions open. User A sends some malformed JSON message, triggers a "this can not happen" scenario (maybe in some FFI code?), and the server goes down. Now User B's session is gone too evne though they should be totally isolated.
<SeanTAllen>
any code that is in a user input path would never be "this can not happen"
<staticassert>
In the presence of abandonment, if adopted, or FFI as is, you couldn't really guarantee that though
<SeanTAllen>
couldnt guarantee what?
<staticassert>
that some code on the user input path wouldn't contain an assertion
<SeanTAllen>
you should never assert on input from a user
<jemc>
staticassert: this is also a big reason why I'm opposed to reaching for abandonment in as many cases as you seem to be advocating for (divide-by-zero, array-out-of-bounds, etc) - under that kind of broad proliferation of abandonment, your scenario seems much more plausible than it is currently
<staticassert>
agree
<staticassert>
I would only want abandonment if I knew it could be isolated, otherwise it seems way too dangerous
<jemc>
and I've always seen it as one of the strengths of pony being that it makes you handle errors locally in such a way that this happens much more rarely
<SeanTAllen>
same
<staticassert>
agree jemc
<jemc>
if abandonment is (by convention, and enforcable via whitelist) only happening at the application level, it would be pretty obvious to you that you have a possible abandonment case in your user-input-handling path, and you'd want to take a look at reevaluating your design, I think
<staticassert>
Exposing errors upfront makes a lot of sense, and is ideal for libraries, I wonder if it's ideal for application code though.
<staticassert>
On a team of multiple people idk about that
<staticassert>
And then it's like... I would never rely on teh whtielist - you'd be splitting the ecosystem I think
<staticassert>
why would I ever, ever use a library that might crash the entire program?
<jemc>
I also agree with you that if I had a server that keeps "persistent" things in memory without persisting them to disk, you wouldn't want "unexpected" errors to crash your program
<staticassert>
I wouldn't - no one would
<jemc>
in which case, you'd want to handle them in some different way that was application-specific
<staticassert>
jemc: a lot of it is also about communicating errors - this is something supervisor trees handle nicely, errors can be (safely) propagated across actor boundaries, and hten when you get to your top level actor you can just return a 500 error or some such thing
<staticassert>
though I think I've said before that a 1 to 1 port of the supervisor tree concept is probably not appropriate for pony
<jemc>
seems like it could be appropriate to handle abandonment at the actor level in *some* cases, if the mechanism is flexible enough to allow for the application-specific ways of recovering
<staticassert>
but on error I could be like 'hey I failed, send a 500 error back'
<jemc>
and maybe it would fall back to crashing the entire program if you didn't use that mechanism
<staticassert>
Yes, exactly
<staticassert>
I don't think there would be one way to handle it
<jemc>
yeah, seems appropriate to me
<staticassert>
Maybe there would be a default - crash the program? But if this actor is a secondary feature, wh ywould I?
<staticassert>
Like if my #1 goal is to serve the user a response, and my #2 goal is to provide them with the info they requested, crashing the server is not viable
<jemc>
well, if there's no meaningful way to recover the actor
<staticassert>
but I coudl totally just be like "hey sorry i messed up" without impacting the primary goal
<staticassert>
but working on distributed systems a server going down would make me very sad, but a server responding a 500 is awesome
<jemc>
I've definitely used production systems where crashing the process is viable, because the processes are redundant, and they are automatically restarted at the OS level
<staticassert>
a working server is nice too :P
<staticassert>
jemc: sure, but actors can be redundant :P
<staticassert>
So you can get that nice stability of multiple machiens, but with multiple actors
<SeanTAllen>
ive worked in systems where we turned the gc off an made sure our state would survive an OOM
<staticassert>
:o
<staticassert>
we've never gone that far here lol
<SeanTAllen>
its one way to tune a gc
<jemc>
also, I think not every possible program fits into the client-server HTTP-style model you're talking about, so it's not a totally universal paradigm to design around in a general purpose language, I think
<staticassert>
I'm having too much fun discussing this though, I need to head home :P thanks for indulging me, I'd be really happy to talk about this during a sync meeting or something
<jemc>
still, I think I'd like to see the proposal itself to discuss more specifically
<staticassert>
jemc: 100% agree, but I'm also not advocating a single action, just *some* way to handle it at a reasonable boundary
<staticassert>
jemc: sure, I've never written an RFC before - could I do something rough and then send it to you or SeanTAllen for some eyes? Or shouuld I just propose it?
<jemc>
and I'm still resistant to reaching for abandonment for things like divide-by-zero or array-out-of-bounds, that can reasonably be handled locally my some (most?) programs
<staticassert>
by rough I mean the best I can do but it will likely just be rough due to experience :P
<staticassert>
jemc: yeah, fair
<staticassert>
I chose those two because convention in other programs is to throw a runtime exception or something equivalent
<staticassert>
in other programming langauges*
<staticassert>
but they both boil down to assertion errors
<jemc>
I'd say just propose it - if you want to ask for feedback privately, you can, but there's no harm in putting a work-in-progress RFC up in an RFC PR and just making it clear that it's still in progress
<staticassert>
cool
<staticassert>
thanks for the discussion - I'll probably be on later
<staticassert>
cya
staticassert has quit [Quit: Page closed]
<SeanTAllen>
if my life settles down some jemc, i hope to be able to return to the runtime backpressure work in a few weeks
<SeanTAllen>
it showed great promise but i found problems
<jemc>
`Promise[Backpressure]`
<SeanTAllen>
hehe
staticassert has joined #ponylang
<staticassert>
i dont get how i can never manage to find that exceptions RFC - I go to github, type 'exceptions' into the issues search bar, and I don't see it :\ I do the same thing in pull requests, nope
<jemc>
staticassert: type it in the search bar, then click the "issues" tab on the left
<jemc>
by default the search bar only searches code, even if you search from the issues or pull requests page
<staticassert>
oh there's a separate repo for rfcs
<jemc>
oh, that too :P
<staticassert>
yeah I think I was linke dto it before, and then i manually entered the RFC number becuase I was listening to the sync call