<asheesh>
Let me sketch this out in non-capnp right now in an etherpad, one sec
<maurer>
Another approach rather than trying to read the protocol and design the capnp from there might be to read the abstractions real irc clients use
<maurer>
since many of them are probably a bit OO already, and may have some good ideas
<kentonv>
indeed
<asheesh>
I once tried to do that.
<asheesh>
The protocol makes that a giant mess due to the "can receive responses to requests you never made" thing.
<asheesh>
If other people know how to get around that, I'd love to hear it, though.
<kentonv>
relatedly, I was trying to read the IMAP spec last night and getting nowhere, so instead I read the Nylas mail sync API and that was way more productive.
<asheesh>
(-:
<asheesh>
I do love IMAP I admit.
<asheesh>
But we do also have many friends at Nylas.
<kentonv>
well I was reading their API specifically to understand what features IMAP has
<asheesh>
Ah hah (-:
<asheesh>
UIDVALIDITY AMIRITE
<kentonv>
API references are much easier to read than RFCs
<asheesh>
I think I learned how IMAP works by understanding offlineimap.
<digitalcircuit>
dwrensha: yeah, I've been having my Groovebasin grains randomly shut down. Either the web socket connection fails, or I get the same Sandstorm HTTP bridge error as you did. I just joined #libgroove in case it's more of a Groovebasin issue.
<kentonv>
I hope that someday people learn how IMAP works by reading our capnp translation. :)
<asheesh>
rfc2capnp
<ckocagil>
has anyone here used dillinger for sketching out specs?
<asheesh>
Oh, interesting
<kentonv>
digitalcircuit: Check the grain debug log after a shutdown, see if the supervisor thought it was not in use and killed itself.
<asheesh>
I might try that rihgt now then
<maurer>
So, you guys implemented email in house, and are planning on implementing irc in house - have you given any thought to making it so that such features (irc driver, email driver, etc.) could be submitted as plugins to a separate area of the app store?
<ckocagil>
it has real time collaborative editing, but without the fancy cursors of etherpad
<kentonv>
digitalcircuit: that said, sandstorm apps in theory need to be resilient to random disconnects and it's quite possible that groovebasin isn't.
<kentonv>
maurer: yes, when we have capnp APIs, any app will be able to export them and others to connect to them via the powerbox.
<kentonv>
maurer: some day I'd like the powerbox to be able to guide you to an app search for apps implementing a requested interface, even
<maurer>
kentonv: Hrm, I guess I'm missing something then, since I thought the IRC driver or the email driver were not implementable inside a grain
<maurer>
due to not having capabilities to speak generically on the network
<digitalcircuit>
kentonv: fortunately, I don't think it's that. Reading the debug log, the Sandstorm supervisor only shuts down the grain some time after playback stops. Usually it's one of the following errors: web socket error: Error: write EPIPE at errnoException (net.js:904:11) at Object.afterWrite (net.js:720:19)
<digitalcircuit>
Or: sandstorm/sandstorm-http-bridge.c++:433: error: exception = sandstorm/sandstorm-http-bridge.c++:420: failed: Failed to parse HTTP response from sandboxed app.; error = stream ended at an unexpected time
<digitalcircuit>
stack: 0x4cd2c0 0x469922
<digitalcircuit>
(Whoop, missed the extra newline at the end)
<asheesh>
... hi keturn
<asheesh>
I guess you wrote the Twisted file I am reading through.
<kentonv>
maurer: a grain could implement the interfaces backed by something other than the network. But also a grain can request an IP network capability, which typically only the admin has. Drivers are in fact normal apps that typically request such a capability.
<maurer>
kentonv: OK, cool, I didn't realize grains could request that capability
<maurer>
kentonv: if I install a grain that is requesting such a capability from the market, will I see some kind of alert letting me know about it?
<digitalcircuit>
I'm suspecting it's not Sandstorm's fault. I'll try debugging further with those in #libgroove.
<kentonv>
maurer: the grain has to do a powerbox request for the capability, which you can choose not to answer.
<maurer>
kentonv: Ah, so it's done at runtime rather than at install time
<maurer>
I guess that makes some amount of sense
<kentonv>
maurer: yeah, we don't do install-time requests. :)
<maurer>
(I'm still uncertain which is correct, but it's certainly defensible)
joshbuddy has quit [Read error: Connection reset by peer]
joshbuddy_ is now known as joshbuddy
<asheesh>
zarvox: FWIW if the goal is to make something as beautifully capnp-y as possible, then I would say the best thing to write on top of v1 of a driver would be an IRC log bot.
<asheesh>
The "funny" thing about the IRC driver is that the more beautiful and capnp-y it is, the more work it is to integrate with existing IRC clients.
<asheesh>
One would presumably end up running an ircd in the grain that the IRC client speaks to.
<asheesh>
(since the existing IRC clients really want to parse something)
bpierre_ has joined #sandstorm
<asheesh>
Howdy bpierre_
paroneay` has joined #sandstorm
garrison has joined #sandstorm
garrison has quit [Changing host]
garrison has joined #sandstorm
hunterm___ has joined #sandstorm
rollig has joined #sandstorm
maurer1 has joined #sandstorm
simonv3 has quit [*.net *.split]
paroneayea has quit [*.net *.split]
NhanH has quit [*.net *.split]
rolig has quit [*.net *.split]
garrison_ has quit [*.net *.split]
maurer has quit [*.net *.split]
hunterm__ has quit [*.net *.split]
bpierre has quit [*.net *.split]
rollig is now known as rolig
NhanH has joined #sandstorm
hunterm___ is now known as hunterm__
<zarvox>
maurer1: I believe that runtime requests are almost universally better than install-time requests for several reasons:
<zarvox>
1: the user has more context about how the permission being requested will be used
maurer1 is now known as maurer
<zarvox>
2: the user has a decent UX for substituting an alternate implementation if they so desire
<maurer>
zarvox: #1 makes sense for some permissions but not others imo - it makes a lot of sense for e.g. gps or camera, things that do not usually need continuous use
<zarvox>
3: app developers are discouraged from assuming the availability of the privilege
<maurer>
#2 I don't understand at all
bpierre_ is now known as bpierre
<asheesh>
Like Mock Location on Android, but per location request, not globally for your whole phone
<zarvox>
Assume there are multiple such permissions that an app would request - getting bombarded by all of them at install time and having to replace each one is confusing
simonv3 has joined #sandstorm
<zarvox>
And there are situations where "up-front" permission grants are suitable - in particular, if the user won't be around to grant the privilege in context, like "Find my phone/remote wipe" functionality
<asheesh>
Howdy simonv3
<asheesh>
nwf: btw please e-meet one of the powerboxers, namely zarvox ^
<zarvox>
If the request/grant are part of the user's workflow already, then you have a decent opportunity to e.g. provide a mock location, or a mock ipnetwork driven by a pcap file, or something else
<zarvox>
if it's just "gets the privilege if installed" then you have to make all those choices up-front, and you can't revise them at runtime either
<asheesh>
zarvox: nwf is a cs.jhu.edu friend of mine who is one of the last people alive excited about Plan 9
<asheesh>
I might be spinning that wrong
<zarvox>
hi nwf! pleased to e-meet you!
<nwf>
EHLO!
<asheesh>
nwf is a friend of mine who, ~1y ago, was super excited I was joining Sandstorm because the funding announcement post says there'll be a powerbox and that got him super excited.
<zarvox>
:D
<nwf>
All true.
<kentonv>
I remember you telling me about that
<asheesh>
The funding announcement post was one of the first things published after I joined, as context.
<zarvox>
For me, one of the goals of powerbox is to show that a great many things are actions which should be considered alterable by the user, and to shift the landscape toward trusted UI over confirmation dialogs or install-time warnings.
<zarvox>
And thanks to capnproto schema IDs, we can match interfaces without having to understand precisely what they do, which lets us make trusted UI for arbitrary types and interfaces.
jadewang has quit [Remote host closed the connection]
<asheesh>
Feel free to edit or provide remarks to me; I attempted to imbibe a lot of object-oriented kool-aid while writing it, but who knows.
<asheesh>
I also realize it's not literally capnp.
<digitalcircuit>
If appropriate, and pardon that I don't know if you need to handle it at all here, consider the IRCv3 specs, too: http://ircv3.net/irc/
<zarvox>
asheesh: I think that IrcDriver isn't quite the right interface - I think that the driver gets to pick where the connection goes, not the client
<asheesh>
Yeah, but maybe the client gets to make a hint?
<kentonv>
ideally a client would powerbox-request a single channel
<zarvox>
that's reasonable, but it'd be in the powerbox request, not in the driver interface
<zarvox>
a single channel? where do PMs go?
<kentonv>
that's an interesting question that I had an answer for at one point, let me try to remember
<kentonv>
so in the channel you have a user list, which should have capabilities attached. Those capabilities allow opening a PM channel to that user. Ideally this would somehow create a new grain specifically for the PM, or reuse an existing grain that has the same user pair
<kentonv>
this is mostly straightforward except the "reuse an existing grain" part
<zarvox>
also it's plausible that an IRC connection driver could chain to an IRC channel driver or something
<kentonv>
yes, I'd imagine you could request either
<kentonv>
but a good, security-friendly app requests the minimum possible
<zarvox>
also the fact that we don't exactly have offer sessions figured out, nor how you'd make a good smooth UI for integrating all this :P
<zarvox>
Also, what happens when you are in a channel, and a user PMs you?
<asheesh>
You are both blowing my mind, but not in a bad way, just I have to think much harder than I had expected to process this conversation.
<zarvox>
Are you obligated to open all the grains for all the users in all the channels you might be in, just to be able to monitor them for PMs? What if a user PMs you by nick?
<asheesh>
kentonv: Keep in mind that "request a channel" is almost impossible in IRC, in theory.
<asheesh>
If you /join #foo the server is not obligated to put you in #foo
<asheesh>
It might put you in #foo-unregistered or just do nothing at all.
<kentonv>
asheesh: the IRC driver can assume "reasonable behavior"
<asheesh>
Or it might put you in #bar and #baz
<asheesh>
I _guess_. But redirects are a real thing, fwiw.
<zarvox>
I suspect that the "server connection" is the right abstraction to start with, and also will hopefully be a small enough pill to swallow for implementers to get started
<asheesh>
I do like object-capability Channels, in theory.
<asheesh>
You could imagine just timing out if the Channel never shows up after N seconds.
<zarvox>
I think it'd be reasonable to have Channels be a type exposed there! and maybe even how you send messages!
<asheesh>
Ya see "class Channel" in the proposal
<zarvox>
I just think that a useful IRC client will need to request more than just a single Channel.
<asheesh>
Right, yeah, many will.
<zarvox>
Maybe bots won't.
<asheesh>
Some won't, like IRC log bots.
<asheesh>
Right.
<asheesh>
whoa
<asheesh>
i,i remote code execution on a bot can only cause it to spam the channel
<asheesh>
anyway
<kentonv>
I think it makes sense to design the API to have both a "Connection" and a "Channel", and design the driver to accept either kind of request, and we will probably start out with apps that request a full connection and then get channels from it but hopefully someday we have apps that request channels and get the security benefits
<zarvox>
+1
<asheesh>
I would probably write an IRC Log Bot app that works that way.
<kentonv>
indeed, for an app that obviously only should talk to a single channel, being written from scratch, being able to request channels probably makes things easier.
<kentonv>
how to handle PMs from users that share no channels with you is an interesting question.
<asheesh>
I see PMs as a server-level action not a channel-level action. See also IrcClient.onPrivateMessage()
<kentonv>
UX-wise I feel like you should get a notification of the PM, and clicking on it opens the designated PM grain between you and that user
<zarvox>
And for a logbot you don't care about PMs :)
<asheesh>
FWIW I don't see how that helps me run my desktop quassel client speaking to the quasselcore on Sandstorm, but OK
<kentonv>
right, I'm no longer thinking about the logbot
<asheesh>
I guess you're talking about a separate fractally object-capability hardened web IRC client.
<kentonv>
yes
<asheesh>
(-:
<kentonv>
I want to imagine the ideal so that we don't do anything which prevents evolution in that direction
<asheesh>
My goal with an IRC capnp would be to support that in theory, yeah.
<pdurbin>
asheesh: you're writing an IRC logging bot?
<kentonv>
I think on an incoming PM the IRC driver would in fact create a new grain (or find an existing grain) and then "share" it to you via a notification.
<asheesh>
Well I'm vaguely working with possibly-zarvox-if-he-wants to write enough Sandstorm infrastructure to support an IRC logging bot, which might result in me actually writing the logging bot, or might result in me stapling an existing logging bot to the corresponding Sandstorm infrastructure.
<pdurbin>
ah :)
<kentonv>
and if you want to open a PM to some username on some server, you do a powerbox request for some sort of "PersonalMessage" object.
<asheesh>
kentonv: I'm surprised it's the driver's place to create grains; that's news to me, but maybe it makes sense.
<asheesh>
To me that's one optional, separate thing you'd connect the driver to, but I guess I'm splitting hairs here.
<kentonv>
asheesh: in some cases, yes. Here, the driver is creating a grain as an alternative to hosting a UiView within the driver process, the purpose being to get isolation. Otherwise the driver itself is a big target.
<kentonv>
I think we need this for the web publishing driver too
<asheesh>
But right now a driver can't create grains, right? I was hoping to make something that was actually usable soonish.
<asheesh>
That fact could be an implementation detail of the driver, though.
<kentonv>
well right now we don't support any concept of drivers
<asheesh>
I dunno, I can go connect IpNetwork to some grain and call it day (-;
<asheesh>
but yeah
<kentonv>
but right, the ability to create subordinate grains does not yet exist
<kentonv>
that is something that we want, independent of drivers
<asheesh>
Yeah, totally agreed.
<kentonv>
in any case, here it's an implementation detail that increases security, but isn't strictly needed
<asheesh>
b
<kentonv>
I really, really do want to see chat rooms and PMs (whether IRC, rocket.chat, or whatever) all end up being separate grains, though, as otherwise some of our security claims are a lot weaker.
<kentonv>
food time!
<pdurbin>
asheesh: speaking of logging chat, have you seen https://sideway.com ?
<asheesh>
You have caused me to fork the mkdocs readthedocs theme and wonder freshly about the future of mkdocs.
simonv3 has quit [Quit: Connection closed for inactivity]
<asheesh>
git commit -m "Import readthedocs theme from mkdocs" # now I am in control of MY OWN DESTINY
simonv3 has joined #sandstorm
|jemc| has quit [Ping timeout: 260 seconds]
|jemc| has joined #sandstorm
simonv3 has quit [Quit: Connection closed for inactivity]
|jemc| has quit [Ping timeout: 264 seconds]
<pdurbin>
asheesh: heh. sure. maybe it's something of a solved problem. I maintain a list of free IRC logging services at http://wiki.greptilian.com/irc/logging but I'm still interested in what they come up with.
<asheesh>
synchrone (I know you're away but) - presumably browsers will never do WebDAV so we could whitelist any client that sends WebDAV-y headers? cc: kentonv
<asheesh>
Hi tonyrei! Too bad we missed you. Send us an email if you have a question! We'd love to answer. community@sandstorm.io
<asheesh>
orangedolphin: I know you're not here, but: yes! You totally can. Maybe we can make that clearer. This is specifically withe rgard to static publishing, like using WordPress to publish a blog on asheesh.org or whatever your domain is.
<ToeiRei>
asheesh, I was just playing with a sandstorm instance
<ToeiRei>
so never mind. I'm just starting to explore and see if it fits my needs
mnutt has joined #sandstorm
<asheesh>
Ah, great, ToeiRei !
<asheesh>
I hope it does and that you'll say more about your needs and how it does/doesn't!
<ToeiRei>
In my case it's about NodeJS stuff I should host.
<asheesh>
Well there are lots of packages of nodejs apps. I've been enjoying nodebb a lot today.
<ToeiRei>
I've seen the two most important ones for me already as I could throw out one machine.
<ToeiRei>
etherpad and rocket chat
<ToeiRei>
but I have not yet managed to get the services out there without the sandstorm authentication
neynah has joined #sandstorm
<asheesh>
Right, yeah, you can't without the auth. You can make an public sharing link though.
<asheesh>
I must wander off for a bit, but feel free to ask more questions; I'll try to answer them later, and/or you can email!
<ToeiRei>
but thanks. I'll have a look into things a bit more
<dwrensha>
good thing there's another demo button there!
<asheesh>
But not on the github, only on the homepage!
<asheesh>
But yeah (-:
<mrdomino>
asheesh: say i want to just like pipe my sandstorm instance to a new computer in the same dc
<asheesh>
sudo service sandstorm stop
<mrdomino>
and then i'm just gonna use nginx to proxy to the other instance until the dns flops over
<asheesh>
sudo tar cf /opt/sandstorm | ssh user@host '(mkdir move-this-to-opt-sandstorm ; tar xf -)'
<mrdomino>
wait it's dyndns
<mrdomino>
so nvm on that part
<asheesh>
Yeah, give it 60-120sec and you'll be in good shape.
<mrdomino>
ok so literally just that?
<asheesh>
Yeah, that's all that's needed.
<asheesh>
I must AFK a bit!
wolcen has quit [Ping timeout: 264 seconds]
mnutt has quit [Quit: My Mac has gone to sleep. ZZZzzz…]
isd has joined #sandstorm
mwcampbell has joined #sandstorm
<mrdomino>
d'oh mongod died
simonv3 has quit [Quit: Connection closed for inactivity]
mrdomino has quit [Remote host closed the connection]
mrdomino has joined #sandstorm
<mrdomino>
and we're back
<digitalcircuit>
Welcome back! I didn't know you could use your sandcats.io domain as your IRC hostname :)
<mrdomino>
yeah i mean it's just irssi sitting on my droplet
<mwcampbell>
Why not? Just set the reverse DNS record for your IP
<digitalcircuit>
I mistakenly had thought the DNS record needed special modification too. Noted! No such luck for residential, I guess.
<mwcampbell>
I wonder how common it is to run a Sandstorm server at home on a residential connection. Last time I checked, my ISP (Cox cable) blocks incoming connections to port 80 for residential users.
<digitalcircuit>
Time Warner Cable doesn't for now. And if port 443 is unblocked, that's doable. Custom port works..ish, though it's a pain to remember to type correctly (Sandstorm doesn't do HSTS yet).
<digitalcircuit>
At least, not if you're using a third-party router :)
<digitalcircuit>
mrdomino: i.e. having multiple folks streaming?
<mrdomino>
yup
<mwcampbell>
I went to the URL and I hear music.
<digitalcircuit>
Still having the random Groovebasin dropouts, mrdomino? I haven't yet had time to test further myself; I think certain audio files are more prone to causing it.
<mwcampbell>
oh, that might have been because I was uploading just now
<mwcampbell>
mrdomino: The music stopped.
<dwrensha>
uploads should not cause the music to stop
<digitalcircuit>
Still playing over here... However, mrdomino left the Sandstorm page. I have a vague hunch that unexpected disconnections -might- cause issues for Groovebasin.
<mrdomino>
yeah uploads causing the server to turn over is kinda unfortunate
droyo has joined #sandstorm
<mrdomino>
well we're back
<mrdomino>
(i left the page when my laptop suspended probably :-p)
<mrdomino>
also like an intelligent person i now have the debug log up
<mwcampbell>
mrdomino: Sorry, I didn't mean I was uploading to the server. Just that I was uploading in general (to Dropbox in this case), and that caused my connection to be slow for a bit.
<mwcampbell>
So the only real problem is that the stream stopped when you left the page.
<Zertrin>
hi! I've read a bit above [21:14:09 UTC+1] that to move sandstorm one has just to tarpipe /opt/sandstorm to somewhere else. Now, I've wanted to try sandstorm on my server but didn't want to just install it in case it spread everywhere and is a pain to remove if needed. So I created a VM to install it in. So my question is: if using the official install script, what is the footprint of the installation of sandstorm outside of /opt/sandstorm. There must be
<Zertrin>
something at least for being able to do service sandstorm stop (or systemctl stop sandstorm)? Or am I mistaken?
<kentonv>
Zertrin: it creates /etc/systemd/system/sandstorm.service as well as symlinks "sandstorm" and "spk" under /usr/local/bin
<Zertrin>
is that all? (nice)
<Zertrin>
I'd consider adding that info somewhere accessible, since I've been looking for it quite a long time and even read the script in its entirety many times, but wasn't sure since maybe after the script ends, sandstorm complete some additional setup tasks
<Zertrin>
If you this that would be good to document it, I could provide a PR for the documentation about that
<kentonv>
the comment at the top of install.sh lists these, although it's slightly outdated in that it mentions sysv init but not systemd
<kentonv>
I agree we should document it better
<kentonv>
and have a "sandstorm uninstall" command
<Zertrin>
indeed that would also be nice
<kentonv>
but yeah, Sandstorm puts basically everything under /opt/sandstorm and self-containerizes so that it doesn't need to touch the rest of your system
<Zertrin>
good to have confirmation here, thanks!
<kentonv>
that said running in a VM is a great idea for defense-in-depth. :)
<Zertrin>
I've been facing a second issue, I've tried to Proxy the HTTPS traffic through the Apache in a virtualhost outside of the VM, but of course that's a bit complicated since Apache has no access to the secret key and the certificate, and because these are rotated quite often.
<Zertrin>
So my current solution is just to run sandstorm on another HTTPS port (different than 443) and do the correct forwarding through the firewall.
<Zertrin>
Any thoughts on this issue? (the goal would be to still run sandstorm in the VM and to proxy through Apache/nginx in a virtualhost from the host, or alternatively, install sandstorm on the host (to facilitate access to sandstorm data inclusine Cert material from the host) and still proxy it in a virtualhost)
<asheesh>
I need to integrate that into the docs, but that has an overview.
<asheesh>
Zertrin + kentonv : re: /opt/sandstorm containing basically all the files: looks like https://sandstorm.io/install should say what files it adds.
<kentonv>
Zertrin: Basically if you're using free Sandcats certs then you're going to want to forward the whole port without an HTTP-level proxy in between, since it's hard to load the proxy with the right certs as you note. If you have other HTTPS sites to serve at the same server, probably the easiest (but pricey) thing is to get a wildcard cert for your own domain
<Zertrin>
oh that's probably new
<asheesh>
Zertrin: Yeah I think I added it yesterday (-: and Pablo Jamar wrote it ~2 weeks ago!
isd has joined #sandstorm
<Zertrin>
:)
<Zertrin>
thanks !
<Zertrin>
indeed that confirm my analysis of the current options. I'll probably wait for Let's Encrypt to do wildcard for now (even if that's in a while)
<Zertrin>
Otherwise I've had a pretty happy time with sandstorm for now. Will try to find time to setup a dev environment in the next weeks to try at improving/adding packages