<zarvox>
I also hit your error when attempting to upload to Oasis
<zarvox>
Also works fine locally.
<mnutt_>
ahh hmm, I vaguely recall reading something about "all assets can't be over 1MB" but wasn't sure if that was referring to just the screenshots or also the icons, if it's for everything then it's over 1MB, but I would have expected it to fail locally for that too.
<zarvox>
Likewise.
<ocdtrekkie>
zarvox: The ETag thing means I can update an app which wasn't able to be built for a while.
<ocdtrekkie>
Also the sorting, zarvox. I am a bit fan of sorting.
<zarvox>
mnutt_: looks like I can't upload that package to our test blackrock instance either, so it's definitely something different in blackrock
<zarvox>
ocdtrekkie: :D
<zarvox>
I am also a fan of sorting! Now that we have individual grain sizes, there's more than one potentially-useful field to sort on
<ocdtrekkie>
I am excited about that too
<ocdtrekkie>
I think it was one of my wishlist items.
<zarvox>
I remember about a year ago when we were first embarking on the shell redesign, neynah wanted grain sizes, and sorting, and we had to sadly say "we can't do that, there are bad technical reasons why we don't have that information"
<zarvox>
but now we have that data! so we can build what we originally wanted!
<ocdtrekkie>
This was one of the weeks where I was actually anxious for a release. :P
<zarvox>
Glad my blur-day antics are appreciated :)
eternaleye has quit [Changing host]
eternaleye has joined #sandstorm
<zarvox>
mnutt_: my reading of the blackrock code and the sandstorm code is that they should have the same manifest size limits, and that it's limited to 1048576 capnp words, or 8MB, and with each asset no larger than 1MB
<zarvox>
I'm probably gonna punt this one to Kenton and see if he can figure out if it's a blackrock bug or sandstorm being too lax somewhere or something else
kentonv has joined #sandstorm
<zarvox>
kentonv: thanks for dropping in! see logs of the past ~hour :)
<kentonv>
mnutt_: Blackrock has an undocumented limit of 1MB on all metadata, I think
<kentonv>
due to the way the storage backend works
<kentonv>
unclear whether that limit should be raised somehow or if we should lower the limit in the spk tool to match
<kentonv>
having many megabytes of screenshots is probably not great for the app market experience
<zarvox>
FWIW there are 1.5MB of screenshots, so if the manifest is capped with a 1MB hard limit, that'll be the culprit
<kentonv>
I mean, we could also push the limit up in blackrock
<kentonv>
it's the limit on the size of a single capnp object stored in the blackrock storage backend
<mnutt_>
when you say screenshots, are you including icons as well?
<zarvox>
yes, icons are also included in the manifest
<mnutt_>
ah, ok
<mnutt_>
I'll try to get the whole thing under 1MB and that should probably do it. Sorry for all the back and forth, I read about that limitation but got confused because it worked on my personal server
<zarvox>
No worries, thanks for mentioning your confusion so we can make things less confusing :)
<kentonv>
what is mongo's limit on a document?
<kentonv>
16MB, apparently
<kentonv>
if 16MB of BSON is not a performance problem then I guess 16MB of capnp should be just fine.
<zarvox>
:)
<zarvox>
Probably worth leaving the cap at 8MB so we can't accidentally go over the limit with metadata in staticAssets
<mnutt_>
I think my hitting the limit was a combination of wanting to use retina screenshots + having to switch the icons from svg to png due to some design updates. the 600x600 icon is 335kb now :-/
<kentonv>
eek, that's a large icon...
jemc has quit [Ping timeout: 244 seconds]
<kentonv>
is that the app market icon?
<zarvox>
yeah
<kentonv>
hmm that worries me since everyone who hits the app market front page needs to download all the icons.
<zarvox>
how do other people like Apple/Google solve this problem?
<mnutt_>
I would assume they'd get all the 300x300 (150x150@2x) icons, and that the 600x600 icon is only on the detail page?
<kentonv>
mnutt_: FWIW I am going to go ahead and increase the blackrock object size limit in today's release
<kentonv>
I think the app market may currently always use the larger icon because we haven't had time to make it adaptive
<mnutt_>
ah, I see
<kentonv>
but not 100% sure about that
<mnutt_>
hrm, so pngs + noise do not go together well. a 600x600 jpeg is 75kb. I'll switch to that.
<kentonv>
oh yeah, that's a good plan
TheWhisperSP3 has quit [Read error: Connection reset by peer]
TheWhisperSP3 has joined #sandstorm
<kentonv>
mnutt_: can you try uploading to testrock.sandstorm.io now?
<mnutt_>
sure
<mnutt_>
I'm assuming it's fine to use my credit card there? I need to sign up for a plan in order to upload
<kentonv>
mnutt_: use a stripe test card, e.g. 4242 4242 4242 4242
<mnutt_>
ah ok
<mnutt_>
kentonv: it works just fine on testrock
<kentonv>
great, after the Oasis update at midnight it should work on Oasis too
<kentonv>
mnutt_: btw I just put a $288 credit on your Oasis account since we've giving that to all app developers
<mnutt_>
thanks kentonv!
<kentonv>
should show up on your account settings
<mnutt_>
btw, new davros release
<kentonv>
mnutt_: hmm I'll have to make sure not to accept it until after the update
<kentonv>
actually, sorry, I'm rejecting it for now because Asheesh might accidentally accept it otherwise.
<mnutt_>
I resized the screenshots to reduce the filesizes, should be alright
<kentonv>
ohhhh
<kentonv>
dang, I shouldn't have rejected then
<kentonv>
... hmm I apparently don't have the easy ability to un-reject
<mnutt_>
well, it turned out alright because I just noticed something annoying with the icons I want to fix. I'll resubmit
<kentonv>
oh ok
<kentonv>
cool
<kentonv>
approved!
<kentonv>
( mnutt_ )
<mnutt_>
awesome, thanks
<mnutt_>
hopefully it will fix the issues people have been having with owncloud client crashing :-/
jemc has joined #sandstorm
<kentonv>
hmm I see a lot of
<kentonv>
err
<kentonv>
mnutt_: hmm I see a lot of cases of thumbnails being broken
<kentonv>
content-length: 0
<mnutt_>
hmm, are all of them broken or just some?
<kentonv>
seems like PNGs but I don't have many data points
<mnutt_>
ok, I'll investigate
<kentonv>
yeah jpgs work, pngs don't
<mnutt_>
well, yep, that's it. I think I missed a library when building
<mnutt_>
ok, submitted a fix as 0.21.4 (there was one in between, 0.21.3, that can be ignored)
<kentonv>
oh I had already approved 0.21.3 :)
<kentonv>
now approved 0.21.4, seems to fix the problem
kxra has quit [Ping timeout: 250 seconds]
jemc has quit [Ping timeout: 260 seconds]
kxra has joined #sandstorm
mnutt_ has quit [Quit: My Mac has gone to sleep. ZZZzzz…]
<zarvox>
I am totally loving this sort grains by size thing.
nwf has quit [Ping timeout: 276 seconds]
nwf has joined #sandstorm
Telesight has joined #sandstorm
davidar has quit [Remote host closed the connection]
M-hrjet has quit [Remote host closed the connection]
eternaleye has quit [Remote host closed the connection]
JonTheNiceGuy[m] has quit [Remote host closed the connection]
M-hrjet has joined #sandstorm
davidar has joined #sandstorm
eternaleye has joined #sandstorm
JonTheNiceGuy[m] has joined #sandstorm
mnutt_ has joined #sandstorm
amyers has joined #sandstorm
amyers has quit [Ping timeout: 250 seconds]
amyers has joined #sandstorm
amyers has quit [Ping timeout: 250 seconds]
dwrensha has joined #sandstorm
jemc has joined #sandstorm
Lionel_Debroux_ has quit [Ping timeout: 276 seconds]
pie_ has joined #sandstorm
pie_ has quit [Changing host]
pie_ has joined #sandstorm
mnutt_ has quit [Quit: My Mac has gone to sleep. ZZZzzz…]
mnutt_ has joined #sandstorm
Lionel_Debroux has joined #sandstorm
mnutt_ has quit [Quit: My Mac has gone to sleep. ZZZzzz…]
TheWhisperSP3 has quit [Quit: Leaving]
TheWhisperSP3 has joined #sandstorm
TheWhisperSP3 has quit [Quit: Leaving]
n8a has joined #sandstorm
TheWhisperSP3 has joined #sandstorm
<mokomull>
zarvox: I'm curious what wins the most bloated storage backend award.
<dzho>
oh, is there an award for that?
<zarvox>
Neither postgres nor mysql scale down to tiny data particularly well.
<dzho>
I thought that was what sqlite3 was for
<zarvox>
Yeah, we encourage apps to use sqlite3 if at all reasonable
<dzho>
zarvox++
<zarvox>
but for some, that would mean rewriting their DB layer, or not using DB types provided by MySQL, or other significant work that may not be supported upstream
<zarvox>
so they just run a MySQL instance, and things still turn out mostly okay
<zarvox>
Piwik is an example of such a package that makes use of MySQL features
<mokomull>
zarvox: heh, I was thinking Mongo actually. Noticed at least one of the apps I tried on the bgp devbox defaulted to like an 8MB hunk with zero data.
<zarvox>
Mongo might not be great with the default options, but since mongo3, the wiredTiger storage backend for mongo can be tuned to take up very little disk space
<mokomull>
But thanks for bringing back terrible memories of InnoDB at imo :)
<zarvox>
Mongo 2.6 was not so great about this, which led to the "niscudb" fork ( https://github.com/kentonv/mongo/tree/niscu ) that we bundled in older versions of meteor-spk (and still have to include to allow migrating data to the new engine)
<mokomull>
huh, neat.
M-hrjet has quit [Remote host closed the connection]
JonTheNiceGuy[m] has quit [Remote host closed the connection]
eternaleye has quit [Remote host closed the connection]
davidar has quit [Remote host closed the connection]
M-hrjet has joined #sandstorm
TheWhisperSP3 has quit [Quit: Leaving]
TheWhisperSP3 has joined #sandstorm
WhisperSurface has joined #sandstorm
WhisperSurface has quit [Client Quit]
TheWhisperSP3 has quit [Ping timeout: 265 seconds]
TheWhisperSP3 has joined #sandstorm
TheWhisperSP3 has quit [Quit: Leaving]
Telesight has quit [Quit: Leaving.]
davidar has joined #sandstorm
eternaleye has joined #sandstorm
JonTheNiceGuy[m] has joined #sandstorm
TheWhisperSP3 has joined #sandstorm
mnutt_ has joined #sandstorm
<simonv3>
yeah I’ve noticed that all the apps I start with Meteor are 21 mb by default
<simonv3>
(or some such size)
<kentonv>
simonv3: hmm, that shouldn't be true. We explicitly created niscu and later configured wiredTiger to avoid that...
<simonv3>
kentonv: This was a while ago, I’ll double check now
<simonv3>
oh, maybe I was just thinking of the package size actually, nvm
<kentonv>
ahh
<kentonv>
21mb is not bad for package size
<simonv3>
I see a fresh quick survey size is 338kB, which is pretty good!
<kentonv>
yep, hard to get mongo to go smaller than that
<simonv3>
With collection out I need to figure out how I’m going to organize this stuff, definitely have some grains overload
isd has joined #sandstorm
<ocdtrekkie>
TinyTinyRSS is currently the worst Sandstorm app on storage as far as I can tell.
<ocdtrekkie>
MySQL is it's backend, and it doesn't clean up data well (at all?) Mine's grown to 91 MB.
larjona has quit [Remote host closed the connection]
larjona has joined #sandstorm
mnutt_ has quit [Quit: My Mac has gone to sleep. ZZZzzz…]