alanshaw changed the topic of #ipfs to: Heads Up: To talk, you need to register your nick! Announcements: go-ipfs 0.4.22 and js-ipfs 0.40 are out! Get them from dist.ipfs.io and npm respectively! | Also: #libp2p #ipfs-cluster #filecoin #ipfs-dev | IPFS, the InterPlanetary FileSystem: https://github.com/ipfs/ipfs | Logs: https://view.matrix.org/room/!yhqiEdqNjyPbxtUjzm:matrix.org/ | Forums: https://discuss.ipfs.io | Code of
silotis has joined #ipfs
mowcat has quit [Remote host closed the connection]
opal has quit [Quit: i'm never coming back]
opal has joined #ipfs
mithilarun has joined #ipfs
opal has quit [Client Quit]
opal has joined #ipfs
dqx_ has quit [Read error: Connection reset by peer]
turona has quit [Ping timeout: 272 seconds]
turona_ has joined #ipfs
turona_ has quit [Max SendQ exceeded]
turona has joined #ipfs
<vaultec81[m]>
Hopefully IPFS matures to be a really robust piece of software.
KeiraT has quit [Remote host closed the connection]
KeiraT has joined #ipfs
mithilarun has quit [Remote host closed the connection]
VoiceOfReason has quit [Quit: Free ZNC ~ Powered by LunarBNC: https://LunarBNC.net]
lousiF has quit [Quit: Bye]
BladedThesis has quit [Excess Flood]
BladedThesis has joined #ipfs
distoresite[m] has quit [Ping timeout: 264 seconds]
journey[m] has quit [Ping timeout: 264 seconds]
DirkOlmes[m] has quit [Ping timeout: 264 seconds]
d0tr4ck3t33r has quit [Ping timeout: 264 seconds]
DiscordBridgeswe has quit [Ping timeout: 264 seconds]
djdv[m] has quit [Ping timeout: 264 seconds]
darn[m] has quit [Ping timeout: 264 seconds]
Caleb[m]1 has quit [Ping timeout: 264 seconds]
appDiscord[m] has quit [Ping timeout: 264 seconds]
enricomarinoDisc has quit [Ping timeout: 264 seconds]
Discord[m]2 has quit [Ping timeout: 264 seconds]
chrisDiscord[m] has quit [Ping timeout: 264 seconds]
NastyEbilPiwateD has quit [Ping timeout: 264 seconds]
vbDiscord[m] has quit [Ping timeout: 264 seconds]
NetherwolfDiscor has quit [Ping timeout: 264 seconds]
tplookerDiscord[ has quit [Ping timeout: 264 seconds]
ArunDiscord[m] has quit [Ping timeout: 264 seconds]
denzukoDiscord[m has quit [Ping timeout: 264 seconds]
captain-nemoDisc has quit [Ping timeout: 264 seconds]
kerlanTDiscord[m has quit [Ping timeout: 264 seconds]
FeNiXDiscord[m] has quit [Ping timeout: 264 seconds]
fexra|TRTLDiscor has quit [Ping timeout: 264 seconds]
sblinnDiscord[m] has quit [Ping timeout: 264 seconds]
achingbrainDisco has quit [Ping timeout: 264 seconds]
AutoAIDiscord[m] has quit [Ping timeout: 264 seconds]
ithithDiscord[m] has quit [Ping timeout: 264 seconds]
HarryTmeticDisco has quit [Ping timeout: 264 seconds]
cwDiscord[m] has quit [Ping timeout: 264 seconds]
ZapierDiscord[m] has quit [Ping timeout: 264 seconds]
funwhilelostDisc has quit [Ping timeout: 264 seconds]
simibacDiscord[m has quit [Ping timeout: 264 seconds]
nocentDiscord[m] has quit [Ping timeout: 264 seconds]
anna[m] has quit [Ping timeout: 264 seconds]
GitterIntegratio has quit [Ping timeout: 264 seconds]
b166er has quit [Ping timeout: 264 seconds]
ircer[m] has quit [Ping timeout: 264 seconds]
jnes has quit [Ping timeout: 264 seconds]
wlp1s1 has joined #ipfs
lufi has joined #ipfs
iczero has quit [Read error: Connection reset by peer]
VoiceOfReason has joined #ipfs
jnes has joined #ipfs
mithilarun has quit [Remote host closed the connection]
plexigras has quit [Ping timeout: 240 seconds]
maxzor has quit [Ping timeout: 240 seconds]
tpewnag has quit [Quit: Leaving]
tpewnag has joined #ipfs
Clarth3 has joined #ipfs
daMaestro has joined #ipfs
jcea has quit [Quit: jcea]
lalilulelo has quit [Remote host closed the connection]
lalilulelo has joined #ipfs
Belkaar has quit [Ping timeout: 268 seconds]
Belkaar_ has joined #ipfs
tpewnag has quit [Quit: Leaving]
RamRanRa has quit [Read error: Connection reset by peer]
Trieste has quit [Ping timeout: 258 seconds]
hold_on_fixing_l has joined #ipfs
lalilulelo has quit [Ping timeout: 272 seconds]
hold_on_fixing_l is now known as lalilulelo
Trieste has joined #ipfs
user_51 has quit [Ping timeout: 268 seconds]
hurikhan77 has joined #ipfs
user_51 has joined #ipfs
kakra has quit [Ping timeout: 258 seconds]
lalilulelo has quit [Remote host closed the connection]
lalilulelo has joined #ipfs
Clarth3 has quit [Quit: Leaving]
MDude has quit [Quit: Going offline, see ya! (www.adiirc.com)]
CCR5-D32 has joined #ipfs
verin0x1 has joined #ipfs
verin0x has quit [Ping timeout: 268 seconds]
verin0x1 is now known as verin0x
rhymeswithbike[m has joined #ipfs
shizy has joined #ipfs
elfatherbrown_ has joined #ipfs
elfatherbrown has quit [Ping timeout: 240 seconds]
shizy has quit [Remote host closed the connection]
delli3_ has joined #ipfs
delli3_ has quit [Max SendQ exceeded]
delli3 has quit [Ping timeout: 248 seconds]
zeden has quit [Quit: WeeChat 2.6]
AbramAdelmo_ has quit [Read error: Connection reset by peer]
AbramAdelmo has joined #ipfs
is_null_ has joined #ipfs
is_null has quit [Ping timeout: 260 seconds]
daMaestro has quit [Quit: Leaving]
dodecahedrane[m] has joined #ipfs
elfatherbrown_ has quit [Ping timeout: 246 seconds]
maxzor has joined #ipfs
c0ncord has quit [Ping timeout: 246 seconds]
MrSparkle is now known as fellape
maxzor has quit [Remote host closed the connection]
CCR5-D32 has quit [Quit: ZZZzzz…]
}ls{ has joined #ipfs
stoopkid_ has quit [Quit: Connection closed for inactivity]
hello23786a has joined #ipfs
bornjre has joined #ipfs
cha18 has left #ipfs [#ipfs]
malfort has quit [Remote host closed the connection]
kaotisk has quit [Read error: Connection reset by peer]
malfort has joined #ipfs
kaotisk has joined #ipfs
maxzor has joined #ipfs
bengates has joined #ipfs
ylp has joined #ipfs
RamRanRa has joined #ipfs
ygrek_ has joined #ipfs
<eleitl[m]>
bonedaddy (Discord): You wrote "Performance of go-ipfs is not suitable for large scale archiving of data access by clients through IPFS". Would that include client access to a distributed storage backend for a dataset in 10-100 Tbyte order of magnitude?
<eleitl[m]>
The dataset is static, and centrally maintained. The client access is decentral.
<eleitl[m]>
If go-ipfs can't do that, what specifically does it fail at? And what kind of custom client would be needed, then?
<eleitl[m]>
I've already noticed there are visibility issues for the wrapper directory of the Gutenberg dataset I've published yesterday in three instances. I expect this is due to go-ipfs having isses dealing with wrapping a flat directory of 56 k small files with some 50 Gbytes total.
<eleitl[m]>
I will try to repackage the dataset in a layer of hex-named directories, with two layers for the larger datasets.
cxl000 has joined #ipfs
is_null_ has quit [Read error: Connection reset by peer]
cwahlers has quit [Ping timeout: 272 seconds]
cwahlers has joined #ipfs
is_null has joined #ipfs
ipfs-stackbot has quit [Remote host closed the connection]
ipfs-stackbot has joined #ipfs
Soltero_7 has joined #ipfs
Soltero_7 has left #ipfs [#ipfs]
xcm has quit [Remote host closed the connection]
xcm has joined #ipfs
pecastro has joined #ipfs
<lastdigitofpi[m]>
bonedaddy (Discord) eleitl 100TB, 83 million files. We're prepared to write a custom client from the groupd up if that's what it would take.
<lastdigitofpi[m]>
* bonedaddy (Discord) eleitl 100TB, 83 million files. We're prepared to write a custom client from the ground up if that's what it would take.
<eleitl[m]>
So the current go-ipfs can't deal with that scale, right? Where exactly is the problem located? If you would be rewriting from scratch, what kind of base toolkit would you start with? rust-libp2p? Not using libp2p at all?
<lastdigitofpi[m]>
also to note: this data set is already been handled very well by bittorrent - but bittorreent doesn't work well for fine-grained access, pinning, or addressibility, hence the interest in moving to IPFS
captain_morgan20 has quit [Read error: Connection reset by peer]
captain_morgan20 has joined #ipfs
maxzor has quit [Ping timeout: 265 seconds]
captain_morgan20 has quit [Ping timeout: 240 seconds]
captain_morgan20 has joined #ipfs
natewalck has quit []
natewalck has joined #ipfs
hello23786a has quit [Remote host closed the connection]
captain_morgan20 has quit [Quit: Ping timeout (120 seconds)]
captain_morgan20 has joined #ipfs
vmx has joined #ipfs
bret has quit []
bret has joined #ipfs
i9zO5AP has quit [Ping timeout: 268 seconds]
captain_morgan20 has quit [Read error: Connection reset by peer]
djoek[m] has joined #ipfs
mikolalysenko has quit []
mikolalysenko has joined #ipfs
teleton has joined #ipfs
ttt7659[m] has joined #ipfs
Jonathan[m]3 is now known as kiwi_uk[m]
<voker57>
since the hash in question could be found with DHT but not fetched, it is probably connectivity issues
<voker57>
wrong IPs, ports not being forwarded, smth like that
<voker57>
I don't think size of dataset matters here
jcea has joined #ipfs
mowcat has joined #ipfs
<eleitl[m]>
@voker57 Can you give me some hints as to how to debug connectivity issues at IPFS level?
<eleitl[m]>
I don't think it's the ports. Perhaps it's issues with QUIC or something else in the config.
<eleitl[m]>
I had no issues publishing data sets of some 40-140 Gbyte in the past, but these were relatively few files, larger, and I have made a lot of changes to the IPFS infra since.
Trieste has quit [Ping timeout: 265 seconds]
<voker57>
eleitl[m]: see `ipfs swarm peers` to see if you are being connected to a specific node, issue a manual connect command to see if it fixes the problem, afterwards might parse `ipfs log tail` to see what is going on and why your connections are the way they are
Trieste has joined #ipfs
plexigras has joined #ipfs
KempfCreative has joined #ipfs
plexigras has quit [Ping timeout: 265 seconds]
moul has quit []
moul has joined #ipfs
georgyo has quit []
georgyo has joined #ipfs
plexigras has joined #ipfs
AbramAdelmo_ has joined #ipfs
AbramAdelmo has quit [Ping timeout: 240 seconds]
<eleitl[m]>
Like ipfs swarm connect /ip6/2a02:120b:2c65:f00:2d10:5c0d:40fa:924d/tcp/62486/ipfs/QmPGho1CbyAvS7qJqoazdvphLDamZfR9FrP1fkUi39m1MV
AbramAdelmo_ has quit [Read error: Connection reset by peer]
AbramAdelmo has joined #ipfs
Jokera has quit [Read error: Connection reset by peer]
elfatherbrown has joined #ipfs
mithilarun has quit [Remote host closed the connection]
mithilarun has joined #ipfs
arma has joined #ipfs
mithilarun has quit [Ping timeout: 265 seconds]
<c0ncord>
ok, this is kind of a weird question, i admit
<c0ncord>
but i need a specific scenario
arma has quit [Client Quit]
<c0ncord>
*have
mart1n has quit [Ping timeout: 240 seconds]
arma has joined #ipfs
<c0ncord>
i have a specific scenario where i need to authorize access to a file
<c0ncord>
once the peer has a copy of this file data, i don't care what they do with it
<c0ncord>
but to release this data myself, i need authorization; in other words, the peer needs to send an authorization token or something
<c0ncord>
otherwise, i won't share this file
<c0ncord>
how to accomplish that with ipfs?
mart1n has joined #ipfs
llorllale has quit [Quit: WeeChat 1.9.1]
<carsonfarmerDisc>
get them to send you the token, and their public address (their peer id is enough), once they've done that, encrypt the file for them using their public key, add it to ipfs, and send them the cid
<carsonfarmerDisc>
now they are the only ones who can decrypt the file
<carsonfarmerDisc>
you'll obviously have to manage access tokens etc, but that's pretty much a requirement of your above proposed system anyway
<carsonfarmerDisc>
(in the above scenario, no private information need every be shared, except the access token, for which you should probably use a secure connection)
kn0rki has joined #ipfs
<c0ncord>
carsonfarmerDisc: then what happens when i have multiple requests at one, let's say 1000 requests of the same file, should i encrypt it one thousand times?
<c0ncord>
that could work with small sized files, but middle sized to large files, it's a no-no.
<carsonfarmerDisc>
well if you don't care that multiple peers have the same key, then you can just do it once with a symmetric key, and send them the key over a secure channel
elfatherbrown has quit [Ping timeout: 240 seconds]
KeiraT has quit [Quit: KeiraT]
<carsonfarmerDisc>
but if you are going to be sharing multiple files with peers, you might want to check out our stuff for this kinda thing: github.com/textileio/go-threads
ipfs-stackbot has quit [Remote host closed the connection]
elfatherbrown has joined #ipfs
ipfs-stackbot has joined #ipfs
plexigras has joined #ipfs
symb0l[m] has left #ipfs ["User left"]
<c0ncord>
encryption is not practical, it's an overkill
<c0ncord>
especially for large files
elfatherbrown has quit [Ping timeout: 240 seconds]
mithilarun has joined #ipfs
mithilarun has quit [Remote host closed the connection]
mithilarun has joined #ipfs
KeiraT has joined #ipfs
mowcat has quit [Remote host closed the connection]
mart1n has quit [Quit: WeeChat 2.7]
KempfCreative has quit [Ping timeout: 265 seconds]
plexigras has quit [Ping timeout: 268 seconds]
plexigras has joined #ipfs
<c0ncord>
not only on the "server" side, but on the "client" side as well