<KineticSloth8StoneBlue[m>
this has been disouraging for sometime... the biggest complaint I hear from lots of devs, and their reason for refusing to host on SD, is failed uploads... 😦
<DasSkelett[m]>
Interesting, when I tested with big files it returned a 400, not a 408.
<KineticSloth8StoneBlue[m>
someone said 408 is a timeout? vOv
<DasSkelett[m]>
> 408 Request Timeout
<DasSkelett[m]>
> The server timed out waiting for the request. According to HTTP specifications: "The client did not produce a request within the time that the server was prepared to wait. The client MAY repeat the request without modifications at any later time."[41]
<DasSkelett[m]>
You are right
<DasSkelett[m]>
So I guess there is some setting hidden in either the traffic server, the web server or gunicorn or flask that sets 15 minutes as upper request time limit.
<KineticSloth8StoneBlue[m>
hmm.. anyway to change/adjust that, and would it be bad to do so? vOV
<DasSkelett[m]>
There probably is a way. Just need to find where this limit is set.
<DasSkelett[m]>
The web server on alpha has an option `Timeout 300 # Timeout: The number of seconds before receives and sends time out.`. Don't think it's the correct one, we're searching for 900.
<KineticSloth8StoneBlue[m>
when someone tries to upload a file, is there anything, like in a header, that tells the server what size file is coming?
<KineticSloth8StoneBlue[m>
to expect?
<DasSkelett[m]>
I think there's `Content-Length`.
<KineticSloth8StoneBlue[m>
hmmm.. would it be worth considering shortening the time limit for smaller files, and leaving it longer for huge files? vOv
<DasSkelett[m]>
There's another setting `ProxyTimeout 1200`, don't think it's the one we're looking for either.
<DasSkelett[m]>
Not sure if something like this is possible.
<KineticSloth8StoneBlue[m>
IF you can find where it is, and if its easy to change, of course.. lol
<DasSkelett[m]>
Don't know Apache good enough to know whether it is that flexible.
<KineticSloth8StoneBlue[m>
hmm.. well said KNES choked @ ~10min in... finally succeeded @ ~12min... vOv
<KineticSloth8StoneBlue[m>
he says this is, and has always been discouraging for him listing on SD... says he's ready to just fallback to Curse 😬
<KineticSloth8StoneBlue[m>
hmm.. Well said KNES choked @ ~10min in... finally succeeded @ ~12min... vOv
<KineticSloth8StoneBlue[m>
he says this is, and has always been discouraging for him listing on SD... says he's ready to just fallback to Curse 😬
<KineticSloth8StoneBlue[m>
I dont think the actual speed/time the uploads take, is the issue... its the effort to have to have to repeatedly restart them vOv
<DasSkelett[m]>
Sure, having to re-upload again and again is annoying.
HebaruSan[m] has joined #spacedock
<HebaruSan[m]>
So, not to belabor the obvious, but a client senidng a large upload is sending content constantly, should be obvious it's still there. Is the reverse proxy saying that *SpaceDock* is timing out because it doesn't send anything while it waits?
<HebaruSan[m]>
So, not to belabor the obvious, but a client sending a large upload is sending content constantly, should be obvious it's still there. Is the reverse proxy saying that *SpaceDock* is timing out because it doesn't send anything while it waits?
<DasSkelett[m]>
That's what I found in the logs when I tested it back then.
<DasSkelett[m]>
I think the new "SpaceDock: Missfire" error page is returned by the traffic server. So it might be that one getting inpatient? But no way for me to confirm this since I have not access to it.
<DasSkelett[m]>
I think the new "SpaceDock: Missfire" error page that Well got is returned by the traffic server. So it might be that one getting inpatient? But no way for me to confirm this since I have not access to it.
<DasSkelett[m]>
I think the new "SpaceDock: Missfire" error page that Well got is returned by the traffic server. So it might be that one getting inpatient? But no way for me to confirm this since I have no access to it.
VITAS[m] has joined #spacedock
<VITAS[m]>
i s did move the mod storage on the 30th that should solve the Read only issue we had soemtimes
<DasSkelett[m]>
The timeout is a different problem though.
<VITAS[m]>
75mb isnt as large as some 600mb+ mods but maybe theres (apart from inet and client issues) a limit somewhere and/or we should look into another upload script?
<DasSkelett[m]>
I'm pretty sure there _is_ a limit somewhere, not a size limit, but a time limit, because in my tests the connection dropped _exactly_ after 15 minutes.
<VITAS[m]>
sd and rev proxy are new setups since the 30th so stuff could have changed there too
<VITAS[m]>
im using the same cfg files (a little bit less ramcache and diskchache on the rev proxy)
<VITAS[m]>
is there a way to test it without the rev proxy?
<VITAS[m]>
e.g. upload from alpha or the direcct connect url?
<VITAS[m]>
we have to know is it a.) flask b.) client c.) gunicorn d.) apache e.) rev proxy f.) none of those
<VITAS[m]>
The maximum amount of time Traffic Server can remain connected to a client. If the transfer to the client is not complete before this timeout expires, then Traffic Server closes the connection.
<VITAS[m]>
ill check the config
<DasSkelett[m]>
Sounds like the correct option
<VITAS[m]>
ill rais it to 1h
<VITAS[m]>
i think people should check if they need to uipload large mods trough dialup if that isnt enough
<VITAS[m]>
:D
<VITAS[m]>
done
<VITAS[m]>
we need a test
<DasSkelett[m]>
Yeah I'll try.
<HebaruSan[m]>
On the code side, the only option I'm finding for keeping the connection from timing out is breaking large files into smaller chunks
<KineticSloth8StoneBlue[m>
hmmm... but i'm hearing that small, avg files are having the issue repeatedly... ie up to ~50-150MB vOv
<KineticSloth8StoneBlue[m>
and a lot of times, i do hear that the uploader may have relatively slow internet... vOv
<HebaruSan[m]>
Those are still large files 🙂
<VITAS[m]>
by ksp mod standards yes
<VITAS[m]>
well know in a few mins
<KineticSloth8StoneBlue[m>
hmm.. not really... for parts mods
<KineticSloth8StoneBlue[m>
and actually, THAT is exactly who i hear the most compplaints from: parts mod devs
<DasSkelett[m]>
The median in CKAN seems to be around 600-800 KiB, hard to eyeball.
<HebaruSan[m]>
Aaaaaaanyway the point was that IF we would go with a chunking solution, there is no reason to pick 50MB as the chunk size, when 100KB would work just as well
<KineticSloth8StoneBlue[m>
np 😉
<KineticSloth8StoneBlue[m>
just wanted to make clear from *my* dumb pov, is that its NOT just those huge 1GB+ planet mods that have the issue 😉
<DasSkelett[m]>
I should've passed the 15 minute mark now, and the upload is still going strong.
<HebaruSan[m]>
How are you slowing it down? Got a laptop with dialup?
<DasSkelett[m]>
I'm not, I'm pumping full 10 MBit/s into SpaceDock ^^
<KineticSloth8StoneBlue[m>
any chance you are running a DD-WRT router? .. cuz you should be able to throttle your bandwidth yourself with it
<KineticSloth8StoneBlue[m>
I do it with my kidz internet access 😉
<VITAS[m]>
my switch can go down to 645kbit/s too per port
<VITAS[m]>
ups 64
<DasSkelett[m]>
OPNsense, there should also be an option for this.
<VITAS[m]>
but 10mbit/s works too
<VITAS[m]>
:D
<VITAS[m]>
so we can say: test succsessfull lets see if there are any more complaints from now on
<DasSkelett[m]>
Yes, looks very good. Gonna close the issue on GitHub.
<VITAS[m]>
maybe add that we extended the time to 1h
<VITAS[m]>
just in case someone iis interested
<DasSkelett[m]>
Yes
<HebaruSan[m]>
<span class="d-mention d-user">KineticSloth8StoneBlue</span> can you ask users who've had problems to try again?
<KineticSloth8StoneBlue[m>
I guess Well's upload finally succeeded, so next (mod) update for him, to try...
<KineticSloth8StoneBlue[m>
He does say to say "THANK you very much!", tho, and that he hopes this *does* work and makes SD more efficient
<KineticSloth8StoneBlue[m>
oops... to clarify, it succeeded right before you all changed it 😛
<VITAS[m]>
so he just got it done in under 15mins?
<VITAS[m]>
at least now its not a game of chance anymore
<HebaruSan[m]>
Hmm, if people were timing out in 15 min with 75 MB files, then 1 hour just translates to 300 MB. Seems like this will still be an issue, just less severe.
<HebaruSan[m]>
Ideally a download in progress just wouldn't time out, no matter how long it lasted, but other routes would still behave as before
<VITAS[m]>
i dont want to raise the limit to high can cause new problems
<HebaruSan[m]>
Right, I'm thinking more of some kind of coding solution
<VITAS[m]>
my opinion is: if you take longer than 1h to upload you might want to try some public wifi or another faster way
<HebaruSan[m]>
A download in progress just shouldn't trigger the traffic server's (reasonable) idle limits
<VITAS[m]>
you have to chunk it
<VITAS[m]>
and thus have a new connection
DasSkelett[m] has quit [Quit: Client limit exceeded: 6]
<VITAS[m]>
imagine coming up with a mod that takes over 1h to fit trough your inet connection
KineticSloth8StoneBlue[m has quit [Quit: Client limit exceeded: 6]
DasSkelett[m] has joined #spacedock
<DasSkelett[m]>
FWIF, active downloads shouldn't time out by default:
<DasSkelett[m]>
> proxy.config.http.transaction_active_timeout_out is disabled (set to 0) by default. proxy.config.http.transaction_active_timeout_in is set to 900 seconds by default.
<HebaruSan[m]>
*edit:* ~~A download in progress just shouldn't trigger the traffic server's (reasonable) idle limits~~ -> An upload in progress just shouldn't trigger the traffic server's (reasonable) idle limits
<HebaruSan[m]>
*edit:* ~~Ideally a download in progress just wouldn't time out, no matter how long it lasted, but other routes would still behave as before~~ -> Ideally an upload in progress just wouldn't time out, no matter how long it lasted, but other routes would still behave as before
<VITAS[m]>
youre welcome to implement chunking :)
<HebaruSan[m]>
Chunking disgusts me as a concept, but it seems to be the only true solution to this
<HebaruSan[m]>
(So I'm conflicted)
KineticSloth8StoneBlue[m] has joined #spacedock
<KineticSloth8StoneBlue[m]>
yeah.. thats why i was wondering if the limit could be based on somehow on expected filesize, if SD could detect the expected filesize at the begining of the upload vOv
<HebaruSan[m]>
<span class="d-mention d-user">KineticSloth8StoneBlue</span> as I understand the status quo, no, no SpaceDock-owned code actually runs until *after* the upload is complete
<HebaruSan[m]>
Before that it's all Flask
<HebaruSan[m]>
And you'd think that Flask might have some kind of keep-alive message it could send, but I found nothing of the sort
DasSkelett[m] has quit [Quit: Client limit exceeded: 6]
<VITAS[m]>
the limit exists to prevent to many connections
<VITAS[m]>
and misuse of those connections
<VITAS[m]>
so i dont want to raise the limit to high
<HebaruSan[m]>
Yeah, and that's legit, and actually argues for a shorter limit.
<HebaruSan[m]>
If somebody wanted to connect a bot farm all at once and just leave them hanging without sending a GET or POST, 1 minute would make more sense.
<VITAS[m]>
HebaruSan: you could check out alternative uploading scripts we can plop into the template
<VITAS[m]>
the one we are using is quite dated
<VITAS[m]>
im sure there are better ones now
<KineticSloth8StoneBlue[m]>
exactly... thats why I would think it would be nice to be able to detect the incoming filesize, before adjusting the limit too long vOv
<VITAS[m]>
sd cant adjust the con length
<VITAS[m]>
the filesize is always detected
<VITAS[m]>
its part of the protocol
<VITAS[m]>
i think this is a luxury problem now
<KineticSloth8StoneBlue[m]>
i thought it might be part of the protocol
<VITAS[m]>
im ofcause always for improvements
<VITAS[m]>
no the limit is set in the reverse proxy
<VITAS[m]>
the gatekeeper for the whole system
<VITAS[m]>
sd is just one vm on that system
<VITAS[m]>
it can (to a point) send http header flags to controll the proxy but not every aspect like global hard limits
<HebaruSan[m]>
I just kind of dread the new bugs that might come out with chunking
<HebaruSan[m]>
I just kind of dread the new bugs that might come out with chunking
<HebaruSan[m]>
In cases that are working fine today
<HebaruSan[m]>
I don't think anybody's contesting that this is a bug or that it would be nice to fix it <span class="d-mention d-user">KineticSloth8StoneBlue</span>
<VITAS[m]>
to make it more clear how the setup works:
<VITAS[m]>
spacedock (gunicorn) -> apache webserver (combines dynamic and static content) -> ats /apache traffic server / reverse proxy (for aching and security)
<KineticSloth8StoneBlue[m]>
i know... 😉
<KineticSloth8StoneBlue[m]>
you all have been supre awesome
<VITAS[m]>
there insnt a bandwidth problem on my side/ SD side
<VITAS[m]>
not even at release
<KineticSloth8StoneBlue[m]>
hmm.. interesting... I usually only max out @ ~3-3.5MBs download, when most other places, i hit ~7-8 ...vOv
<VITAS[m]>
its more like since we streamlined SD years ago (and continue to do so) the server is (unless theres another DDOS attack) half asleep
<KineticSloth8StoneBlue[m]>
*edit:* ~~hmm.. interesting... I usually only max out @ ~3-3.5MBs download, when most other places, i hit ~7-8 ...vOv~~ -> hmm.. interesting... I usually only max out @ ~3-3.5MBs download on SD, when most other places, i hit ~7-8 ...vOv
<VITAS[m]>
i think the individual limit is 500 Mbit/s
<HebaruSan[m]>
Yeah how are CPU loads looking now? Do we still need all those gunicorn workers?
<VITAS[m]>
transfer speeds have to do with the size of a file
<VITAS[m]>
or more like how long its transfer lasts
<KineticSloth8StoneBlue[m]>
well, thats well above my (supposed) 100mbps down
<VITAS[m]>
connections (in general on the internet) ramp up speeds over time
<VITAS[m]>
if you dl a lot of small files it has no chance to do so
<KineticSloth8StoneBlue[m]>
really??... hmmm.. with my ISP... its the opposite 😦
<VITAS[m]>
no its part of the ip protocol
<HebaruSan[m]>
TCP
<VITAS[m]>
ok
<VITAS[m]>
right
<VITAS[m]>
tcp
<HebaruSan[m]>
Which also contains various hacks to speed up artificially at the start IIRC
<VITAS[m]>
imagine it like your computer gaining trust in a connection
<VITAS[m]>
that it doesnt corrupt data if transfered at higher speeds
<VITAS[m]>
or larger sizes
<HebaruSan[m]>
It's just because TCP adapts to whatever bandwidth you have, which it doesn't know in advance
<HebaruSan[m]>
When to send the next packet? Hard question in the general case
<VITAS[m]>
try dl some of those large mods with 1GB plus
<VITAS[m]>
you will notice
<KineticSloth8StoneBlue[m]>
idk... my download speeds on large files are pretty consistent vOv
<KineticSloth8StoneBlue[m]>
but I sometimes know my ISP lets me max for the 1st 30secs or so, then it can drop quite a bit, but be consistent the rest of the download
<VITAS[m]>
ISPs are known to amputate the interweb
<KineticSloth8StoneBlue[m]>
where is the SD server actually located at?
<VITAS[m]>
germany
<KineticSloth8StoneBlue[m]>
thought mebbe 😉
<KineticSloth8StoneBlue[m]>
was always wondering if it could be EU traffic limiting my download speeds from SD vOv
<VITAS[m]>
EU traffic limiting?
<VITAS[m]>
whats that?
<KineticSloth8StoneBlue[m]>
i mean mebbe just an EU link somewhere
<VITAS[m]>
never heared that traffic from a specific continent is restricted except in africa, china, russia and north korea
<VITAS[m]>
* africa has actual problems with inet bandwith. there arent enough cables going there
<VITAS[m]>
states beeing dicks
HebaruSan[m] has quit [Quit: Client limit exceeded: 6]
DasSkelett[m] has joined #spacedock
<DasSkelett[m]>
It might be your ISP not buying enough transit bandwith.
<KineticSloth8StoneBlue[m]>
idk... I seem to hit 75% or better on most of my large downloads from everywhere, except SD 😛
<DasSkelett[m]>
Then it might be the transit provider of your ISP not having a good connection to Hetzner, direct or indirect.
<KineticSloth8StoneBlue[m]>
I seem to hit ~30-40% from SD
<KineticSloth8StoneBlue[m]>
consistently on both counts
<VITAS[m]>
large sites have CDNs
<KineticSloth8StoneBlue[m]>
vOv
<VITAS[m]>
so they have servers closer to you
<VITAS[m]>
that mirror files
<KineticSloth8StoneBlue[m]>
ahhh
<KineticSloth8StoneBlue[m]>
ahhh 😉
<VITAS[m]>
still: its mroe about restrictions along the way and not with SD itself
<VITAS[m]>
more
<KineticSloth8StoneBlue[m]>
yeah... figured as much
<KineticSloth8StoneBlue[m]>
not complaining... I've always been happy with SD traffic 😉
<DasSkelett[m]>
Yeah, interconnection is complicated, and most ISPs suck at it.
<VITAS[m]>
yes im just surprised and wanted to find out if theres something i can improve
<VITAS[m]>
and please noone suggest mand in the middle caching services now :)
<KineticSloth8StoneBlue[m]>
i doubt.. i was just curious why the difference seems always consistent for me
HebaruSan[m] has joined #spacedock
<HebaruSan[m]>
Woo! I guessed the docker command to get into my local frontend!
<HebaruSan[m]>
Woo! I guessed the docker command to get into my local frontend!
<HebaruSan[m]>
(It wasn't hard)
<DasSkelett[m]>
Anyways, I can confirm that uncached files from SpaceDock to a netcup hosted VM, who have direct peering with Hetzner, download at 5-10 MB/s = 40-80 MBit/s, cached files are 70-80 MB/s = 560-640 MBit/s.
<DasSkelett[m]>
Good job <span class="d-mention d-user">HebaruSan</span> !
<VITAS[m]>
-it
<VITAS[m]>
:)
KineticSloth8StoneBlue[m] has quit [Quit: Client limit exceeded: 6]
<HebaruSan[m]>
Thoughts on building frontend with browserify? I think I have it working in a local branch
<VITAS[m]>
everything that cleans up the frontend
<VITAS[m]>
but there are several attempts at that allready
DasSkelett[m] has quit [Quit: Client limit exceeded: 6]
HebaruSan[m] has quit [Quit: Idle timeout reached: 10800s]
Darklight[m] has joined #spacedock
<Darklight[m]>
Not sure if you figured it out yet but gunicorn is set to a 1200 second timeout
<Darklight[m]>
I guess we could bump it, something about gunicorns bad handling of lingering connections
<Darklight[m]>
1200 seconds is 20mins though ;)
HebaruSan[m] has joined #spacedock
<HebaruSan[m]>
gunicorn should be keeping busy during file uploads
<VITAS[m]>
gunicorn is always bussy thats the problem :)
<VITAS[m]>
one of the major problems with kerbalstuff was that they ran only one instance without static content filters (everything was served by gunicorn) so only one person could req at one time
<VITAS[m]>
it isnt like that anymore but threads are still limited
<VITAS[m]>
i guess we could increase a bit to e.g. 30mins
<VITAS[m]>
but i think chunking is still a good idea
<Darklight[m]>
Is there a problem with very long timeouts?
<VITAS[m]>
potentialy
<VITAS[m]>
you can keep alive connections easier
<VITAS[m]>
and block resources
<VITAS[m]>
and i dont know if zombie connections with long timeouts are a thing
<Darklight[m]>
I guess the other way would be an "import from URL" feature, but that could he flakey
<Darklight[m]>
Aaanyway, gtg to work ;)
<VITAS[m]>
im good with times up to 1h
<VITAS[m]>
thats what i set ats to
<VITAS[m]>
if there is a way that lets e.g. apache controll uploads and lets us keep gunicorn out of the picture and at a low timeout....