<DasSkelett[m]>
Okay, PR should be merge-ready now
<DasSkelett[m]>
There are mods without default version in the database 😦
<VITAS[m]>
and now?
<VITAS[m]>
set a default for the column?
<HebaruSan[m]>
Can we search for them and delete them?
<VITAS[m]>
delete mods?
<DasSkelett[m]>
We could set the latest version version of each mod without default version as default version, however I suspect there are also mods without any version at all.
<VITAS[m]>
if they dont have a default ver are they still functional?
<HebaruSan[m]>
Nope
<DasSkelett[m]>
No
<VITAS[m]>
k so we should delete them in the long run and prevent them from exsting again
<VITAS[m]>
what the best solution now
<VITAS[m]>
as a quick fix to get this running?
<VITAS[m]>
we are on a clock here :)
<HebaruSan[m]>
Well what effect should that have on scores
<VITAS[m]>
can you test it on alpha if it doesnt cause any new problems there
<HebaruSan[m]>
One thing at a time, it has to get past code review first
<DasSkelett[m]>
✅ approved
<DasSkelett[m]>
We can only test the "should be" case on alpha, not what's happening on production, because it's not possible to set default_version(_id) to null, not even manually.
<VITAS[m]>
i know
<VITAS[m]>
i want to prevent new errors
<VITAS[m]>
well talk about how to improve testing afterwards
<HebaruSan[m]>
<span class="d-mention d-user">DasSkelett</span> go ahead and do migrations or whatever on the alpha server
<HebaruSan[m]>
I won't mess with it
<DasSkelett[m]>
Okay, hang on
<DasSkelett[m]>
Well, alpha also likes to freeze on downgrades
<DasSkelett[m]>
Problem is I can't restart the database for alpha
<HebaruSan[m]>
Can't we? We can sudo
<HebaruSan[m]>
Should be able to do anything I thought
<DasSkelett[m]>
Database is external, on another server
<DasSkelett[m]>
We're not running the docker containers there
<HebaruSan[m]>
Oy
<VITAS[m]>
the db server also runs other things too
<DasSkelett[m]>
Like if the offset is somehow screwef up.
<DasSkelett[m]>
This is sooo weird. Can this be caused by us directly accessing the web server without ATS in front of it? I have no idea where our code could cause this.
<HebaruSan[m]>
Or repeated failed migrations somehow corrupting the db?
<VITAS[m]>
youre using ats
<VITAS[m]>
i can reload that sec
<DasSkelett[m]>
Oh we are? Hmm.
<DasSkelett[m]>
Would be a weir corruption. The db always returns mod, and it seems to have all of them, it just never returns the one we ask for.
<DasSkelett[m]>
Would be a weird corruption. The db always returns mod, and it seems to have all of them, it just never returns the ones we ask for.
<VITAS[m]>
restarted
<DasSkelett[m]>
Mhh, didn't help :/
<HebaruSan[m]>
Maybe the `and_filters` thing doesn't work when empty?
<HebaruSan[m]>
All signs seem to point to something within `search_mods`
<HebaruSan[m]>
Maybe another glitch from default_version being None, or other similar corruption?
<VITAS[m]>
i remember a bug like this years ago but have no idea why this happened
<VITAS[m]>
does the db return the full set ?
<HebaruSan[m]>
It's supposed to only return what is needed now
<VITAS[m]>
aka is it a problem with db querys or display
<VITAS[m]>
the slowness is because im running rsync
<VITAS[m]>
on the same host
<DasSkelett[m]>
/browse/top is not that slow, only /api/browse/top. But it even calls the same method to get the mods.
<VITAS[m]>
caching does things?
<VITAS[m]>
api wouldnt be cached i think
<VITAS[m]>
anyways
<VITAS[m]>
problem at hand :)
<DasSkelett[m]>
Maybe it doesn't like the mods not being in the storage?
<DasSkelett[m]>
It would be interesting to see what queries are made to the database.
<DasSkelett[m]>
And what the db returns.
<VITAS[m]>
can we?
<DasSkelett[m]>
You can configure postgres to log queries
<VITAS[m]>
is there another way to get this working now?
<VITAS[m]>
and add it to the lsit we have to look at later?
<VITAS[m]>
list
<DasSkelett[m]>
I mean, all the other pages seem to work.
<DasSkelett[m]>
*/browse/top would be broken, but that doesn't affect the rest of the site.
<DasSkelett[m]>
*/browse/top would be broken, but that doesn't affect the rest of the site.
<VITAS[m]>
ok
<DasSkelett[m]>
Speaking of which, you are still moving the storage, right?
<VITAS[m]>
yes
<VITAS[m]>
i would have thought it would be faster
<VITAS[m]>
500mbit/s
<DasSkelett[m]>
Although, if that behaviour is a result of some sort of db corruption during the upgrade process, we should fix that before we turn the site on again, in case we have to restore from the backup.
<VITAS[m]>
yes but how likely is that and can we test if its corrupt?
<DasSkelett[m]>
Looking at the mod table might reveal something.
<VITAS[m]>
what do you need me to do?
<DasSkelett[m]>
Look at the score column of the mod table and check if there are values.
<DasSkelett[m]>
Exactly. And that's the reason I don't saw it on my system nor alpha nor beta, because all mods combined don't have more than 30 releases.
<VITAS[m]>
just do SELECT DISTINCT and youre golden for now
<HebaruSan[m]>
Can you try the original query without `JOIN modversion ON mod.id = modversion.mod_id` ?
<VITAS[m]>
unique results
<HebaruSan[m]>
OK, <span class="d-mention d-user">DasSkelett</span> you want me to PR that, or do you want to look at it more?
<DasSkelett[m]>
PR it. Just tried removing `.join(Mod.versions)` and it seems to work.
<VITAS[m]>
as i said select distinct works without removing it
<HebaruSan[m]>
Yeah but that's probably making the DB do more work and whatnot, the idea is to do it correctly
<VITAS[m]>
correctly is to fix the join :)
<VITAS[m]>
fast is to remove the join
<HebaruSan[m]>
Depends if the join is neede
<HebaruSan[m]>
Depends if the join is needed
<VITAS[m]>
i cant say
<HebaruSan[m]>
If it's not supposed to be there, then removing it is correct
<HebaruSan[m]>
This function is searching for mods, not mod versions
<VITAS[m]>
im assume that its there for a reason
<VITAS[m]>
if that data isnt needed youre correct that its the correct way
<HebaruSan[m]>
<span class="d-mention d-user">VITAS</span> don't worry it's going to be so efficient after all this, the old code was working around so many of its own bugs with hacks
<VITAS[m]>
it is
<VITAS[m]>
Diamondriverplays:
<Diamondriverplays[m]>
Any eta on when its up
<VITAS[m]>
i hope it will be back up soon
<Diamondriverplays[m]>
k
<VITAS[m]>
1h
<VITAS[m]>
seems like my error page is doing its job
DasSkelett[m] has quit [Quit: Client limit exceeded: 6]
HebaruSan[m] has quit [Quit: Client limit exceeded: 6]
DasSkelett[m] has joined #spacedock
<DasSkelett[m]>
lol
<VITAS[m]>
im working on it :)
Diamondriverplays[m] has quit [Client Quit]
<VITAS[m]>
cfcx means connection failed connect
HebaruSan[m] has joined #spacedock
<HebaruSan[m]>
OK, pull master now pls
<VictorMout[m]>
wth that bot seems awfully intelligent
<HebaruSan[m]>
It's a bridge bot, it posts what VITAS says in another channel
<VITAS[m]>
so apache webserver on sd host isnt runnign (as it should be)
<VictorMout[m]>
ah
<VITAS[m]>
sec i pull
<VITAS[m]>
and thx VictorMout
<VITAS[m]>
:D
<VITAS[m]>
i assume i dont have to migrate the db
<HebaruSan[m]>
Right
<DasSkelett[m]>
No
<VITAS[m]>
done
<DasSkelett[m]>
It works! 🎉
<HebaruSan[m]>
If we could come up with an alembic migration in this timeframe and test it that would be big brain status for sho
<HebaruSan[m]>
WOOO
<HebaruSan[m]>
Good investigation, I would not have figured that one out by myself, I don't think
<DasSkelett[m]>
And no more BDArmory on page 2!!! 😄
VictorMout[m] has quit [Client Quit]
Savage[m] has joined #spacedock
<Savage[m]>
What happened to Space dock?
<Savage[m]>
D:
<VITAS[m]>
updating the crap out of it
<HebaruSan[m]>
Planned downtime
<VITAS[m]>
Savage will be up soonish
<VITAS[m]>
see and thats why i wanted to link a page
<Savage[m]>
Of course, Haven't played KSP in literal months, but 2am on a random night i decide to play it's down. Just my fucking luck
<Savage[m]>
F
<HebaruSan[m]>
And that's why I wanted the page to say "Planned downtime" 🙂
<DasSkelett[m]>
<span class="d-mention d-user">VITAS</span> is it safe if I already test mod creation? Or should I wait for the storage to finish transfering?
<VITAS[m]>
lol
<VITAS[m]>
bad luck i guess
<Savage[m]>
Yea 100% should say it's planned or some shit
<Savage[m]>
I thought it got like perma shut down
<VITAS[m]>
uhmm you can test creation but not update
<Savage[m]>
that page isn't very informative
<VITAS[m]>
suggestions?
<VITAS[m]>
Savage
<Savage[m]>
well, Don't start with "SpaceDock isn't here." Sounds like it's like gone gone
<Savage[m]>
Maintenance, updates, something that says 1.You know it's down. 2.You're working on it 3.Will be back soon-ish
Webchat533 has joined #spacedock
Webchat533 has quit [Client Quit]
<Savage[m]>
a bit vague
<HebaruSan[m]>
<span class="d-mention d-user">VITAS</span> I think everything is working, <span class="d-mention d-user">DasSkelett</span> you notice anything else off?
<VITAS[m]>
the message is also shown if there is another error not only downtime
<VITAS[m]>
i dont always know it is
<HebaruSan[m]>
Oh I should check some API stuff
<VITAS[m]>
sometimes it crashes
<VITAS[m]>
:D
<VITAS[m]>
or you made the server shit itself
<Savage[m]>
Ahhh, is there anyway to differentiate? multiple caches images to present for different down reasons?
<HebaruSan[m]>
Time zones are present, hooray
<DasSkelett[m]>
So far everything I tested works. Didn't test mod creation and update just yet though.
<VITAS[m]>
yes there is and youre welcome to stick arround and help me make better ones after were done
<VITAS[m]>
:)
<Savage[m]>
getting married, and moving, and have active orders, all in the next week
<Savage[m]>
This is like my last bit o free time for a while
<Savage[m]>
But sure
<HebaruSan[m]>
Some of the stuff you want may be here:
<VITAS[m]>
i wanted to ad a link to a page where we can post more info
RB101[m] has quit [Client Quit]
HebaruSan[m] has joined #spacedock
<HebaruSan[m]>
It is high in visual appeal
<FireStriker[m]>
thats a good idea
<VITAS[m]>
*looks at HebaruSan and DasSkelett *
<VITAS[m]>
seeeee!
<VITAS[m]>
:D
<Savage[m]>
Hey Vitas
<Savage[m]>
Question
<HebaruSan[m]>
Dude none of this is news to us
<HebaruSan[m]>
We told you the landing page should explain it
<VITAS[m]>
Savage will save us (in terms of error pages)
<Savage[m]>
eventaully
<Savage[m]>
eventually
<VITAS[m]>
i believe in you
<Savage[m]>
: D
<Savage[m]>
😄
<Savage[m]>
: D
<VITAS[m]>
savage shell but soft core
<Savage[m]>
#0000
<Savage[m]>
wtf
<VITAS[m]>
letter T
<Savage[m]>
How did you 😂
<Savage[m]>
How did you : D
<FireStriker[m]>
i could see, Discord Link + Error was gonna end with a bunch of people entering and asking why its down and if your lucky mentioning cfcx.
<FireStriker[m]>
Im surprised i have only seen one join and ask since i joined
<VITAS[m]>
youre a ksp fan you must be a nice person
<VITAS[m]>
my official ttitle was: senior it consultant for corporate security access control
<VITAS[m]>
so basicly the person who has the keys to every thing that controlls the keys to everything
<FireStriker[m]>
did you have to tell them to turn the 787s off and on again?
<VITAS[m]>
no i revoked access cards
<VITAS[m]>
the last one was my own :D
<FireStriker[m]>
😦
<Savage[m]>
u sound like ur job was pretty high up
<VITAS[m]>
it was well paying until it wasnt
<Savage[m]>
You don't say "mock surprise :O"
<FireStriker[m]>
if you didnt known if the 787 was on ground power for a month or so it had to be powered down or the guidance would malfunction
<VITAS[m]>
btw airbus final assembly is in hamburg too
<FireStriker[m]>
it could be fixed but turning it off before 30 days avoided the issue all togther
<FireStriker[m]>
i wonder how many 787s are gonna need attention after covid
<FireStriker[m]>
wait no not ground power
<FireStriker[m]>
just siting not powered
<VITAS[m]>
so when some saudi prince bought a new private plane they where assembled at airbus, send across the city to lufthansa technik and where customized to the buyers liking
DasSkelett[m] has quit [Quit: Client limit exceeded: 6]
VITAS[m] has joined #spacedock
<VITAS[m]>
somehow stuff froms torage still isnt loading
FireStriker[m] has quit [Quit: Client limit exceeded: 6]
Darklight[m] has joined #spacedock
<Darklight[m]>
Did I miss anything?
<HebaruSan[m]>
3 hotfixes
Webchat111 has joined #spacedock
FreeThinker[m] has quit [Quit: Client limit exceeded: 6]
VITAS[m] has quit [Quit: Client limit exceeded: 6]
KarolOfGutovo[m] has joined #spacedock
<KarolOfGutovo[m]>
Should spacedock be up rn?
DasSkelett[m] has joined #spacedock
<DasSkelett[m]>
Some thumbs load, some don't. Can you check whether `godarklight_151/DarkMultiPlayer/thumb_DarkMultiPlayer-1456094845.1546948.png` exists in the storage?
<Webchat111>
what happend
<DasSkelett[m]>
Planned maintenance @Webchat111
<KarolOfGutovo[m]>
How long is it estimated to last?
<Webchat111>
i miss it already
<Webchat111>
when will it be up and running again
<DasSkelett[m]>
Unknown <span class="d-mention d-user">KarolOfGutovo</span> @Webchat111, maybe one or two hours, possibly longer. Hard to say.
<Webchat111>
ok thx
<KarolOfGutovo[m]>
Nice. Can't wait to download another share of mods that I will never use but appreciate nonetheless
<Webchat111>
also can i ask a question
Darklight[m] has quit [Client Quit]
FireStriker[m] has joined #spacedock
<FireStriker[m]>
be patient it will be up soon™️
HebaruSan[m] has quit [Quit: Client limit exceeded: 6]
Darklight[m] has joined #spacedock
<Darklight[m]>
Errr, when did the update start?
<FireStriker[m]>
um
<Darklight[m]>
On pc now so I can check dasskellet
<Webchat111>
how do you update mods because 1.10 is tomorow
<DasSkelett[m]>
06:02 UTC
<Darklight[m]>
16:02 my time oof
<Darklight[m]>
Hours ago
<Webchat111>
7pm in my time then
<FireStriker[m]>
17:04 here
<Webchat111>
so how do you update mods on spacedock
<Webchat111>
because i only started using it for a couple of weeks
<DasSkelett[m]>
You can't right now @Webchat111. Be patient.
<Webchat111>
i mean after its online
<FireStriker[m]>
are you updating your mod?
<Darklight[m]>
darklight@sd1:/srv/storage/kspmods/godarklight_151/DarkMultiPlayer$ ls -alh /srv/storage/kspmods/godarklight_151/DarkMultiPlayer/thumb_DarkMultiPlayer-1456094845.1546948.png
<FireStriker[m]>
If a mod is not working and isnt getting maintained you could always look at the post by LinuxGuruGamer on how to maintain a mod. This is if it doesnt have a NoDerivatives license. If it has one of them you are just gonna have to wait for the author to return
<FireStriker[m]>
how does that irc bot work?
<Darklight[m]>
Odd.... 50's
<Darklight[m]>
Odd.... 502's
<VITAS[m]>
yes
<VITAS[m]>
all the errors
<VITAS[m]>
i only get them when acc either stuff from storage or using internal domains
<VITAS[m]>
not with lan ip or external domain and dynamic or static content from gunicorn
<Darklight[m]>
I'm using the actual DNS for the new endpoint, I am not seeing any new accesses in apache so the problem exists higher up maybe? I'll try an ssh tunnel in and see if it's fine
<VITAS[m]>
you should check the error log in /var/log/apache2/spacedock.info-access.log
<VITAS[m]>
i see acc there
<VITAS[m]>
i think it has to do with apache webserver config and storage config
<VITAS[m]>
sd itself is fine i think
<Darklight[m]>
Oh I thought it was in the root folder
<VITAS[m]>
i moved it
<VITAS[m]>
didnt delete it
Webchat818 has joined #spacedock
<VITAS[m]>
maybe acc rigths or paths to storage?
<VITAS[m]>
the apache config worked before on that host
Darklight[m] has quit [Quit: Client limit exceeded: 6]
VITAS[m] has joined #spacedock
<VITAS[m]>
the chat should always be that popular
<FireStriker[m]>
i was about to ask does cdcx mean anything or is it random?
DasSkelett[m] has quit [Quit: Client limit exceeded: 6]
Jordi2006[m] has quit [Client Quit]
private9001[m] has joined #spacedock
<private9001[m]>
do you have an eta of when the site will be back up?
DasSkelett[m] has joined #spacedock
<DasSkelett[m]>
<span class="d-mention d-user">VITAS</span> easy, just take down SpaceDock regularly 😛
VITAS[m] has quit [Client Quit]
Darklight[m] has joined #spacedock
<Darklight[m]>
I think vitas did something as it is 403 😛
<FireStriker[m]>
Was the eta 14:00 UTC?
<DasSkelett[m]>
Well, everything is a 403 now.
<FireStriker[m]>
could be shorter could be longer
<private9001[m]>
ok thank you
<private9001[m]>
ok thank you
<FireStriker[m]>
i plan to keep this place active after the downtime
<FireStriker[m]>
nice foke in here
<Darklight[m]>
Give us a little bit is all, we are just seeing some very strange stuff
<FireStriker[m]>
^
<Darklight[m]>
I don't get why we're getting weird headers, I've only seen this once before in a bug with uhttpserver with LMP (nothing to do with spacedock)
<FireStriker[m]>
what kind of funky headers
DasSkelett[m] has quit [Client Quit]
VITAS[m] has joined #spacedock
<VITAS[m]>
i think its you
<VITAS[m]>
because i get normal headers
private9001[m] has quit [Client Quit]
DasSkelett[m] has joined #spacedock
<DasSkelett[m]>
Oh we're back to 502s now.
<FireStriker[m]>
as long as we dont get a 404
<VITAS[m]>
yes i know because im playing with the apache config
FireStriker[m] has quit [Quit: Client limit exceeded: 6]
HebaruSan[m] has joined #spacedock
<HebaruSan[m]>
Alright, I'm going offline for a couple of hours. Good luck.
<Darklight[m]>
I think what may be worth doing is seeing if apache is using different mods between old prod and new prod
<Darklight[m]>
I'll check that out
<VITAS[m]>
so to sum it up: we dont know whats wrong. we got half working by commenting out bits and the other half isnt for no apparent reason
<VITAS[m]>
good times :/
<VITAS[m]>
i can go by ip if needed
<VITAS[m]>
seems like we have to fix some stuff soon anyway
<VITAS[m]>
my thoughts: lets go with lb and without bypass for content for now and try to come up with a solution on sd1b (it should act the same way)
<Darklight[m]>
The only difference is the old server has mod_proxy_hcheck
<Darklight[m]>
I guess I can add that...
<VITAS[m]>
yes
<VITAS[m]>
just remember add everything on a & b
<VITAS[m]>
not only on one
Webchat374 has joined #spacedock
<VITAS[m]>
we wills cratch our hads later if we dont
<VITAS[m]>
< hands out vouchers
<VITAS[m]>
sorry for the inconvinience
DasSkelett[m] has quit [Quit: Client limit exceeded: 6]
FireStriker[m] has joined #spacedock
<FireStriker[m]>
whats it doing this time
<VITAS[m]>
we dont know thats the problem
<VITAS[m]>
:D
<VITAS[m]>
in a nutshell: i set up a new host to run SD from. i copied over the mod storage folder and got it working. today we updated the code and i synced the mod storage. somehow its now throwing 400 errors when accessing the mod storage
<FireStriker[m]>
oh so like all the mm patches i write, matches the syntax, nothing wrong, ERROR
<FireStriker[m]>
im currently troubleshooting 2
<FireStriker[m]>
one that is a warning and one thats an error
<Darklight[m]>
Going to change some stuff back to prod-like, if the error exists with hcheck we should just disable proxypass for content - gunicorn offloads it anyway
<VITAS[m]>
yes
<VITAS[m]>
and then find a more permanent solution suing sd1b
<VITAS[m]>
using
<FireStriker[m]>
was the new code working before the migration?
HebaruSan[m] has quit [Quit: Client limit exceeded: 6]
<VITAS[m]>
yes
<VITAS[m]>
we have testing
<VITAS[m]>
at least some
Darklight[m] has quit [Quit: Client limit exceeded: 6]
DasSkelett[m] has joined #spacedock
<DasSkelett[m]>
Well, mostly.
<VITAS[m]>
but we could use beta testers
Webchat374 has quit [Quit: webchat.esper.net]
<FireStriker[m]>
i wonder what broke during the migration then
<FireStriker[m]>
I didnt think spacedock was so touch wood lol
<DasSkelett[m]>
Currently they're working on the updated Apache Web Server behaving differently than the old one.
<VITAS[m]>
uhm it is crapy old code we try to improve over time
FireStriker[m] has quit [Quit: Client limit exceeded: 6]
Darklight[m] has joined #spacedock
<Darklight[m]>
We are clearly fighting an apache bug
<Darklight[m]>
Maybe something is odd about mount
<Darklight[m]>
Let's not proxypass, I can try to reproduce this later with those mount settings
<VITAS[m]>
i remember when i set the host up i had similar issues and got it to work without knowing why
<Darklight[m]>
For now let's just get it up I think
DasSkelett[m] has quit [Client Quit]
FireStriker[m] has joined #spacedock
<FireStriker[m]>
now how do we get the bug out of the apache, it could do some damage in that
<FireStriker[m]>
mind my puns
<VITAS[m]>
something todo with chars in the config that disturbed it and i couldnt see or soemthing
<VITAS[m]>
FireStriker: my finger nails curl from that level of pun
<Darklight[m]>
I've set it back to prod-like, just with content not proxypass'd but remember we do have xsendfile
<Darklight[m]>
I'd say get skellet to try uploading
<VITAS[m]>
DasSkelett: ?
<VITAS[m]>
go
DasSkelett[m] has joined #spacedock
<DasSkelett[m]>
Okay
Webchat854 has joined #spacedock
Webchat854 has quit [Client Quit]
FireStriker[m] has quit [Client Quit]
James566[m] has joined #spacedock
<James566[m]>
hey
<James566[m]>
is the side down for maintenance
<VITAS[m]>
yes
<VITAS[m]>
but we are nearly done
<James566[m]>
Ah ok thanks
<Darklight[m]>
I think it will be up soon, just hit a webserver bug for sure
<VITAS[m]>
< waits on DasSkelett
<VITAS[m]>
you might have problems with redirects to spacedock.info
<DasSkelett[m]>
Updating and Creating new mods works, updating old mods works, downloading both old and new mods works when circumventing the redirect by editing the url.
<DasSkelett[m]>
We are Go from my side ✅
<VITAS[m]>
ok
<DasSkelett[m]>
*edit:* ~~Updating and Creating new mods works, updating old mods works, downloading both old and new mods works when circumventing the redirect by editing the url.
<DasSkelett[m]>
We are Go from my side ✅~~ -> Creating new mods works, updating old mods works, downloading both old and new mods works when circumventing the redirect by editing the url.
<RockyTV[m]>
oh it is, but driving me insane lately cuz I haven't been doing anything at all (other than watching people code to try and learn how the repos are structured and how each project works), hopefully it will change today, I already got access to their azure devops
<VITAS[m]>
i had to sort a room with tons of computer cables once
<VITAS[m]>
i know how you feel
<VITAS[m]>
they should give you some test proj where you can try out what you observed
<VITAS[m]>
would help learning
FireStriker[m] has quit [Quit: Client limit exceeded: 6]
Darklight[m] has joined #spacedock
<Darklight[m]>
So I guess I go into sd1b and comment out that same line...
<Darklight[m]>
Done
<VITAS[m]>
yes
<VITAS[m]>
and if you stillw ant to you can play with sd1b to solve the problem
<VITAS[m]>
you can temp use prod db for it
<VITAS[m]>
its then as easy as changing a dns entry to put them into production
<RockyTV[m]>
<span class="d-mention d-user">HebaruSan</span> do you have any screenshots for #293 (modpack fixes)?
DasSkelett[m] has quit [Quit: Client limit exceeded: 6]
KineticSloth8StoneBlue[m] has joined #spacedock
<KineticSloth8StoneBlue[m]>
so there are rumours that supposedly, KSP 1.10 is set to release today? vOv
<VITAS[m]>
yes
<VITAS[m]>
someone (tm) will have to add it when it does
<KineticSloth8StoneBlue[m]>
🤦 I dont think I'll be doing it... I pretty much forgot how, and I'm not about to screw it up, when the KSP update is happening around the *same time* 😦
<RockyTV[m]>
1.10 releases tomorrow, no? july 1st
<VITAS[m]>
timezones?
<VITAS[m]>
ok then i have to
Darklight[m] has quit [Quit: Client limit exceeded: 6]
DasSkelett[m] has joined #spacedock
<DasSkelett[m]>
Speaking of adding game versions, I don't think Balsa has one yet.
<DasSkelett[m]>
(And the search isn't quite working as intended yet...)
<VITAS[m]>
ill check
<VITAS[m]>
< stands up from bed
<VITAS[m]>
btw how bad would it be if i added 1.10 now?
<DasSkelett[m]>
YOu'd get a bunch of confused mod authors thinking they missed the update 😛
<DasSkelett[m]>
You'd get a bunch of confused mod authors thinking they missed the update 😛
<VITAS[m]>
youre right i cant find it either
<VITAS[m]>
strange
<VITAS[m]>
what was the version of the demo again?
<RockyTV[m]>
0.25.something
<VITAS[m]>
not precise enough :)
<KineticSloth8StoneBlue[m]>
ppfffttt... people would complain if 1.10 is set early on SD.... complain if its not set within a couple minutes *after* 1.10 releases ... vOv
<KineticSloth8StoneBlue[m]>
Do whats easiest for SD... 😛
<VITAS[m]>
esiest would be now because im allready adding versions to things
<VITAS[m]>
but ill wait a bit and let you do it :P
DasSkelett[m] has quit [Quit: Client limit exceeded: 6]
Darklight[m] has joined #spacedock
<Darklight[m]>
I added a directory to my "spacedock checker" vm, It's version ``2.4.38-3+deb10u3`` meanwhile ubuntu's is ``2.4.41-4ubuntu3``
<VITAS[m]>
balsa?
<Darklight[m]>
But my server runs debian/unstable which is ``2.4.43-1``
<VITAS[m]>
context?
<Darklight[m]>
I tried serving a cifs fileystem
<Darklight[m]>
It worked fine, as prod always had, it looks like I need to get a ubuntu vm to test..
<VITAS[m]>
what are you exactly trying to do?
<Darklight[m]>
Reproduce the bug
<VITAS[m]>
as i said: try sd1b
<VITAS[m]>
it should have the same bug
<VITAS[m]>
and if you fix it we can switch over to it
<Darklight[m]>
I commented out the proxypass line for now, but I am just proving to myself that it is apache without messing up a prod box
<VITAS[m]>
sd1b isnt live
<VITAS[m]>
apart from changing storage content there isnt anything to mes sup
<VITAS[m]>
its exactly why it exists: to have something that is the same as whats currently serving and we can update without taking down spacedock
<VITAS[m]>
i still need to know balsas exact version number
<VITAS[m]>
for now its demo-0.25
<Darklight[m]>
The demo can't have mods apart from my patched version, when the beta drops though that is when you're going to be able to use mods
<RockyTV[m]>
what bug? storage one?
<RockyTV[m]>
or balsa-related?
<VITAS[m]>
is till want a version to be present
<VITAS[m]>
lets call it a signal of support
<VITAS[m]>
RockyTV: theres a bug in apache webserver we encountered while updating SD Darklight is looking into it
<VITAS[m]>
i on the other hand have to add a version for balsa to the db (because somehow my first attempt failed)
hyx5020 has quit [Quit: webchat.esper.net]
hyx5020 has joined #spacedock
hyx5020 has quit [Client Quit]
KineticSloth8StoneBlue[m] has quit [Quit: Client limit exceeded: 6]
DasSkelett[m] has joined #spacedock
<DasSkelett[m]>
The API is _really_ slow in some cases, not sure what that's about.
Darklight[m] has quit [Quit: Client limit exceeded: 6]
FireStriker[m] has joined #spacedock
<FireStriker[m]>
has anyone ever had any issues with npp
<FireStriker[m]>
im currently fighting the indent system to get it to work correctly
<DasSkelett[m]>
It seems to be `mod_version()` that is slowing things down. So maybe `Modversion.download_count()`?
<DasSkelett[m]>
Nope, not using Notepad++
<FireStriker[m]>
well there goes that lol, if you need me i will be trying to beat it back into shape or figuer out how to turn that feature off
<VITAS[m]>
api was never fast
RockyTV[m] has quit [Quit: Client limit exceeded: 6]
Darklight[m] has joined #spacedock
<Darklight[m]>
I confirmed the bug definitely exists in the latest ubuntu apache2, it does not exist in debian/stable's version, I will test debian/unstable which is newer than ubuntu
<DasSkelett[m]>
It should've been sped up with all the new indices and the score precalculation. But now https://spacedock.info/api/browse/top is timing out.
FireStriker[m] has quit [Quit: Client limit exceeded: 6]
<VITAS[m]>
we could try something different than apache webserver for the job
RockyTV[m] has joined #spacedock
<RockyTV[m]>
any logs?
<DasSkelett[m]>
(All the other /browse pages are timing out too)
<RockyTV[m]>
looks at nginx
<VITAS[m]>
yes
<RockyTV[m]>
spacedock is a bit slower than what it was an hour ago
<VITAS[m]>
because we removed the bypass
<VITAS[m]>
and i might need to tweek some settings in ats and for the sd containers
<DasSkelett[m]>
Everything except the api seems to be quite fast still.
<VITAS[m]>
i gave sd less ram
<VITAS[m]>
10gb instead of 20
<VITAS[m]>
because i think it doesnt need such a big slice
<VITAS[m]>
ats also has to build its cache again
<RockyTV[m]>
wow, /api/browse/top is still loading (about 3 minutes now)
<VITAS[m]>
i think its a good idea to get a more realistic testbed
<VITAS[m]>
cant be tat we only discover what doesnt work when we put it into prod
<VITAS[m]>
we need stuff like db seeding, load simulation and tests
<DasSkelett[m]>
We'd need a db with 2k mods, 45k users, a bunch of versions per mod, a big bunch of download events per mod version.
<DasSkelett[m]>
These are the most performance impacting tables.
<VITAS[m]>
db seeding
<VITAS[m]>
can do that for you
<VITAS[m]>
it fills the db with predefined typs of values
<Darklight[m]>
Confirmed that debian's .43 version is not affected by the bug (although tested it on my real server, so it isn't the most-simple-case, but still...)
<DasSkelett[m]>
MAybe I can firgure out some code to put into the `./spacedock` script for seeding.
<VITAS[m]>
Darklight: could you check if the bug is allready posted?
<VITAS[m]>
DasSkelett: normaly there are libs for it mostly part of the db layer aka alembic
<VITAS[m]>
you dont have to write that by hand
<VITAS[m]>
because the db layer knows the db structure
<VITAS[m]>
you then simply runa command telling it to put x amount of entries in the db
<VITAS[m]>
the laravel vue version of sd i started to code can do that
<DasSkelett[m]>
SqlAlchemy might have an option for that, I don't think Alembic has.
<VITAS[m]>
i can even define: this is a persons name so i want first and last names
<Darklight[m]>
Where sdmods is a cifs mount with these options: ``guest,rw,vers=3.1.1,cache=strict,uid=33,forceuid,gid=33,forcegid,file_mode=0777,dir_mode=0777,soft,nounix,serverino,mapposix,rsize=4194304,wsize=4194304,echo_interval=60,actimeo=1``
<Darklight[m]>
I gotta go now, it's past midnight and I work early tomorrow again
<Darklight[m]>
Short story: ubuntu affected, debian is not
<VITAS[m]>
Darklight: apache should be made aware of the bug if they arent
<DasSkelett[m]>
<span class="d-mention d-user">RockyTV</span> I found that, they are doing it manually there though. Let me check what VITAS found.
<VITAS[m]>
Darklight: thx for your help :)
<Darklight[m]>
Imma sleeeeep
<Darklight[m]>
But yeah all g, hopefully you don't have too many fires left over 😛
<Darklight[m]>
And even then our "bug" isn't a huge one because we have xsendfile
<VITAS[m]>
DasSkelett: it should be something that can be run seperatly as a command and uses the existing migration defnitions as rule or even integrate into them
<VITAS[m]>
with laravel its as easy as defining your db structure and then adding options for the seeder to that config
<DasSkelett[m]>
I don't think this should be an alembic script.
<DasSkelett[m]>
Seeding is not really part of the migration. I would keep it separate, e.g. as command in `./spacedock`
<Darklight[m]>
I don't know what you mean by seeding, I was massively distracted as you saw, but just hammer spacedock's API if you need to have a test database setup
<RockyTV[m]>
^
<RockyTV[m]>
shouldn't be too hard to create 45k users with a for loop
<VITAS[m]>
DasSkelett: it is seperate but to be able to seed it has to know the structure
<DasSkelett[m]>
Directly using SqlAlchemy to insert would probably be faster than going via a webserver
<Darklight[m]>
It definitely is
<Darklight[m]>
Alrighty, going for real ^_^
<DasSkelett[m]>
<span class="d-mention d-user">VITAS</span> our structures are defined in `objects.py`
<DasSkelett[m]>
We might have to remove the `"downloads": version.download_count()` key again.
<VITAS[m]>
and i want to eat my spätzle now. pleas dont kill the site
<VITAS[m]>
:)
<VITAS[m]>
can you somehow ratelimit the bot?
<VITAS[m]>
its againr amping up
<DasSkelett[m]>
Not really
<DasSkelett[m]>
<span class="d-mention d-user">HebaruSan</span> are you around? Do you think it's worth a try to remove the per-version download count from the api again for now?
HebaruSan[m] has joined #spacedock
<HebaruSan[m]>
I'm catching up now. The API is slow?
<HebaruSan[m]>
The test looks for that property, just take that out
<HebaruSan[m]>
Also are we starting in alpha again?
<HebaruSan[m]>
Also are we starting in alpha again?
<DasSkelett[m]>
Ah damn, misclicked.
<HebaruSan[m]>
So what is `DownloadEvent.query.count()` in prod? Like 2 billion?
<DasSkelett[m]>
A lot, I guess.
<HebaruSan[m]>
Are the alpah and beta servers running? I get 502 and 503
<HebaruSan[m]>
Are the alpha and beta servers running? I get 502 and 503
<VITAS[m]>
dont know might also have been affected
<HebaruSan[m]>
OK <span class="d-mention d-user">VITAS</span> pull master to fix the API slowness issue
<VITAS[m]>
on prod?
<VITAS[m]>
!!
<DasSkelett[m]>
Yes
<VITAS[m]>
if that breaks anything i will come and get you
<HebaruSan[m]>
Well deploy it however you want
<HebaruSan[m]>
You don't have to run just a `git pull` because I said so, but the changes are there
<VITAS[m]>
exactly
<VITAS[m]>
i do it because i trust you and you take responsibility for your suggestions :)
<VITAS[m]>
if you dont trust your code enough to do not mess up anything i wont either
<DasSkelett[m]>
We wouldn't have merged it if we though it breaks something.
<DasSkelett[m]>
We wouldn't have merged it if we thought it breaks something.
<HebaruSan[m]>
RIght. I thought we were in emergency mode trying to solve this as it was
<VITAS[m]>
my alternative would be to limit the bots connections
<VITAS[m]>
until we fixed it
<DasSkelett[m]>
This won't solve it. It will always slow down over time, the bot isn't the only one accessing SpaceDock.
<HebaruSan[m]>
`DownloadEvent` is like that old saying, "Water, water, everywhere, but not a drop to drink"
<VITAS[m]>
i dont want you to say stuff like "your choice" i know its mine. it feels like you can say "was your choice" if it explodes
<VITAS[m]>
makes me cautious
<HebaruSan[m]>
Well however you do it, get that change into prod
<HebaruSan[m]>
Otherwise the server will stay broken
<DasSkelett[m]>
But we won't. We know it was my code change + PR, and HebaruSan approved it. If it explodes, we all are at fault. I will never blame you for that.
<HebaruSan[m]>
What about your A/B thing? Could you try it in the inactive one?
<DasSkelett[m]>
Apparently, once inserted SQL doesn't like giving data away again.
<VITAS[m]>
you can use last id
<VITAS[m]>
to get the last incremental id inserted
<DasSkelett[m]>
Best case "unseeding" would still work after I did some other changes to the db, like manual mod uploads.
<VITAS[m]>
just truncate the table and be done with it
<VITAS[m]>
:D
<HebaruSan[m]>
There's nothing at the db level to "reset all tables to previous state" is there?
<VITAS[m]>
if its random test data
<VITAS[m]>
no
<VITAS[m]>
you can do backups
<VITAS[m]>
like dump to sql file and then empty db and run that again
<VITAS[m]>
i think this is getting way more complex than a dirty fix should be :)
<HebaruSan[m]>
Meanwhile I forget <span class="d-mention d-user">DasSkelett</span>, were we ready to merge CKAN#3059?
<DasSkelett[m]>
Good question. It looks like I have already marked most files as "Viewed" (which I use to remember which files I've already reviewed and am happy with).
<DasSkelett[m]>
Let me try it out for SpaceDock mods again, then I think we're good.
<DasSkelett[m]>
So yeah, the subquery thing gets me further, but then it fails becuase of the foreign key constraint and the rows referencing each other.
<DasSkelett[m]>
Looks like we _do_ have to setup the cascade stuff, but for now, I'm going to drop the entire database and recreate it.
<HebaruSan[m]>
Wow, SQL is really picky
pizzaoverhead_ has joined #spacedock
pizzaoverhead has quit [Ping timeout: 378 seconds]
<DasSkelett[m]>
Ouch, creating 10k users takes 60 minutes, even with bulk inserts :/
<HebaruSan[m]>
~3 per second?
<DasSkelett[m]>
Around that, yes. Just clocked up my PC, now I'm at around 6 per second.
<DasSkelett[m]>
That's actually just generating the user objects, inserting should be done in one big bulk at the end.
<DasSkelett[m]>
Maybe I can remove some string concatenations.
<HebaruSan[m]>
Wait are you inserting 10k in one block?
<DasSkelett[m]>
Trying it ^^
RockyTV[m] has joined #spacedock
<HebaruSan[m]>
Or 100 or something
<RockyTV[m]>
try inserting like 1k per block, no?
<RockyTV[m]>
or comitting each single one after you create it 😛
<HebaruSan[m]>
Might this be something to do directly in SQL, without the sqlalchemy layer taking up memory and CPU?
<DasSkelett[m]>
Did insert after each one in the beginning, wasn't very fast either.
<DasSkelett[m]>
Good idea <span class="d-mention d-user">HebaruSan</span> , let's try.
<HebaruSan[m]>
After all all those objects just exist to be compiled down to an SQL statement
<HebaruSan[m]>
(unless my understanding of sqlalchemy is way off, which it may very well be)
<VITAS[m]>
i think its more about one insert per row or one for all
<DasSkelett[m]>
Found the trouble-maker: `user.set_password()`. Secure hashing takes time...
<HebaruSan[m]>
It's always the hashes!
<VITAS[m]>
how do you generate the values?
<DasSkelett[m]>
Hehe, should've known better by now.
<VITAS[m]>
you are still at seeding the db arnt you?
<DasSkelett[m]>
Yes
<VITAS[m]>
how do you generate the values?
<DasSkelett[m]>
Which values? Passwords?
<VITAS[m]>
no to seed the db
RockyTV[m] has quit [Quit: Client limit exceeded: 6]
VictorMout[m] has joined #spacedock
<VictorMout[m]>
You know this might not be the most secure way to talk about modifying your website might wanna make a private channel
<VITAS[m]>
why?
<VITAS[m]>
its open source
<HebaruSan[m]>
They're not saying anything confidential
<DasSkelett[m]>
Oh, I'm not yet filling out much, just what's needed. Currently all the values are the same / count up.
<VictorMout[m]>
Ah i didn’t know that
<VITAS[m]>
youre welcome to join
<DasSkelett[m]>
If nobody's posting any secret connection strings, everything's fine 😬
<HebaruSan[m]>
Also last time I advised VITAS to make a private channel he just made another public one
<VITAS[m]>
DasSkelett: you should randomize vars they can trigger bugs that way
<VITAS[m]>
i dont know much about discord
<DasSkelett[m]>
<span class="d-mention d-user">VITAS</span> yep, planned. But want to get the basic functionality working first, plugging in randomization should be fairly easy afterwards.
<VITAS[m]>
only that it scans my drive and watches my processes
<VITAS[m]>
DasSkelett: you write evrything yourself?
<VITAS[m]>
why?!
<DasSkelett[m]>
Not sure what you mean with writing everything myself.
<VITAS[m]>
if youre using a ready made seeding function you usaly define the type of values your tables have and it generatres matching ones
<VITAS[m]>
e.g. names
<VITAS[m]>
you just tell the seed command how many entries you want
<DasSkelett[m]>
I have yet to find a seeding command that you just need to give a number of rows.
<VITAS[m]>
whatever you use needs to know the structure of your db first
<DasSkelett[m]>
So the whole seeding process with 50k users, 10k mods and 100k mod versions (10 per mod) takes 1m 15s right now.
<VITAS[m]>
its not about troughput either
<HebaruSan[m]>
`DownloadEvent`?
<DasSkelett[m]>
Don't have it yet <span class="d-mention d-user">HebaruSan</span>
<VITAS[m]>
i want as much codes others maintain for us as possible
<HebaruSan[m]>
I bet the DownloadEvent table is a high percentage of prod's total disk space
<VITAS[m]>
id wish we would use established processes and tools
<DasSkelett[m]>
for loops are established, I'd say.
<VITAS[m]>
sigh
<VITAS[m]>
if you insist
<HebaruSan[m]>
As is sqlalchemy
<HebaruSan[m]>
It's not like he's composing SQL commands with a bash script
<HebaruSan[m]>
I kind of wonder if it would make sense to create that list with a generator comprehension expression thing
<DasSkelett[m]>
Again, if you have a certain tool you want me to use <span class="d-mention d-user">VITAS</span>, tell me, but I don't want to search through Pypi for hours figuring out what each module does, how it works and if it's usable in our case and maintained. And trying it out, realizing it doesn't work as expected, ditching it, trying another one...
<HebaruSan[m]>
`users = (User(...) for i in range(user_count))`
<HebaruSan[m]>
Then we'd have the option of batching them in smaller groups later if we want to
<HebaruSan[m]>
Could probably copy/paste Leon's sqs queue helper function
<VITAS[m]>
there are 3 seeders on the page. i dont know the one fitting your needs
<VITAS[m]>
was a long day im tired
<VITAS[m]>
night
<HebaruSan[m]>
Good night!
pizzaoverhead_ has quit [Quit: Leaving]
VictorMout[m] has quit [Quit: Client limit exceeded: 6]