VITAS changed the topic of #spacedock to: Problems?: https://github.com/KSP-SpaceDock/SpaceDock/issues | Matrix/Riot Chat: https://im.52k.de +spacedock:52k.de Feel free to ask for help, we only bite a little bit! | If you want to help, please check https://github.com/KSP-SpaceDock/SpaceDock-Backend/issues/5 :) | <VITAS> inet users have the attentionspan of a squirrel...
DasSkelett[m] has quit [Quit: Client limit exceeded: 6]
VITAS[m] has quit [Quit: Client limit exceeded: 6]
HebaruSan[m] has quit [Quit: Idle timeout reached: 10800s]
VITAS[m] has joined #spacedock
<VITAS[m]> Rise and shine!
HebaruSan[m] has joined #spacedock
<HebaruSan[m]> Good morning
<HebaruSan[m]> The master branch is ready
<VITAS[m]> there he is
<VITAS[m]> k
HebaruSan[m] has quit [Quit: Client limit exceeded: 6]
DasSkelett[m] has joined #spacedock
<DasSkelett[m]> Morning! Way too early to do such stuff 😆
<VITAS[m]> yes thats because im doing it now
<VITAS[m]> way to early for most people to use SD
<VITAS[m]> also we have the rest of the day for fixing everything that breaks :P
HebaruSan[m] has joined #spacedock
<HebaruSan[m]> It's only 11 PM in California, prime gaming time
<HebaruSan[m]> Sorry, "23:00"
<DasSkelett[m]> Let's hope they've already installed all their mods.
<VITAS[m]> did i do it?
<VITAS[m]> any config changes i need to make?
<DasSkelett[m]> Yup, hang on...
<VITAS[m]> and https part?
<DasSkelett[m]> See here for the diff of the example config:
<DasSkelett[m]> Doesn't need anything new, we make use of the already existing `protocol` setting.
<DasSkelett[m]> And don't forget the alembic migrations. But they might take some time.
<VITAS[m]> hmpf i changed the names of the db users and now ansible is stuck
<VITAS[m]> sec ill get the hammer
<VITAS[m]> site is up so no rush
<VITAS[m]> whats ksp game id for a strange value
<VITAS[m]> i know what it does but why do we have it set in config?
<VITAS[m]> why not flag in db?
<DasSkelett[m]> IT's been initially added for the automated version import from CKAN, so it knows where to put those versions. It's also used for `/api/kspversions` now.
<VITAS[m]> if we can move it to the db in some way i would be happier
<VITAS[m]> the code shouldnt be that ksp specific if possible
<VITAS[m]> db migratring prod takes a long time
<HebaruSan[m]> Creating an issue for that would help us to remember to come back to it, if you have time
<VITAS[m]> yes could you do that?
<VITAS[m]> ill go back to updating prod
<HebaruSan[m]> Well I don't know your full reasoning, it would help me to understand if you wrote it out
<VITAS[m]> ok ill do it later where is the pr?
<HebaruSan[m]> I think you mean this one: https://github.com/KSP-SpaceDock/SpaceDock/pull/294
<VITAS[m]> and what monster of migration is this? :/
<VITAS[m]> thx
<HebaruSan[m]> The monster migration is probably this, extracting data out of DownloadEvent: https://github.com/KSP-SpaceDock/SpaceDock/pull/295
<VITAS[m]> there
<DasSkelett[m]> Already done?
<VITAS[m]> stillt he db migrating
<VITAS[m]> im getting nervous
<DasSkelett[m]> Ah okay
<DasSkelett[m]> Yep. We have to find the right DownloadEvents for each of the ~15k mod versions out of a table with >10 000 000 DownloadEvent rows.
<VITAS[m]> its doing stuff says db dash and cpu load
<DasSkelett[m]> It took me already 2 minutes on a test db with 100k mod versions but almost no DownloadEvent....
<HebaruSan[m]> On the bright side this migration is a first step away from needing DownloadEvent anymore
<VITAS[m]> thats a good point
<VITAS[m]> and for now my ansible script is working
<DasSkelett[m]> Yep. When we remove that table the database size will probably be halved or something.
<VITAS[m]> sd down -> me nervous :/
<VITAS[m]> comented ont he old and spun of in a new
<HebaruSan[m]> Excellent, now we won't forget
<HebaruSan[m]> Maybe this could be done along with PR 274 if it fleshes out the integrations system to support more than just CKAN
<VITAS[m]> https://spacedock.info/kerbal-space-program <- fallback page is working too
<VITAS[m]> i still hope you will support balsa
<VITAS[m]> :)
<VITAS[m]> would it be possible to add a read only mode to SD?
<VITAS[m]> so we could make the DB read only, copy it, migrate that copy and put it back when we update the code?
<VITAS[m]> so SD would be useable (partly) while beeing updated
<VITAS[m]> also PLEASE tell me beforehand if you know db migration wont be done in a few minutes :)
<DasSkelett[m]> I have told you around 4 times....
<VITAS[m]> you did?
<DasSkelett[m]> Yes, I did.
<VITAS[m]> hmpf
<VITAS[m]> so no excuse
<VITAS[m]> well
<VITAS[m]> nothing to do but wait
* HebaruSan[m] uploaded an image: unknown.png (15KB) < https://matrix.52k.de/_matrix/media/r0/download/52k.de/YNGguPLRHAmEHHEIzhAhhAae >
* HebaruSan[m] uploaded an image: unknown.png (29KB) < https://matrix.52k.de/_matrix/media/r0/download/52k.de/fBpcJOHTHfKDczKdrHWiENxZ >
<VITAS[m]> still going
<HebaruSan[m]> I don't suppose alembic has a progress bar?
<DasSkelett[m]> Ha, don't think so. But if you have query logging enabled, you might be able to spot which mod we're at right now.
<DasSkelett[m]> Actually, we are doing it by ModVersion, not Mod. If it automatically sorts by id, you could use that one as progress hint.
egg has quit [Ping timeout: 194 seconds]
<VITAS[m]> 1h and no end
egg has joined #spacedock
<VITAS[m]> now db usage is way down
<VITAS[m]> can i interrupt migration and restart iut?
<VITAS[m]> it
<DasSkelett[m]> Not safely I think
<VITAS[m]> check please
<VITAS[m]> also: why is there a crappy "download" text on the thumbnail mouse over?
<DasSkelett[m]> Crappy?
<VITAS[m]> i should pay more attentrin :/
<VITAS[m]> its overlaying into the mod name and its blue on blue
<DasSkelett[m]> I see, interesting, doesn't do on alpha. Did you already invalidate the cache of ATS?
<VITAS[m]> yes ansible does that
<VITAS[m]> flushes cache
<VITAS[m]> anyway is everything on prod like it should be?
<VITAS[m]> can i relay now? :)
<VITAS[m]> relax
<DasSkelett[m]> Are you sure? It still looks like it's serving the old static files.
<VITAS[m]> just did it manualy
<VITAS[m]> my guess compiler cache of flask
<VITAS[m]> the thing where it puts all the generated css and js files
<DasSkelett[m]> Whoops, I've told you the wrong game id earlier.
<DasSkelett[m]> It's `3102`, not `3201`. Could you change that in the config again, please?
<VITAS[m]> yes ofcause
<DasSkelett[m]> Thanks! So far everything else looks good, most importnat of all uploading and downloading works.
<VITAS[m]> yep but css seems broken
<VITAS[m]> but not in a "people instantly notice" kind of way
<VITAS[m]> time for breakfast
<DasSkelett[m]> Yeah, for example the bootstrap.css still says `last-modified: Sat, 03 Oct 2020 15:15:11 GMT`
<RockyTV> upgrade successful?
<DasSkelett[m]> So far, looks like it, aside from that CSS cache problem.
<DasSkelett[m]> Oh, looks like that one also fixed itself now.
HebaruSan[m] has quit [Quit: Client limit exceeded: 6]
* DasSkelett[m] uploaded an image: fccddadb-d011-42cd-83e7-6ee0b1c37d7d.png (58KB) < https://matrix.52k.de/_matrix/media/r0/download/52k.de/hompOGJCJNKlwgJEQhXrWZwc >
<DasSkelett[m]> Hm, 155 releases is a bit too much for the per-version download count graph
<VITAS[m]> please improve your testing :)
<DasSkelett[m]> Haha, have to spam some releases to a mod on alpha ^^
<VITAS[m]> im serious. we all have to do something about these problems
<DasSkelett[m]> Which problems?
<VITAS[m]> unforseen problems when /fater deployment
<VITAS[m]> after
<VITAS[m]> i have to read more what you write and do some sort of dry run before
<VITAS[m]> and you should figure out how to better emulate prod like settings
<DasSkelett[m]> Emulating production without production data is hard.
<VITAS[m]> wherent you seeding the db?
<DasSkelett[m]> Sure, but that only emulates the amount of production data, not the actual data, like inconsistencies and what not.
<VITAS[m]> btw ive to shutdown alpha for a minute
<VITAS[m]> even prod data wont show all edge cases
<VITAS[m]> all we can do is fill the db with a large amount of different stuff that maxes out fields and charset
<DasSkelett[m]> And the only problem I can see right now is that minor visual one with download stats, that affects maybe half a dozen mods on production or so.
<VITAS[m]> goal is to learn from every time we deploy stuff and get better at it
<VITAS[m]> so question is what can we as a team do to minimize downtime, bugs, hickups and all that jazz
<VITAS[m]> have time to do a bit of voip?
<DasSkelett[m]> Later, gonna eat some breakfast myself now.
<VITAS[m]> please do :)
<VITAS[m]> i should send you a care package some time like leveller and me are doing
<VITAS[m]> self made beefjerky and such
<VITAS[m]> (if youre into that)
<DasSkelett[m]> That sounds good. Actually, I might come to Hamburg soon.
<VITAS[m]> how come?
<DasSkelett[m]> My sister moved to Hamburg for university recently, and I want to visit her / her new home soon.
<VITAS[m]> cool whats she studying?
<DasSkelett[m]> "Irgendwas mit Medien" (Something with media) 😄
<VITAS[m]> mediasience?
<VITAS[m]> mediadesign?
<VITAS[m]> media...ahcrap :D
<VITAS[m]> just ping me when you want to voip
<VITAS[m]> ill do some server stuff till then
<DasSkelett[m]> Media acting and moderation of something like this it's called.
<DasSkelett[m]> Yep, I'll let you know.
<VITAS[m]> why is there an sql server running on alpha?
Vali[m] has joined #spacedock
<Vali[m]> Aaah Hamburg, great city. Got relatives there
<VITAS[m]> i got a burger there :P
<DasSkelett[m]> Hmm, can't find it to be a dependency of anything. Only a suggestion of `postgresql-client-12`. Maybe it was a recommendation back then and got installed that way?
<DasSkelett[m]> Anyways, got time to do some talking if you want <span class="d-mention d-user">VITAS</span>
<VITAS[m]> cool
<VITAS[m]> sec
Vali[m] has quit [Quit: Idle timeout reached: 10800s]
DasSkelett[m] has quit [Quit: Idle timeout reached: 10800s]
VITAS[m] has quit [Quit: Idle timeout reached: 10800s]
VITAS[m] has joined #spacedock
* VITAS[m] uploaded an image: image.png (1211KB) < https://matrix.52k.de/_matrix/media/r0/download/52k.de/jiTdboSOxvCGLlNPJoYcNQPe >
HebaruSan[m] has joined #spacedock
<HebaruSan[m]> Maybe we should give LGG a page that lists all his mods in a spreadsheet-like format and allows him to mass-update their compatibility
<HebaruSan[m]> Something linked off the user profile
<HebaruSan[m]> Just a giant table with the mod name in the left column and a dropdown in the right, then a save button at the bottom
<HebaruSan[m]> Just a giant table with the mod name in the left column and a game version dropdown in the right, then a save button at the bottom
<VITAS[m]> LGG?
<HebaruSan[m]> Mod author linuxgurugamer
<VITAS[m]> ah him
<VITAS[m]> had a long voip chat with DasSkelett
<VITAS[m]> we are sick of the frontend so we decided to push for a solt in replacement
<VITAS[m]> he can tell you more
<VITAS[m]> slot
<HebaruSan[m]> Ahh finally getting rid of the hacked up bootstrap
<HebaruSan[m]> Planning to keep the same/similar visual style?
<VITAS[m]> yes
<VITAS[m]> team effort
<HebaruSan[m]> Sounds good
<VITAS[m]> imagine things doing what we are expecting!
<VITAS[m]> :D
<HebaruSan[m]> Do you guys have an expectation of how long it will take?
<VITAS[m]> depends on who wants to help
<VITAS[m]> all peoplke need to know is css and html
<VITAS[m]> and be willing to follow guidelines
HebaruSan[m] has quit [Quit: Client limit exceeded: 6]
DasSkelett[m] has joined #spacedock
<DasSkelett[m]> Plan is to keep Bootstrap because that's what we know and it works and we keep the visual style this way. But Bootstrap 4 of course.
<DasSkelett[m]> Only features we already have right now, we can add new stuff later.
VITAS[m] has quit [Quit: Client limit exceeded: 6]
HebaruSan[m] has joined #spacedock
<HebaruSan[m]> And hopefully installing it with npm on demand rather than copying it into our gig
<HebaruSan[m]> And hopefully installing it with npm on demand rather than copying it into our git
VITAS[m] has joined #spacedock
<VITAS[m]> also admin lte as base template
<DasSkelett[m]> Yeah installing it with npm should be possible. Definitely agree with you there, this way we can absolutely make sure nobody will ever manipulate the Bootstrap files by hand.
<VITAS[m]> lets start at the beginning first :)
<HebaruSan[m]> Cool, so presumably there will be a git branch for this. What about a dedicated test server?
<HebaruSan[m]> I imagine this would be a pre-alpha staging area, so the code would flow `new-ui` -> `alpha` -> `beta` -> `master`
<VITAS[m]> Idea is to first do dummy pages for every view
<VITAS[m]> once they behave and look how they should we can make template files off them
<VITAS[m]> bonus: you just need text editor and browser for the stage
<VITAS[m]> so more people can pitch in
<HebaruSan[m]> Looks like making a fork of that AdminLTE project and committing to it probably isn't how it's intended to be used
<HebaruSan[m]> We should install it with npm like any other frontend dependency
<HebaruSan[m]> You're already 229 commits behind
<HebaruSan[m]> So <span class="d-mention d-user">VITAS</span> if <span class="d-mention d-user">DasSkelett</span> and I create a `new-ui` branch in git for this project, can you give us a dedicated test server for it, analogous to alpha and beta?
<RockyTV> why not use alpha?
<RockyTV> or you mean a server for the frontend only?
<HebaruSan[m]> Well if you think you can re-do the whole site in 2 weeks, alpha would make sense. I'm assuming it will take longer to get a full rewrite stable.
<HebaruSan[m]> In the meantime, we may need to do fixes on the existing site
<HebaruSan[m]> Hence keeping the existing dev pathway clear
DasSkelett[m] has quit [Quit: Client limit exceeded: 6]
HebaruSan[m] has quit [Quit: Idle timeout reached: 10800s]
VITAS[m] has quit [Quit: Idle timeout reached: 10800s]