<DasSkelett[m]>
<span class="d-mention d-user">VITAS</span> to summarize the discussion where we need you: HebaruSan proposed a patch to let the database "index" table columns that we frequently sort by. Darkllight tested the effects on a database with 10k mods, and measured what I'd say is roughly a ~25% decrease on response time on average. (Seehttps://github.com/KSP-SpaceDock/SpaceDock/pull/281 for the data).
<DasSkelett[m]>
However it would also increase the database size by arounrd 28%.
<DasSkelett[m]>
<span class="d-mention d-user">VITAS</span> to summarize the discussion where we need you: HebaruSan proposed a patch to let the database "index" table columns that we frequently sort by. Darkllight tested the effects on a database with 10k mods, and measured what I'd say is roughly a ~25% decrease on response time on average. (Seehttps://github.com/KSP-SpaceDock/SpaceDock/pull/281 for the data).
<DasSkelett[m]>
The question is, do you have any objection with this or is it okay for you?
<DasSkelett[m]>
However it would also increase the database size by arounrd 28%.
<DasSkelett[m]>
The question is, do you have any objection with this or is it okay for you?
<DasSkelett[m]>
<span class="d-mention d-user">VITAS</span> to summarize the discussion where we need you: HebaruSan proposed a patch to let the database "index" table columns that we frequently sort by. Darkllight tested the effects on a database with 10k mods, and measured what I'd say is roughly a ~25% decrease on response time on average. (Seehttps://github.com/KSP-SpaceDock/SpaceDock/pull/281 for the data).
<DasSkelett[m]>
However it would also increase the database size by arounrd 28%.
<DasSkelett[m]>
The question is, do you have any objections with this or is it okay for you?
VITAS[m] has joined #spacedock
<VITAS[m]>
size is no problem. storage and ram are plentyfull
<VITAS[m]>
im pro speed :)
<DasSkelett[m]>
Yeah thought so 😉
<VITAS[m]>
anything you need from me?
<DasSkelett[m]>
Just needed your Go for this one.
<DasSkelett[m]>
Though there's another thing I wanted to ask you:
<VITAS[m]>
Go
<VITAS[m]>
I hope we can make Balsa Mods a thing. I want to look for perspectives for SpaceDock to exist when T2 doesnt support ksp anymore and ksp2 wont (in my view most likely) not support unregulated mods.
HebaruSan[m] has joined #spacedock
<HebaruSan[m]>
Yeah we started a Balsa section on alpha and beta to test that
<DasSkelett[m]>
It is ^^ Just wanted to start at the beginning 😛
<DasSkelett[m]>
Or in other words, since you are already planning changes to the storage mounting with Darklight, it would be neat if you could somehow consider this.
<DasSkelett[m]>
Or in other words, since you are already planning changes to the storage mounting with Darklight, it would be neat if you could somehow consider this.
<HebaruSan[m]>
Rather than new config, what about putting `Game.short` in the path?
VITAS[m] has quit [Quit: Client limit exceeded: 6]
HebaruSan[m]1 has joined #spacedock
<HebaruSan[m]1>
Rather than new config, what about putting `Game.short` in the path?
<DasSkelett[m]>
Good idea. Didn't think much about the actual changes of the config yet, but yeah, sounds easier to maintain and would go without editing the config on prod. for new games.
<DasSkelett[m]>
Good idea. Didn't think much about the actual changes of the config yet, but yeah, sounds easier to maintain and would go without editing the config on prod. for new games.
HebaruSan[m] has quit [Quit: Client limit exceeded: 6]
VITAS[m] has joined #spacedock
<VITAS[m]>
from a dir structure view a subfolder for each game would be great
<DasSkelett[m]>
Maybe you also have to ban the <span class="d-mention d-user">matrix-appservice-discord-t2bot</span> on Discord? It's still here.
<HebaruSan[m]>
Dangit the part I wanted to screenshot scrolled while I was trying to clip it
<HebaruSan[m]>
Everything you're saying is still doubling up
<VITAS[m]>
test 123
<VITAS[m]>
good
<HebaruSan[m]>
lol
DasSkelett[m] has quit [Client Quit]
DasSkelett[m] has joined #spacedock
<DasSkelett[m]>
Maybe you also have to ban the <span class="d-mention d-user">matrix-appservice-discord-t2bot</span> on Discord? It's still here.
VITAS[m] has quit [Quit: Client limit exceeded: 6]
HebaruSan[m] has quit [Quit: Client limit exceeded: 6]
HebaruSan[m] has joined #spacedock
<HebaruSan[m]>
But yes, the code as it stands auto-creates all the dirs needed to store a mod, all we would have to do is update how it generates the path
DasSkelett[m] has quit [Client Quit]
VITAS[m] has joined #spacedock
<VITAS[m]>
k
DasSkelett[m] has joined #spacedock
<DasSkelett[m]>
To update the storage pathes in the database I'd do an alembic script. I think we could even move the actual folders in there, although I don't know if you want to do it there.
<VITAS[m]>
like store the path in the db?
<HebaruSan[m]>
It's already in `ModVersion.download_path`, just needs to be updated
DasSkelett[m] has quit [Client Quit]
<HebaruSan[m]>
<span class="d-mention d-user">DasSkelett</span> which of 277 or 281 should be merged first?
<HebaruSan[m]>
Or does it matter?
VITAS[m] has quit [Quit: Client limit exceeded: 6]
DasSkelett[m]1 has joined #spacedock
<DasSkelett[m]1>
Somewhat, the one that is merged second has to update the alembic script. SO I'd say do #281 first, then I'll rebase and change the alembic script of #277
<DasSkelett[m]1>
Somewhat, the one that is merged second has to update the alembic script. So I'd say do #281 first, then I'll rebase and change the alembic script of #277
<HebaruSan[m]>
OK, 281 is now merged
<HebaruSan[m]>
And we need to run the alembic stuff from the command line on alpha, right?
<DasSkelett[m]1>
Yes. `source bin/activate; python3 spacedock database migrate` should do it.
<HebaruSan[m]>
For both merges, or can we do them together?
<DasSkelett[m]1>
We should also be able to do them together, which is actually probably the better idea so we can check if that really works so we don't get problems when upgrading the prod db all at once.
HebaruSan[m] has quit [Quit: Client limit exceeded: 6]
DasSkelett[m]1 has quit [Quit: Client limit exceeded: 6]
HebaruSan[m] has joined #spacedock
<HebaruSan[m]>
OK, let's give these migrations a try!
DasSkelett[m] has joined #spacedock
<DasSkelett[m]>
Just tested locally on my system, went fine. Let's see how alpha does.
<HebaruSan[m]>
You can handle the restart stuff
<DasSkelett[m]>
Oh okay, I'd have let you do it 🙂
<DasSkelett[m]>
Done
HebaruSan[m] has quit [Quit: Client limit exceeded: 6]
<Darklight[m]>
I think that might be an issue given that KSP has such a loooong release cycle these days, I'd probably give it the full score if the version is compatible with the current KSP version, the idea is to derank old incompatible mods right?
<VITAS[m]>
you could even do complex stuff like: who with this os has downloaded other mods in this timespan
<Darklight[m]>
If it's to get BDA out of page 2, I'd probably want compatible versions to give full marks for the current version, but it may work out completely fine doing what you do 😉
<HebaruSan[m]>
Do we have a penalty for NOT being compatible with current?
<DasSkelett[m]>
And yes, there might be stats at stats.52k.de, but once again, I don't know how to access them, _what_ stats there are, how the API works... Neither do we have any framework at all to access it yet, whereas the database is here and works and I know how to request data from it.
<DasSkelett[m]>
<span class="d-mention d-user">VITAS</span> OSes recommendations are a different topic.
HebaruSan[m] has quit [Quit: Client limit exceeded: 6]
<VITAS[m]>
that was an example :)
<VITAS[m]>
im only saying that theres complex data back to when sd started
<VITAS[m]>
with api and web interface
<DasSkelett[m]>
Yes we do <span class="d-mention d-user">HebaruSan</span> , but it maxes out at -20 AFAIK, which is negligible with +154255 from the total downloads.
<DasSkelett[m]>
<span class="d-mention d-user">Darklight</span> giving compatible versions full score is a good idea 👍
<DasSkelett[m]>
*edit:* ~~<span class="d-mention d-user">VITAS</span> OSes recommendations are a different topic.~~ -> <span class="d-mention d-user">VITAS</span> recommendations are a different topic.
VITAS[m] has quit [Quit: Client limit exceeded: 6]
HebaruSan[m] has joined #spacedock
<HebaruSan[m]>
I don't think the score has a compatibility check currently
<HebaruSan[m]>
It has some age checks
<Darklight[m]>
Also, you possibly may want to clamp delta to a sane value, that's up to you though
<DasSkelett[m]>
Oh whoops, yeah that's only an age check.
<Darklight[m]>
May be unnessecary
<HebaruSan[m]>
OK, most obvious idea to me right now is, if incompatible with current version, apply 50% penalty
<HebaruSan[m]>
Too harsh? Not harsh enough?
<Darklight[m]>
I think that's fine
<Darklight[m]>
Because people only need to update and then their score comes back
<Darklight[m]>
I do know a lot of people that run 1.8.1 because of mods though, I see it all the time and the release cycle has gotten so long that I have a 1.8 backport for DMP when people ask
<HebaruSan[m]>
Next most obvious idea is, 10% penalty for each game version that you fall behind
<HebaruSan[m]>
So you keep 90% of your score if we're on 1.9.1 and your mod is for 1.8.1
<HebaruSan[m]>
But if we're on 1.9.1 and your mod is for 1.1, you lose 80%
<HebaruSan[m]>
I'm not sure if that's easily calculable in SD's data model tho
<Darklight[m]>
That would be pretty easy
<Darklight[m]>
For each game version come up with a multiplier
<Darklight[m]>
So long as you can sort GameVersion, and I am pretty sure the versioning is sanely sortable
<DasSkelett[m]>
Had a similar idea in mind at first <span class="d-mention d-user">HebaruSan</span> , but we don't have any version comparison, so we'd have to rely on something like sorting by `GameVersion.created`, which I'm not sure whether it's a good idea.
<Darklight[m]>
Then use verMultiplier when calculating scores
<DasSkelett[m]>
Yeah I'm more worried about the sortin of `GameVersion`.
<DasSkelett[m]>
Yeah I'm more worried about the sorting of `GameVersion`.
<Darklight[m]>
Oh you would also need to make it gameaware but that is trivial
<Darklight[m]>
We can write a sorter for gameversion, there isn't so many gameversions that I'd be concerned about it, but afaik they are created in order so you could probably leave them unsorted
<HebaruSan[m]>
So are we comfortable with 0 or negative scores for a mod 10+ versions behind?
<HebaruSan[m]>
Hmm, the negative side of things may require further thought. High download count becomes its own penalty at that point.
<DasSkelett[m]>
Urgh, `/api/kspversions` returns all GamVersions, not only KSP versions...
<DasSkelett[m]>
But looking at `/api/3102/versions`, it _does_ look like the versions are all ordered in a sane way. So we might be fine sorting by id, hoping that the unlikely event of backports will never happen in one of the games we support.
<HebaruSan[m]>
KSP is what I care about at this point 👍
<HebaruSan[m]>
So, BDArmory is 26 versions behind
Darklight[m] has quit [Quit: Client limit exceeded: 6]
<HebaruSan[m]>
How about a 3% penalty per version? BDA would drop to 33936
<HebaruSan[m]>
(At 4% it would go to 0)
DasSkelett[m] has quit [Quit: Client limit exceeded: 6]
HebaruSan[m] has quit [Quit: Client limit exceeded: 6]
DasSkelett[m] has joined #spacedock
<DasSkelett[m]>
It would be really cool to see the effect on the production db, but checking some download counts of mods, I think this would have the desired effect.
Darklight[m] has joined #spacedock
<Darklight[m]>
That approach is expensive but given that it is done in a celery thread periodically I think that's ok
<Darklight[m]>
Oh wait... would it be?
<Darklight[m]>
I'd have to test, it might be fine
<Darklight[m]>
It depends on how many thousand calls it makes to the db
DasSkelett[m] has quit [Quit: Client limit exceeded: 6]
HebaruSan[m] has joined #spacedock
<HebaruSan[m]>
Well the `all` list will have 79 elements
<HebaruSan[m]>
And compat is O(1)
<HebaruSan[m]>
And `compat` is O(1)
Darklight[m] has quit [Quit: Client limit exceeded: 6]
HebaruSan[m] has quit [Client Quit]
DasSkelett[m] has joined #spacedock
<DasSkelett[m]>
I think it's two database calls per mod. The Python-side of thing probably is quite fast.
<DasSkelett[m]>
Check out alpha first, run alembic upgrade head, then checkout HebaruSan's branch.
<Darklight[m]>
Kk
<Darklight[m]>
Crapping out with the same error, let me check a few things
<Darklight[m]>
Hrmm... working tree clean
<DasSkelett[m]>
With #287 `get_mod_score` accesses `Mod.game`, which wasn't the case when I wrote the upgrade script, so I didn't include it in the skeleton class.
<HebaruSan[m]>
Ahh, I'll include that with the packaging fix
<DasSkelett[m]>
But good thing you discovered this, this could threw us over during the production db upgrade too.
<Darklight[m]>
We would have caught it in alpha probably
<DasSkelett[m]>
Nope, alpha is already upgraded. Beta maybe.
<Darklight[m]>
I don't know if we have a big "alpha failed to deploy" warning
<Darklight[m]>
Alpha worked fine
<Darklight[m]>
Oh, you have skellet/alpha
<Darklight[m]>
That doesn't point at actual alpha
<DasSkelett[m]>
Well since the automatic deployment doesn't really work, and is only partially automated, that warning wouldn't be very useful.
<DasSkelett[m]>
What do you mean?
<Darklight[m]>
Nevermind it's all good, in any case for some reason, upgrading to yours worked, then upgrading to the score penalty worked
<Darklight[m]>
Which doesn't make a lot of sense to me
<DasSkelett[m]>
Hm, maybe I should've copied over `get_mod_score()` instead of importing it in the alembic script.
<DasSkelett[m]>
Would've been more "future proof"
<HebaruSan[m]>
Could we make the object classes importable?
<DasSkelett[m]>
That doesn't seem to work, sqlalchemy somehow doesn't recognize the imported modules.
<Darklight[m]>
Commented the speed thing, seems good to me
<DasSkelett[m]>
I just tried importing `Media` instead of implementing the skeleton in the score upgrade script, and get this: `AttributeError: 'Mod' object has no attribute 'media'`
<DasSkelett[m]>
So it doesn't populate the backref.
<DasSkelett[m]>
Going to push a few tweaks to the upgrade script to get it working
<HebaruSan[m]>
Sounds good
<HebaruSan[m]>
Curious to see what I missed, I thought I got all the properties
<DasSkelett[m]>
Not quite, but most of them.
<DasSkelett[m]>
Done. `Game.id` is needed because it's the primary key, `GameVersion.game_id` is needed to actually map the ids togethers (needed by `Realtionship` to work)
<HebaruSan[m]>
Oh! Yes, I did notice that about `id`, just forgot to come back and fill it in