<HebaruSan[m]>
<span class="d-mention d-user">DasSkelett</span> with the `./start-server.sh` script, is a default admin user created? I think it worked but I'm not sure how to get in.
<HebaruSan[m]>
~~<span class="d-mention d-user">DasSkelett</span> with the `./start-server.sh` script, is a default admin user created? I think it worked but I'm not sure how to get in.~~
<HebaruSan[m]>
Found it!
<Darklight[m]>
``` File "/var/www/virtual/spacedock.info/htdocs/SpaceDock/alembic/versions/2020_06_23_17_49_36-85be165bc5dc.py", line 17, in <module>
<Darklight[m]>
from KerbalStuff.celery import calculate_mod_scores
<Darklight[m]>
ModuleNotFoundError: No module named 'KerbalStuff'```
<Darklight[m]>
Another one 😛
<Darklight[m]>
I was going to downgrade to master's db so we had a clear upgrade path
<Darklight[m]>
So that I could simulate a prod update
<Darklight[m]>
That's a bit more of a confusing one, maybe alembic fucks with the cwd
HebaruSan[m] has quit [Quit: Idle timeout reached: 10800s]
Darklight[m] has quit [Quit: Idle timeout reached: 10800s]
DasSkelett[m] has joined #spacedock
<DasSkelett[m]>
Whoops, the import isn't even used. However I have no idea why this comes up for you, shouldn't it also trhow on my system?
<DasSkelett[m]>
`from KerbalStuff.search import get_mod_score` is used though, so that'll probably fail again for you.
<DasSkelett[m]>
It also didn't came up on alpha. Did you run alembic from a different directory, not the repo root, or something?
godarklight[m] has joined #spacedock
<godarklight[m]>
Ran it where it should, the spacedock folder
<godarklight[m]>
I'll test again
micha_uk has joined #spacedock
DasSkelett[m] has quit [Quit: Client limit exceeded: 6]
godarklight[m] has quit [Quit: Client limit exceeded: 6]
micha_uk has quit [Read error: Connection reset by peer]
RockyTV[m] has joined #spacedock
<RockyTV[m]>
#289 <span class="d-mention d-user">HebaruSan</span> have you tried removing the double forward slashes in the download route?
<HebaruSan[m]>
Yeah it's not the best, the routes could have been better designed
<HebaruSan[m]>
But I think we can make them work
<RockyTV[m]>
is there a way to scrape through all of ckan's metadata files and remove these urls? or redirect them idk
<HebaruSan[m]>
It is possible to update old metadata, but it would be very likely to cause problems for users depending on the timing of how it was done
<HebaruSan[m]>
Either they would have the new metadata before SpaceDock updated, or they'd have the old metadata after SpaceDock updated, either way 404s galore
DasSkelett[m] has joined #spacedock
<DasSkelett[m]>
Also hundreds of download URLs posted in forum threads.
<HebaruSan[m]>
I think folks usually use the main mod info page, but possibly also that yes
<RockyTV[m]>
hm, the other option would be to check if mod_name is download and call the method for requesting the mod version
<HebaruSan[m]>
Then `/mod/id/download` would do different things depending on which mod it was?
<RockyTV[m]>
something like `if mod_name is 'download': return download(mod_id, None, None)`
<RockyTV[m]>
and yeah, the important parameter there is mod_id, so if you have `/mod/10/download` it'd redirect to `/mod/10//download`, if you change the id it changes which mod you're referring to
<HebaruSan[m]>
I don't think we should do that
<HebaruSan[m]>
`/mod/id/download` should be consistent for all mods
<RockyTV[m]>
what do you mean?
<HebaruSan[m]>
The same thing I've been saying about the possibility of a mod named 'download'
<HebaruSan[m]>
It makes it impossible for that format to work alongside existing URLs
<RockyTV[m]>
uhm I can't see how it would fail if we have that if logic I posted above on the `mod` method
<HebaruSan[m]>
OK, more details then. Suppose mod 1 is named 'hello' and mod 2 is named 'download'.
<HebaruSan[m]>
`/mod/1/download`: Downloads the mod
<HebaruSan[m]>
`/mod/2/download`: Loads the mod info HTML page, can't generate a generic download link
<RockyTV[m]>
> can't generate a generic download link
<RockyTV[m]>
why not?
<HebaruSan[m]>
BECAUSE IT WOULD LOAD THE MOD INFO HTML PAGE!
<RockyTV[m]>
you can access a mod page via `/mod/2/`
<HebaruSan[m]>
Yes, you can, but you can also include the name in the URL as of forever
<HebaruSan[m]>
That should not break
<RockyTV[m]>
what we'd need to do is remove all usage of `/mod/<id>/<name>` on the frontend
<RockyTV[m]>
and on the API
<RockyTV[m]>
and reserve keywords like latest and download
<RockyTV[m]>
and version
<HebaruSan[m]>
And then migrate CKAN, and explain to users why all their bookmarks broke for months and months of support hell
<HebaruSan[m]>
Just not worth it for this one tiny thing
<RockyTV[m]>
well it's not worth if we're doing everything at the same time
<RockyTV[m]>
we could update the API to not return the path to the mod info with name on it (same for the frontend), then fix old occurrences
<RockyTV[m]>
and add a warning when you access via mod_name that the url is being deprecated
<RockyTV[m]>
also, if you're worried about having a mod named download, I'd ask you to run a query on prod to check for mods and versions with "reserved" keywords
<RockyTV[m]>
we have 4 occurrences of returning a url with mod name in it
<RockyTV[m]>
on the API, that is
<RockyTV[m]>
also,
<RockyTV[m]>
> /mod/2/download: Loads the mod info HTML page, can't generate a generic download link
<RockyTV[m]>
well it would work anyways, you just wouldn't have an info page for a mod named download (and we shouldn't have a mod named download)
<RockyTV[m]>
you can literally pass anything you want to the mod_name parameter and the page still renders
DasSkelett[m] has quit [Quit: Client limit exceeded: 6]
Darklight[m] has joined #spacedock
<Darklight[m]>
Don't change the API
<Darklight[m]>
There is no reason to break things
<Darklight[m]>
It's not just ckan, it would also be everyones links
<RockyTV[m]>
the api won't change, the idea is to pass None to the mod_name in mod urls
HebaruSan[m] has quit [Quit: Client limit exceeded: 6]
DasSkelett[m] has joined #spacedock
<DasSkelett[m]>
I'm also for not breaking any existing URLs.
<DasSkelett[m]>
Maybe we can find a different way to avoid the double //
HebaruSan[m] has joined #spacedock
<HebaruSan[m]>
Well in practice the way would be to put the name in between them, that's what the PR does for the Download button.
<HebaruSan[m]>
The // is just for convenience in case you want a shorter URL
<Darklight[m]>
I am pretty sure you can put anything there though and it doesn't matter
<DasSkelett[m]>
Ah, I see. Well that doesn't sound like a problem then.
RockyTV[m] has quit [Quit: Client limit exceeded: 6]
VITAS[m] has joined #spacedock
<VITAS[m]>
just keep in mind that existing links to mods on the interwebs need to work
<Darklight[m]>
Yeah I think that was settled already 😛
<Darklight[m]>
Yeah I think that was settled already 😛
<Darklight[m]>
Also, alpha and prod are kinda diverging a long way
<VITAS[m]>
k i havent read all the things here :)
<VITAS[m]>
in terms of link syntax or code?
<Darklight[m]>
I mean in terms of there's lots of work being done by dasskellet/hebarusan that hasn't been deployed, you two are going to have to pick a time when you're happy with things
<VITAS[m]>
i know they do a lot of work.
<VITAS[m]>
i would like to fix multigame as soon as possible to support stuff like balsa
<VITAS[m]>
the next planed update is next month
<HebaruSan[m]>
Documenting bugs with multi-game would allow us to start banging out the fixes
<Darklight[m]>
I should probably fix some small annoyances, like the admin panel publish thing
<VITAS[m]>
they ( DasSkelett and HebaruSan ) should feel good about the code
<VITAS[m]>
ill prvide everything you guys need to aid in that
<HebaruSan[m]>
I'm happy with what's in beta at the moment in terms of functionality and stability
<Darklight[m]>
Holy fucking hell your github profile hebaru
<HebaruSan[m]>
alpha has those speed-up changes which potentially would be great to have in prod, but they do come with inherent risk
<Darklight[m]>
It's like solid green 😛
<VITAS[m]>
we can deploy in stages
<VITAS[m]>
not everything has to be deployed in one go
DasSkelett[m] has quit [Quit: Client limit exceeded: 6]
<VITAS[m]>
you pick what we deploy
<Darklight[m]>
I think the idea is that alpha is treated as future-prod, so it does kinda have to be 😛
<Darklight[m]>
Unless you want to get into a mess with cherry picking
HebaruSan[m] has quit [Quit: Client limit exceeded: 6]
<Darklight[m]>
What was the speedup though, like 100-1000x? 😛
<VITAS[m]>
the alpha beta thing was a offer stemming from what i know about software dev
<VITAS[m]>
i dont midn changing things
<VITAS[m]>
all i care about is that the code we put on prod is solid
<Darklight[m]>
I don't think beta has much use atm, but I really should write the integration thing where I have an automated "does this deploy from master?" check so we can highlight problems
<Darklight[m]>
Along with the AB tests
HebaruSan[m] has joined #spacedock
<HebaruSan[m]>
Actually that alembic issue has made me think beta does have a purpose, as we would have found it there eventually
<HebaruSan[m]>
alpha for continuous testing of stuff, beta for testing of upgrades in big chunks
<Darklight[m]>
Ah yeah that works
HebaruSan[m] has quit [Client Quit]
<VITAS[m]>
it is sad that that we didnt find any beta testers so far
<VITAS[m]>
i agree that this makes it less usefull
Darklight[m] has quit [Quit: Client limit exceeded: 6]
HebaruSan[m] has joined #spacedock
<HebaruSan[m]>
Real testing is a tough thing to motivate in a free project
<VITAS[m]>
yes i know
<HebaruSan[m]>
The best casual testing I've seen in CKAN is when someone reports an issue and I send them a test build to see if it's fixed
<HebaruSan[m]>
Obv not really an option for SD
<VITAS[m]>
you just tell me how you want to structure things and i play the sort of prod gatekeeper :)
<HebaruSan[m]>
Well since I've been the only person migrating changes from alpha to beta so far, for what it's worth, I've privately decided on a soft/casual "feature freeze" for beta.
<HebaruSan[m]>
I want to see the current beta live in prod before moving more stuff there.
<HebaruSan[m]>
If some big problems are found, then of course we fix them, but currently I know of none.
<VITAS[m]>
i think its a good idea youre the person directing commits
<VITAS[m]>
it was the idea from the beginning wasnt it?
<HebaruSan[m]>
I wasn't sure if it was supposed to be you or me or anyone, but this is fine
HebaruSan[m] has quit [Quit: Client limit exceeded: 6]
<VITAS[m]>
i think you know better than me what is what atm
<VITAS[m]>
i know the server side
<VITAS[m]>
but not the commits you guys make
<VITAS[m]>
or more like i dont know them like you do
HebaruSan[m] has joined #spacedock
<HebaruSan[m]>
Fair enough, I've been trying to keep it transparent so it's easy for you to catch up if/when you want to
<VITAS[m]>
i do
DasSkelett[m] has joined #spacedock
<DasSkelett[m]>
Yeah I'm also in a sort of feature freeze phase, that's why I didn't continue work on #274 yet, or started another bigger thing.
<DasSkelett[m]>
It would be neat to have a more exact date on the production upgrade to be able to better organize what to work on though.
<VITAS[m]>
but ive the luxury problem of you guys doing so much work all the time :)
<HebaruSan[m]>
Well I think it's fine to go full steam ahead on alpha still
<HebaruSan[m]>
Well I think it's fine to go full steam ahead on alpha still (to <span class="d-mention d-user">DasSkelett</span> )
<VITAS[m]>
ill have to do that ansible stuff on the new prod thing.
Darklight[m] has joined #spacedock
<Darklight[m]>
I think I will write some deployment tests just so I can flag stuff and we can catch them on the git straight away
DasSkelett[m] has quit [Client Quit]
<VITAS[m]>
good idea
HebaruSan[m] has quit [Client Quit]
DasSkelett[m] has joined #spacedock
<DasSkelett[m]>
Well the problem is, if we find a issue on alpha/beta about to be deployed to prod., we somehow have to merge it to beta without merging other features that aren't supposed to go to production in this round. That would mean messy cherry-picking.
<Darklight[m]>
I think you're going to have to treat alpha as future-prod and then settle on a "I think this release is good" commit
<Darklight[m]>
Because cherrypicking is unmaintainable I think
<DasSkelett[m]>
But how to settle on a "this release is good" commit when we don't know _when_ we will deploy to production?
<DasSkelett[m]>
Basically I'd say we _are_ in a "this release is good" state, should we stop development until late July?
<Darklight[m]>
I think herb already has the answer, merge beta when you think it's good
HebaruSan[m] has joined #spacedock
<HebaruSan[m]>
And my method of cutting the baby in half says: Yes, we stop, on beta
<Darklight[m]>
It's always possible to do hotfixes on beta if something comes back and merge it back into alpha
<DasSkelett[m]>
Well then let's hope we don't find any bug where we would have to somehow merge a fix to beta, without cherry-picking and without merging anything else to beta.
<Darklight[m]>
*edit:* ~~It's always possible to do hotfixes on beta if something comes back and merge it back into alpha~~ -> It's always possible to do hotfixes on beta if something comes up and merge it back into alpha
<DasSkelett[m]>
Although I admit that this is a imaginary problem as of now, we can sort that out if it ever happens.
<Darklight[m]>
Yeah 😛
<Darklight[m]>
Heh. Trying to get a master-state database and it's not downgradeable, I think I have a master-state db backup though because I thought this would happen 😉
<VITAS[m]>
just one rule: no prod data in testing env
<VITAS[m]>
alpha and beta are testing
<Darklight[m]>
I have my own database with 10k testing mods
<VITAS[m]>
thats ok if you seed the db
<DasSkelett[m]>
If you last upgraded your database in a not-yet-merged branch, you also have to downgrade it on this branch, so alembic can access the upgrade/downgrade script.
<Darklight[m]>
I'm aware of that one, but it fails to drop an unnamed foreign key I think
<VITAS[m]>
does alimbic or whatever the name of the db lib is support seeding?
<DasSkelett[m]>
Hmm, I'll test if it works on my system.
<Darklight[m]>
VITAS: I call the mod creation endpoints to see
<Darklight[m]>
VITAS: I call the mod creation endpoints to seed
<Darklight[m]>
``qlalchemy.exc.CompileError: Can't emit DROP CONSTRAINT for constraint ForeignKeyConstraint(<sqlalchemy.sql.base.ColumnCollection object at 0x7f3069f7e5e0>, None, table=Table('mod', MetaData(bind=None), schema=None)); it has no name``
HebaruSan[m] has quit [Quit: Client limit exceeded: 6]
<DasSkelett[m]>
alembic is the tool for db migrations, sqlalchemy is the database<->python framework
<Darklight[m]>
Going from 7993492be4eb to 77f7...
<Darklight[m]>
Downgrade isn't that important - it's mainly there to back out of updates, but that's kinda what the db backup is also for
<Darklight[m]>
But it would be nice to have working 😛
<DasSkelett[m]>
Sure, I always try up- and downgrading several times when adding a new migration.
VITAS[m] has quit [Quit: Client limit exceeded: 6]
HebaruSan[m] has joined #spacedock
<HebaruSan[m]>
inb4 VITAS makes a ahpla server for downgration testing
<Darklight[m]>
Haha I'm going to run something I think, I have done a github integration "checks" bot before but that was YEARS ago and I completely forgot what it did
<Darklight[m]>
I think it was only a proposal for ckan metadata checking to find when ckan didn't put files in gamedata
<HebaruSan[m]>
We just did two PRs for that stuff, if you can write a script we can probably get the rest working
DasSkelett[m] has quit [Quit: Client limit exceeded: 6]
<HebaruSan[m]>
The new hotness is "GitHub Actions" or "Workflows"
Darklight[m] has quit [Quit: Client limit exceeded: 6]
DasSkelett[m] has joined #spacedock
<DasSkelett[m]>
okay, I can reproduce the downgrade problem <span class="d-mention d-user">Darklight</span>
<HebaruSan[m]>
Looks like it's the mod locking change?
HebaruSan[m] has quit [Quit: Client limit exceeded: 6]
Darklight[m] has joined #spacedock
<Darklight[m]>
Alembic documentation has this: "Anonymously named constraints. Give your constraints a name, e.g. UniqueConstraint('col1', 'col2', name="my_name"). See the section The Importance of Naming Constraints for background on how to configure automatic naming schemes for constraints." but I am not sure if that is related or not
<Darklight[m]>
I don't follow database terminology well
<Darklight[m]>
Under things it doesn't support automatically
DasSkelett[m] has quit [Quit: Client limit exceeded: 6]
HebaruSan[m] has joined #spacedock
<HebaruSan[m]>
Kind of sounds like we can call it whatever we like, as long as we use the same name in upgrade and downgrade?
<HebaruSan[m]>
BTW <span class="d-mention d-user">Darklight</span> , example GitHub Action Workflows:
<Darklight[m]>
I think I'd want to host on my machine until we're comfortable it gives some type of usable results, I can get really specific self-hosted, and then if we decide it's something usable, move it to 52k, ab tests in the vm would probably be a bit odd
<Darklight[m]>
I think I want to be simple and just use the checks API
<DasSkelett[m]>
I for one have yet to discover anything "broken" with GitHub hosted runners, although I never did performance checks/benchamrks with them.
<DasSkelett[m]>
So for the database downgrade problem, it looks like we have to patch the migration script to make it downgrade-safe. Already upgraded databases (alpha, beta, local dev envs) wouldn't get the fix, which is probably not a big problem though.
<DasSkelett[m]>
On my local database the constraint seems to have been named `mod_locked_by_id_fkey` by postgres. I'm trying to find how it's named on alpha/beta.
VITAS[m] has quit [Quit: Client limit exceeded: 6]
<DasSkelett[m]>
Filling the name in for `None` does indeed seem to make the downgrade work.
HebaruSan[m] has quit [Quit: Client limit exceeded: 6]
<DasSkelett[m]>
So, try patching the script? Best case it makes the downgrade work if we ever have to use it, worst case it still won't work.
Darklight[m] has quit [Quit: Client limit exceeded: 6]
HebaruSan[m] has joined #spacedock
<HebaruSan[m]>
Sure, might as well
DasSkelett[m] has quit [Quit: Client limit exceeded: 6]
HebaruSan[m] has quit [Quit: Client limit exceeded: 6]
Darklight[m] has joined #spacedock
<Darklight[m]>
I'm gonna set up a clean vm on my server so it can run all the time, I'll keep a "clean state" db that stays on current master
<Darklight[m]>
But I will test downgrade "for the lulz" just incase we need to back out
Darklight[m] has quit [Client Quit]
HebaruSan[m] has joined #spacedock
<HebaruSan[m]>
I am looking for a way to make mypy throw an error if we pass None to the constraint functions, looks like it might be possible, will post updates on that PR