VITAS[m] has quit [Quit: Client limit exceeded: 6]
HebaruSan[m] has joined #spacedock
<HebaruSan[m]>
How would we test whether fewer gunicorn instances is safe for prod?
<Darklight[m]>
Just do it
<Darklight[m]>
It originally had 1
<Darklight[m]>
Then we discovered the worker thread thingies and ran with that for a while
<Darklight[m]>
And then vitas bumped it up to like 10 instances which is why you see what you see in top
<HebaruSan[m]>
Alright
Darklight[m] has quit [Quit: Idle timeout reached: 10800s]
HebaruSan[m] has quit [Quit: Idle timeout reached: 10800s]
VITAS[m] has joined #spacedock
<VITAS[m]>
because the worker thingy didnt scale across cores well
<VITAS[m]>
now the webserver does the LB and it does scale
<VITAS[m]>
it was the major problem of KS in the olden days that it only ran one instance on one core
<VITAS[m]>
so one person could be served at a time
<VITAS[m]>
including mod dl and static content
VITAS[m] has quit [Quit: Idle timeout reached: 10800s]
VITAS[m] has joined #spacedock
<VITAS[m]>
just got word form the DC they want to do another maintence on the 23th between 0800 and 1600 for a few mins
<VITAS[m]>
i think its worth trying to get past that without shuting down stuff
KineticSloth8StoneBlue[m has joined #spacedock
<KineticSloth8StoneBlue[m>
I found a mod hosted on SD, that packages a dependency, and while the dependency has a CC BY-NC-SA 4.0, there is NO attribution given, NO copy of the CC 4.0 license, and the mod itself is listed under MIT
<KineticSloth8StoneBlue[m>
Should anything be done about it?
<VITAS[m]>
you could mail the author to please include one
<VITAS[m]>
if hes on the forums that is
<KineticSloth8StoneBlue[m>
yeah.. no link to forums included 😦
<VITAS[m]>
also the username isnt on the forums?
<VITAS[m]>
we could email that person
<KineticSloth8StoneBlue[m>
weird... looks like they might be Russian, or at least someone who uses cyrillic as native language... and weirder, files inside the mod, have what appear to be possibly Southeast Asian names..(Thai/Viet/Laotian/etc)
<VITAS[m]>
i dont want to act without beeing asked by those affected so if we cant reach the mod authors you could ping the dependency owner and as him if he wants to file a complaint
<VITAS[m]>
so both countries that are trying to scrape and copy sd
<VITAS[m]>
:D
<KineticSloth8StoneBlue[m>
Hmm.. i would attempt to email, but doesnt look like there is a way for SD users to do that between each other?
<VITAS[m]>
nope i want that
<VITAS[m]>
have a link to the mod?
<KineticSloth8StoneBlue[m>
I'll contact the dependency author
<KineticSloth8StoneBlue[m>
heres a link to the SD listing:
<VITAS[m]>
best is for him to come here or pm me on the forums
<VITAS[m]>
i will delete the mod if its a nogo for him but only then
<VITAS[m]>
also can confirm: russian uploaded
<VITAS[m]>
uploader
<KineticSloth8StoneBlue[m>
'k.. dependency author (flywlyx) PM about it 😉
<KineticSloth8StoneBlue[m>
'k.. dependency author (flywlyx) PM'd about it 😉
<VITAS[m]>
good :)
KineticSloth8StoneBlue[m has quit [Quit: Client limit exceeded: 6]
HebaruSan[m] has joined #spacedock
<HebaruSan[m]>
What's the dependency? I missed it when indexing on CKAN
<HebaruSan[m]>
Didn't recognize any of the folder names
VITAS[m] has quit [Quit: Client limit exceeded: 6]
DasSkelett[m] has joined #spacedock
<DasSkelett[m]>
I think they're speaking of GN Drive?
<HebaruSan[m]>
Ahh, likely
KineticSloth8StoneBlue[m] has joined #spacedock
<KineticSloth8StoneBlue[m]>
yeah exactly
<KineticSloth8StoneBlue[m]>
also, even tho they have the GN plugin packaged, they dont have many of the files from GN included... idk if that matters...other than possibly screwing up the GN mod itself if its installed seperately, also vOv
<HebaruSan[m]>
The other files look like they're for the parts that come with GN Drive
<KineticSloth8StoneBlue[m]>
IIRC, flywlx might be on Euro time vOv, and is usually on in evenings... so may not hear from them for a bit
<HebaruSan[m]>
This one has GN Drive bundled, too, in case you want to check it for license violations:
<HebaruSan[m]>
Hmm, actually the DLL went from 10 KB (GNDrive) to 4 KB (Droplet). Maybe the author made a new one.
Darklight[m] has joined #spacedock
<Darklight[m]>
Is this possible to just ignore until someone complains 😛
<HebaruSan[m]>
Well if we're already contacting the GNDrive author, I wouldn't want to sit on knowledge of other possible problems
<VITAS[m]>
i think if we know it we should give a hint
<VITAS[m]>
but i wont delete stuff without beeing asked by the copyright holder
KineticSloth8StoneBlue[m] has quit [Quit: Client limit exceeded: 6]
DasSkelett[m] has joined #spacedock
<DasSkelett[m]>
Sure, the copyright holder might have given their permission that go beyond the license for those cases, after all.
<RockyTV>
so I was messing around with vue and the spacedock api
<RockyTV>
and we might have a problem
<RockyTV>
/api/games doesn't return the gameshort
<VITAS[m]>
yes the gameshort :/
<VITAS[m]>
why not finish the adminlte bootstrap based theme?
<RockyTV>
and there's a 'link' field in Game object which I suppose should be the link to the game, or should it refer to the gameshort?
<VITAS[m]>
it could slot right in
<VITAS[m]>
and clean up the css hell
<VITAS[m]>
i cant remember :D
<VITAS[m]>
i think theres the name of the game and the part of the url
<RockyTV>
the Game object has "short" and "link", short is the url and link is generally empty
<RockyTV>
holy crap there's a bunch of games in spacedock db
<VITAS[m]>
so it might be the link to the game
<VITAS[m]>
yes
<VITAS[m]>
i wrote an importer at some point
<RockyTV>
link as in external link?
<VITAS[m]>
i dont know
<RockyTV>
I'll submit a PR to "fix" it, link will refer to game.short for now
<RockyTV>
actually I'll just remove the link and make it game_short
<VITAS[m]>
dont remove what you dont know
<VITAS[m]>
you can remove it if you have found proof what it was intended for :)
<DasSkelett[m]>
Why not extending `game_info()` to add `"short": game.short`?
<VITAS[m]>
that doesnt sound bat
<VITAS[m]>
im just cautioning everyone. we know that sd is layers of crap and removing seemingly unsused things can mess it up good :)
<VITAS[m]>
thats also why im for using a clean bootstrap theme approach because it would simplyfy the frontend code and lets us see underlying problems without having to modify to much (as we would do when using another css lib)
<VITAS[m]>
the solgan is: "make SD less anoying" :D
<DasSkelett[m]>
I tried replacing the Bootstrap CSS and js files with unmodified Bootstrap 3 files, it breaks a lot. Even worse if you try Bootstrap 4.
<RockyTV>
VITAS[m], for your happiness I'm using bootstrap and vue, I might use AdminLTE for inspiration, I think messing with existing templates is more complicated than making your own, specially when you're adapting it to a frontend stack
<VITAS[m]>
yes we would have to rewrite the whole html and css stuff
<VITAS[m]>
but using bootstrap would keep it similar
<RockyTV>
also I noticed that we have something bad in /browse api
<RockyTV>
it doesn't list per game
<VITAS[m]>
RockyTV (IRC): your free to use what you want and i will use it if its finished and working. im just suggesting that continueing something allready started and having a repo would be nice
<RockyTV>
I'll push my changes to a separate branch on OpenDock
<VITAS[m]>
you can do that
<VITAS[m]>
but ive a clear position on that opendock thing
<VITAS[m]>
and it hasnt moved in years too
<VITAS[m]>
-moved +progressed
<RockyTV>
yes, vue has changed a lot since we last tried
<VITAS[m]>
but one person reinventing large parts of SD didnt work out in the past
<RockyTV>
ugh ssh doesn't work in my cursed machine
<DasSkelett[m]>
If you fix something that could/should be fixed in SpaceDock now, consider keeping it in separate branches so you can easily make PRs for the main SpaceDock code.
<RockyTV>
should it target alpha or master?
<DasSkelett[m]>
Alpha
<RockyTV>
uh okay
<DasSkelett[m]>
Right now there's not too many differences.
<RockyTV>
/api/browse returns all mods on the site, is this something we want?
<DasSkelett[m]>
Why not?
<VITAS[m]>
if it contains pageing
<DasSkelett[m]>
Yep, it does
<VITAS[m]>
then why not
<DasSkelett[m]>
It also filters unpublished ones.
<VITAS[m]>
i would like to have some osrt of req limit
<RockyTV>
yes but if I want to filter by game, should the client do it (e.g. frontend) or should we make it easier for whoever consumes the API?
<VITAS[m]>
set per user
<VITAS[m]>
sort
<RockyTV>
huh
<RockyTV>
because I was thinking we could either add a ?game parameter to all /browse methods or just do /api/<gameshort>/browse or api/<gameshort>/mods to list all mods
<VITAS[m]>
per game would be good
<DasSkelett[m]>
Makes sense. We should be careful where we place the placeholder though, to not get us into a similar situation like with the latest-version-download. By keeping it as a request arg, we wouldn't have that issue. Hmmm
HebaruSan[m] has quit [Quit: Idle timeout reached: 10800s]
<DasSkelett[m]>
VITAS: wouldn't request limits be more the job of the Apache web server? Or do you only want to limit certain endpoints?
<DasSkelett[m]>
Do your (Piwik?) stats show if there are bad boys/bots that should be limited?
<VITAS[m]>
i was thinking about api keys
<RockyTV>
^
<DasSkelett[m]>
Ouch
<VITAS[m]>
i allready ip block some scrapers in the webserver
<RockyTV>
weird, I am now trying to run on alpha branch and gunicorn fails to load
<DasSkelett[m]>
Are they actually scraping the API, or mod zips?
<RockyTV>
I think it was the api, they were replicating spacedock
<VITAS[m]>
both
<VITAS[m]>
they want the metadata too
<VITAS[m]>
e.g. spacedock.ru
<VITAS[m]>
theres even a redirect for zip dl for them
<VITAS[m]>
so they get zips when dl from their ip ranges but all contain only dick pics
<DasSkelett[m]>
I see. Well, somebody got experience in API rate limiting while not screwing legitimate users that are using API through the frontend with XHR?
<VITAS[m]>
you can do x amount of req per key and day
<RockyTV>
do we have people that use the API? :P
<RockyTV>
or X amount of requests per minute/hour
<VITAS[m]>
x can be dapated per key to not hamper ckan
<DasSkelett[m]>
People using the frontend are also using the API in some places, XHR requests and such.
<RockyTV>
yes but they're originating from spacedock.info, no?
<DasSkelett[m]>
The requests?
<RockyTV>
yes
<VITAS[m]>
the target is always the client
<VITAS[m]>
ajax and such
<DasSkelett[m]>
Well no, it's JavaScript code running on the clients doing the requests.
<VITAS[m]>
if you do say 500 req per day i doubt a normal user would hit it
<VITAS[m]>
all depends on the amount of data one req would yield
<DasSkelett[m]>
Can't say anything on that, I have neither any usage data for SpaceDock, nor a number on how often scrapers hit.
<VITAS[m]>
i think a big question would be: can we decouple the api from the "normal" website?
<DasSkelett[m]>
Nooooo, please not
<VITAS[m]>
not as seperate parts of software
<VITAS[m]>
but webpages not doing data req trough the api
<VITAS[m]>
you can e.g. not serve json for the webpages but html with data embeded
<DasSkelett[m]>
How else, though?
<VITAS[m]>
that makes it anoying for scrapers
<VITAS[m]>
they have to adapt their stuff each time we change our html templates
<VITAS[m]>
and we can apply different limits on webreq/s and apireq/day
<RockyTV>
react + styled components makes it difficult as you don't have class names
Darklight[m] has quit [Quit: Idle timeout reached: 10800s]
<RockyTV>
well you do but they're randomized
<VITAS[m]>
webreq/s can be done by ip and apireq can be done by key
<VITAS[m]>
as im typing this i realize that given the state of the templates we are aeons away from implementing any of this :D
<RockyTV>
the question we have to ask: are we going to allow anonymous api access?
<VITAS[m]>
no you would have to reg an SD acc and autom get your key
<VITAS[m]>
in your settings
<DasSkelett[m]>
Okay, as someone who likes to know where we're at before discussing hypothetics, I've taken a look at the code, and I think the only places where we are calling the API to actually fetch data (instead of just using it to "toggle" something or similar), are:
<DasSkelett[m]>
* when creating a mod to get a game's available versions
<DasSkelett[m]>
* the typeahead when adding mods to modpacks
<VITAS[m]>
so i take it would be possible
<DasSkelett[m]>
In theory, yes. We could exempt the first one as well as all the "toggle" api endpoints from the limiting, since they aren't scrape-worthy. We'd have to think about the second one, which returns quite a lot of mod metadata, and we also use a lot of it afterwards.
<VITAS[m]>
maybe first new frontend and then api limits :)