<HebaruSan>
sqlalchemy.exc.OperationalError: (psycopg2.OperationalError) FATAL: password authentication failed for user "spacedock_stageing"
<HebaruSan>
FATAL: password authentication failed for user "spacedock_stageing"
Xinayder has quit [Remote host closed the connection]
RockyTV has joined #spacedock
<VITAS>
HebaruSan, why arent you writing in irc? i read that too :D
<VITAS>
problem might be that (as stated before) the db is empty
<VITAS>
and it might max out psql max connections (for the db server)
<VITAS>
my advice
<VITAS>
only run one iunstance of gunicorn because its only for testing not for many people hammering it
<VITAS>
i check sd6 now
<VITAS>
oh im seeing your err msg yes
<VITAS>
ill check that too
<VITAS>
but the above still is a risk
<VITAS>
(note to self see if max db con can be raised)
<VITAS>
your key allready is in sd6
<VITAS>
have you useds the right host and port?
<VITAS>
we changed the port to 21352 because of reasons
<VITAS>
(have i and micha forgot to tell you?)
<VITAS>
did i give you acc to the db admin site?
<VITAS>
leaving gunicorn running against a wall and leaving isnt good
<VITAS>
sd2 was nearly frozen because of 100% and memory usage
<VITAS>
triplechecked and reset the db passwords
<VITAS>
sd23 is running but needs games and mail foo
<VITAS>
redirect is acting up when trying to login
<VITAS>
goes to spacedock.info not beta
<HebaruSan>
What the hell does gunicorn do that takes so much CPU and RAM? I was just trying to bring up a service. Not used to things that run away like that
<HebaruSan>
That redirect behavior is expected because the hard coding is still there. I need to get into sd6 to start removing it.
<HebaruSan>
This is the exact ssh config I'm using for that:
<HebaruSan>
Host sd-alpha
<HebaruSan>
HostName sd6.srv.52k.de
<HebaruSan>
Port 21352
<HebaruSan>
User HebaruSan
<HebaruSan>
IdentityFile ~/.ssh/id_rsa
<HebaruSan>
ServerAliveInterval 60
<HebaruSan>
RequestTTY force
<HebaruSan>
RemoteCommand /bin/bash -l
<HebaruSan>
$ ssh sd-alpha
<HebaruSan>
HebaruSan@sd6.srv.52k.de's password:
<HebaruSan>
Did you run a diff or just note that the file was there? Maybe it got truncated like Micha's on sd2.
<HebaruSan>
And no, I don't have db admin. I think that's me caught up.
<VITAS>
Yes youre right hardcoding :D
<VITAS>
dont know i was wondering why i couldnt hardly use ssh on sd2
<VITAS>
i just killed it and restarted it
<VITAS>
i then adjusted start.sh to only run one process (systemd might be better) and ran that
<VITAS>
i loged in and checked the file if it was visualy the same as the one on sd2
<HebaruSan>
debug1: Connecting to sd6.52k [10.150.1.52] port 22.
<HebaruSan>
debug1: connect to address 10.150.1.52 port 22: Connection refused
<HebaruSan>
ssh: connect to host sd6.52k port 22: Connection refused
<VITAS>
22
<VITAS>
wrong port
<HebaruSan>
Ahh right
<VITAS>
no message==no problems? :D
<HebaruSan>
No message == investigating the problems :P
<VITAS>
:(
<VITAS>
i belive in you Detective HebaruSan
<HebaruSan>
How did you copy it to sd2 in the first place? I'm concerned that I'm going to mess up ownership and symlinks
<VITAS>
sftp
<VITAS>
using midnightcommander
<VITAS>
i just chown -R everything
<VITAS>
no fucks given :)
<HebaruSan>
I opted to perform the transfer using tar and giving a few more fucks
<HebaruSan>
Seemed to work
<VITAS>
cool :)
<VITAS>
if youre like me and dont know how to do spacedock with systemd ask darklight he did that on production
<HebaruSan>
Oops, tried to start spacedock.target on sd2, it's gone unresponsive again
<HebaruSan>
Help?
<VITAS>
ok
<VITAS>
i usualy start start.sh
<VITAS>
but im oldschool
<VITAS>
round 2...fight :)
<VITAS>
(sd2 rebooted)
<HebaruSan>
Thanks
<VITAS>
if i take the time to set it up right i COULD give you acc to the start stop controlls
<HebaruSan>
Why does gunicorn use 100% CPU?
<HebaruSan>
What is it actually trying to od?
<HebaruSan>
*do
<VITAS>
dont know for me it didnt
<VITAS>
is what i use -> /var/www/virtual/spacedock.info/htdocs/start.sh
<HebaruSan>
Yeah, looking at that now. It's mostly the same as the systemd stuff.
<VITAS>
you may have noticed that i only let one instance run and commented out the others ins tart as well as apaches conf
<VITAS>
you could start by running one instance in foreground and then one in screen
<VITAS>
all i know is that it complains that it runs out of memory
<VITAS>
i dont know if its before or after it hits its head on the wall
<VITAS>
i can increase memory if needed (only gave the machine 512mb i think)
<VITAS>
(its a container so it will need less)
<HebaruSan>
Nah let's keep it small so runaway stuff like this fails quickly
<VITAS>
and we dont want to simulate large amounts of hits
<VITAS>
(i did that before moving to the current prod setup and were god up to 1000 page req/second
<VITAS>
)
<HebaruSan>
I don't understand. start.sh is still kicking off 8 or 10 gunicorn processes despite all but one being commented out.
<HebaruSan>
Although beta.spacedock.info is working again, so that's a plus.
<HebaruSan>
Oh, is it the --workers 6 argument? I guess that makes sense.
<VITAS>
i remember that only starting multiple instances helped allowing multiple requests to be processed in parallel
<HebaruSan>
I'm watching this all in 'top' to figure out wtf these programs do
<VITAS>
yes
<HebaruSan>
sd6's gunicorns start up for a few seconds and then disappear
<VITAS>
tried in foreground?
<VITAS>
and its set up like you did on sd2
<VITAS>
(virtualenv and so on)
<HebaruSan>
I copied sd2's apache conf file and changed it
<HebaruSan>
Heh, I managed to break start.sh by trying to spy on it
<HebaruSan>
The point of running it in 'screen' is so you can connect and view the output, right?
<HebaruSan>
Huh, alpha.spacedock.info is responsive now. I have no idea what changed; maybe it just needed a few minutes to put on its make-up?
<HebaruSan>
So yeah, start.sh seems to work where the systemd stuff does not. I'll make a note on the wiki because I want to do something else now.
<VITAS>
yes
<VITAS>
thank you for what you did so far :)
<HebaruSan>
My pleasure, I'm learning tons doing this
<VITAS>
yes its not every day that you are able to work on such a site and its not at work
<VITAS>
here i can experiment
<HebaruSan>
How do we contact darklight, the systemd stuff is a blocker for the GitHub auto restart trigger
<VITAS>
hes in room #dmp
<VITAS>
just ping him there
<HebaruSan>
Excellent
<VITAS>
I dont know how well veresd in Linux and stuff Micha is but he can do stuff too :)
<HebaruSan>
So stop.sh just sends a ctrl-C to the running screens, that's cute
<VITAS>
yes :)
<VITAS>
darklight always says: keep it simple
<VITAS>
thats why he rips everything out he doesnt understand :)
Daz has quit [Remote host closed the connection]
Daz has joined #spacedock
<HebaruSan>
OK, I whittled the gunicorns in systemd down to 1, and now it starts up without infinite CPU
<HebaruSan>
Is uwsgi in use? Because app.ini says "http-socket = :8009", which would conflict with one of the gunicorns if that's a TCP port
<VITAS>
we tried using that years ago but performance wasnt better
<VITAS>
the main problem kerbalstuff had was that they/he never was able to process more than one page request in parallel
<VITAS>
so his claimed dual socket xeon server for 150$/month was useless (and he didnt seem to know that)
<VITAS>
darklight and i played arroudn with various solutions and the only one working was to use the loadbalancer in apache webserver and multiple instances of gunicorn
<VITAS>
nowdays only 13% or so of all requests (including downloads) are handed to gunicorn at all the rest is done by ATS from various caches
<HebaruSan>
Static stuff, I assume
<VITAS>
not only
<VITAS>
mod files dynamic pages that dont change every call and so on
<HebaruSan>
If I do a "sudo reboot", will sd2 and/or sd6 come back up normally?
<VITAS>
they should
<HebaruSan>
OK, brace for impact, the systemd stuff is enabled to start at boot
<VITAS>
that also poses the question: with gunicorn/the python code not getting all requests (and some handled via apache webserver) are the mod download stats on the page correct?
<VITAS>
k
<HebaruSan>
I think it worked, so we shouldn't need to run start.sh anymore
<VITAS>
it shouldnt cause the pysical sys any harm even with 100% load
<VITAS>
wasent your plant to do something else now and stop tinkering with sd?
<VITAS>
:D
<VITAS>
(not that im sad or willing to stop you)
<HebaruSan>
Coincidentally I was looking at download stats recently. CKAN has been tracking them daily since September, and SD-hosted mods's stats looked similar to GitHub-hosted. If they were being cached, I would think there would be longer flatlines in the graph.
<HebaruSan>
Well CKAN only does that for ones with licenses that allow redistribution
<HebaruSan>
If you brought the whole database live it wouldn't be quite the same
<VITAS>
yes
<HebaruSan>
But yeah, it would be neat to see SD on there with 60% :)
<VITAS>
i see spacedock as kerbalstuff under new management not a new site
<VITAS>
i use the same codebase and do the same thing
<VITAS>
only the name changed and the person who hosts it :)
<VITAS>
i would find it cool to gain afoothold in other games i play
<VITAS>
but i dont have the time to be as present in those communities as im in ksp
<VITAS>
it sadisfies me to see cursforge in such small percentages
<HebaruSan>
Just as long as you don't start sending notifications to CKAN for Factorio mods
<VITAS>
the mods on archive must be quite old now. for what timespan did you make this chart?
<VITAS>
i wont unless you want to support that too :D
<HebaruSan>
Oh the archive is still updated in real time by the bot
<VITAS>
but you have to manualy determine the mods ?
<HebaruSan>
No, we have metadata for all of their licenses
<HebaruSan>
So if they're ARR, they don't get archived, if they're MIT, they do
<VITAS>
i wonder if we could do the archiving for all mods on spacedock whos licenses allow it without the need for them to be on ckan
<VITAS>
as another form of backup
<VITAS>
people have to select a license on SD so we can have a whitelist
* VITAS
is awayre that the longer spacedock exists the more of an museium and timecapsule it becomes
<VITAS>
-i
<HebaruSan>
But the license field is free text. We get a fair number of submissions where someone typed a sentence in there. Part of the CKAN indexing process is cleaning that up manually.
<VITAS>
thats why i do triple georedudant backups. but im open to more :)
<VITAS>
hmm righto
<VITAS>
so future version: make that a dropdown
<VITAS>
so many ideas so little time .... :)
<VITAS>
now its bed time for me
<HebaruSan>
Good night
<VITAS>
dont know your timezone but its 22:30 here
<HebaruSan>
Alpha/beta server emails aren't working. Do we care?