Guest77 has quit [Remote host closed the connection]
Llamato_ has joined #spacedock
Llamato has quit [Ping timeout: 206 seconds]
aradapilot has quit [Ping timeout: 190 seconds]
VITAS[m] has joined #spacedock
<VITAS[m]>
the channel comes alive. Im happy :)
<DasSkelett>
I try to help where I can. And since I encountered the exact same problem a few days ago where your drive went to read-only and I wanted to see what the API returns, it was quite an easy tip to give for me.
AllisTauri[m] has joined #spacedock
<AllisTauri[m]>
tired to update alpha
<AllisTauri[m]>
found that `build-frontend.sh` relies, not surprisingly, on the ability to modify files in `/var/www/virtual/spacedock.info/htdocs/SpaceDock`
<AllisTauri[m]>
meaning that at least some parts of it should be user-owned by `www-data`
<AllisTauri[m]>
VITAS: what say you about this? Is there a reason why it was owned by `root`?
<AllisTauri[m]>
HebaruSa- (IRC): I'd say this is also the reason why the autodeploy didn't work. `www-data` user shouldn't have been able to make `git pull`
VITAS[m] has quit [Quit: Idle timeout reached: 10800s]
<AllisTauri[m]>
Ah, and if we add something to `requirements.txt` there's no way `www-data` could run `pip3 install`
<AllisTauri[m]>
Two points for docker here.
<AllisTauri[m]>
The workaround would be to install everything under SpaceDock with `pip3 install --target`, then set PYTHONPATH in systemd unit files...
<Lartza>
AllisTauri[m], venvvvvvvvvvv
<AllisTauri[m]>
Yea, run system daemon inside venv.
<Lartza>
?
<Lartza>
Just, run anything in venv if you need to install stuff into it
<Lartza>
Unless you are using distro packages, but even then from a reproducability standpoint a venv is probably cleaner
<AllisTauri[m]>
imho venv is a dev tool; but no matter -- currently the SpaceDock on alpha is in itself a venv, so I just use this
<Lartza>
You can't expect to find OS packages for project deps, and polluting your site-packages with pip is a huge no no
<Lartza>
And it has no real disadvantages in production either
<Lartza>
And if you ever run more than one python project on the same server you need venvs either way
<Lartza>
I mean assuming we don't use docker
<AllisTauri[m]>
For end-user apps I always prefer distribution packages with their dependencies hooked to the distro. And for server-side -- containerization in any form. Docker, virtual machines (of which lxc is a kind), vmware...
<Lartza>
But that has it's own gotchas and isn't an inherently BETTER solution to the problem, it's a DIFFERENT solution
<Lartza>
AllisTauri[m], But you can't realistically get distro packages of the version your app needs on every distro, that's the whole issue
<Lartza>
And I hate the abstraction of containers in a lot of cases, though it is one solution
<Lartza>
My server runs zero containers, and at least three venv's for different things
<Lartza>
Okay sometimes it runs one container, a Windows Server VM ;P
<AllisTauri[m]>
It's just different approaches: you make the app, then distro-maintainers made distro package; that's what distros are.
<AllisTauri[m]>
If they don't want your app, user can always install it using appropriate tools, handling dependencies as they see fit.
<AllisTauri[m]>
Packing everything in venv for distribution is also bloating, but not of site-packages, but of the filesystem
<Lartza>
You don't distribute a venv, you distribute a requirements.txt
<Lartza>
And it's not bloating site-packages that's an issue, it's that it's an OS folder pip should never touch
<Lartza>
Only apt/yum/pacman should
<AllisTauri[m]>
What is actually wrong with the SpaceDock alpha is that it runs on VM that id doesn't own. You shouldn't be bothered with permissions and "bloating" of "system" site-packages in a VM dedicated to a serice.
<AllisTauri[m]>
* What is actually wrong with the SpaceDock alpha is that it runs on VM that id doesn't own. You shouldn't be bothered with permissions and "bloating" of "system" site-packages in a VM dedicated to a service.
<Lartza>
minimal python, venv with requirements.txt dependenceis, no bloat ;)
<Lartza>
dependencies
<Lartza>
I don't actually really think docker solves all of the issues with python libraries, I still wouldn't pip install anything to a venv since that always runs as root and who knows what happens with the lib in between you making the Dockerfile and someone installing it in the future
<Lartza>
You still want a venv inside the container
<AllisTauri[m]>
You never want a venv in a container, because a container is not a Dockerfile, it's the actual system in which pip installed everything already. It's frozen in a much deeper way than venv.
<Lartza>
Yes you do
<Lartza>
There is no other proper way to install python libraries
<Lartza>
And have them frozen in a specific version without installing them as root
Webchat857 has joined #spacedock
Webchat857 has left #spacedock [#spacedock]
Bart[m] has joined #spacedock
<Bart[m]>
Couldn't you just run it in a conda environment?
<Bart[m]>
If you want to keep separate versions but at the same time don't want to run it as root.
<Bart[m]>
That's what I do to run meep through junyper, it runs like shit unless you get specific library versions (contrary to what the documentation claims).
<AllisTauri[m]>
<irc_Lartza "And have them frozen in a specif"> There's no issues with root in a container. Container is not a system; it's an environment for a single application. The application owns the container and all it's "system" things, including system pip and system site-packages.
<Lartza>
AllisTauri[m], Of course there are issues with root in a container
<AllisTauri[m]>
Bart: the argument here is actually between full virtualization (VMs, containers) versus version+environment tracking that venv and conda do.
<AllisTauri[m]>
My point is: environment tracking is good for developer and for the end user that wishes more flexibility than their distribution allows. While modern production environments for innumerable reasons should be fully virtualized. And good virtualization always employs single responsibility which, in its turn, means no environment tracking is required within virtual environment.
<Lartza>
sudo ./virus is much worse than ./virus
aradapilot has joined #spacedock
<AllisTauri[m]>
Oh, irc_Lartza , if you say so)
<Lartza>
It's simple as that
<Lartza>
A malicious process inside a container is still malicious
<AllisTauri[m]>
Says the one who doesn't use containers)
<Lartza>
Pff
<Lartza>
I don't use them because I don't need to
<Lartza>
I do where I need to
<Lartza>
But just to clarify, you are saying when creating a container for a python or a node application, installing python3 and python3-pip or nodejs and npm, then running pip/npm as root that then downloads archives that may or may not be compromised is totally fine?
<Lartza>
Instead of installing python3-venv in place of pip, creating a venv, and installing the same libraries there with sane privs and without compromising guest OS stability
<Lartza>
(in way of overriding system packages in site-packages, if your program happens to depend on a newer version than the OS may or may not need)
<AllisTauri[m]>
Installing compromized software in a container as a "root" of the container doesn't compromizes host per se, yes. Container's root is no root in the host. That's one of the reason for containerization.
<Lartza>
And that matters how?
<Lartza>
I mean it does matter but, that solves all the issues how?
<AllisTauri[m]>
What issues?)))
<Lartza>
Your container handling sensitive data for example I guess, like spacedock handling user emails
<Lartza>
etc
<AllisTauri[m]>
This is perpendicular. Emails are in another container, to which the SpaceDock is just a non-privileged client with it's own credentials and rights. Getting "root" in SpaceDock container doesn't compromizes DB. Getting access to DB is not easier with "root" in the container than without it.
<AllisTauri[m]>
* This is perpendicular. Emails are in another container, to which the SpaceDock is just a non-privileged client with it's own credentials and rights. Getting "root" in SpaceDock container doesn't compromise DB. Getting access to DB is not easier with "root" in the container than without it.
<Lartza>
So spacedock has no access any user data and no internet communication with arbitary IP addresses, cool
<Lartza>
*to any user data
<Lartza>
That seems fine then
<AllisTauri[m]>
It's a matter of design. You can have data-vulnerable software system based on containers. What containers solve is vulnerability to the host.
<Lartza>
I agree
<Lartza>
To the fact that containers solve host vulnerability
<Lartza>
The issue is running arbitary code as root inside your container that can do anything inside said container, and compromise it in any way it can inside that confinement
<Lartza>
That is not vetted by anyone
<Lartza>
I mean I'm not saying distro packages can't be compromised either but
<AllisTauri[m]>
And they help to solve data vulnerability by dividing responsibilities between containers, so that no single container, if infected, is able to do much or even any harm.
<Lartza>
I like the optimism
<Lartza>
Your container is linked to a database, that's already a lot of harm
<AllisTauri[m]>
You don't usually make a whole server application in a single container.
<AllisTauri[m]>
A container that needs DB knows a host:port and its own credentials and its own access rights in the DB. That's all.
<AllisTauri[m]>
It's no more or less dangerous that to have any other DB client anywhere on the network.
<AllisTauri[m]>
* It's no more or less dangerous than to have any other DB client anywhere on the network.
<Lartza>
I mean I don't tend to run pip/npm as root on those other DB clients
<Lartza>
For starters
<AllisTauri[m]>
Exactly; you don't run them on DB container.
<AllisTauri[m]>
Did I said anywhere that containers provide absolute security?)
<AllisTauri[m]>
And yes, you may run non-root, and many suggest this as a security measure. There's still no place for additional environment tracking there.
<Bart[m]>
AllisTauri, if you go full virtualization I don't see any particular issue with running things in userscape either.
<Bart[m]>
*root
<Bart[m]>
It'll be elevated to the level the VM runs at and nothing beyond that
<Bart[m]>
The only real risk of granting root access to something inside the VM is that you could expose yourself to potential VM escaping exploits.
<Bart[m]>
Since you'd have access to the raw read/write features and direct memory management over things like mounted drives and folders.
<Bart[m]>
At which point the strength of the VM implementation determines your security.
<AllisTauri[m]>
No issues with running userspace, but no point in venv, and no point in not creating runtime environment on system level
<Bart[m]>
The real issue is idiots running things like docker as root and then elevating things within the container to root as well >_<
<Bart[m]>
Because while the system itself might look safe, I've rarely seen a decent implementation of hardware level virtualization.
<Bart[m]>
And anyway, it's a false sense of security.
<Bart[m]>
And I'm out for a drink, dehydration is getting to me 😄
<AllisTauri[m]>
The only source of security is proper design of your particular system.
<AllisTauri[m]>
🍻
<AllisTauri[m]>
What irc_Lartza has actually convinced me is that there's nothing wrong with venv in production, if you have a non-exclusive production environment that is a given.
<AllisTauri[m]>
But when you can _design_ your production environment, I still think virtualization is a better way.
<AllisTauri[m]>
* What @irc_Lartza:52k.de has actually convinced me of is that there's nothing wrong with venv in production, if you have a non-exclusive production environment that is a given.
<AllisTauri[m]>
But when you can *design* your production environment, I still think virtualization is a better way.
aradapilot has quit [Remote host closed the connection]
aradapilot has joined #spacedock
<Bart[m]>
I disagree on virtualization being better, it's additional overhead for a false sense of security.
<Bart[m]>
If I penetrate the web frontend and get the DB credentials I can still dump the entire DB most likely.
<Bart[m]>
Exploiting the system itself is rarely the goal of an attack, usually it's something more along the lines of getting data
<AllisTauri[m]>
What you describe has nothing to do with either VM or venv, it's app design
<HebaruSa->
I'm skipping some of that scrollback, but AllisTauri, the www-data user already has write access to htdocs/SpaceDock on alpha. I made it GROUP writable.
VITAS[m] has joined #spacedock
<VITAS[m]>
AllisTauri: the folder can be owned by www-dataa just fromw hen i copied it over and forgot to chown it
<AllisTauri[m]>
There was no access to `static`
<VITAS[m]>
can someone explain to me the benefits of vagrant over docker?
<AllisTauri[m]>
And `npm` wanted strict permissions on `.npm` and `.npmrc` which it couldn set
<VITAS[m]>
SD alpha runs in lxc ist more like jails than a fully VM. im able to run quemu and KVM instances too but i would only do that for non linux os
<AllisTauri[m]>
Anyway, I've re-owned SpaceDock to `www-data` in order to build the statics
<VITAS[m]>
to precise im running proxmox
<AllisTauri[m]>
What I meant is that SpaceDock lxc should be, ideally, dedicated purely to SpaceDock, so it should be alright to install packages systemwide. On the other hand, there's no way we run gunicorn as root, so autodeployment would still need a way to install packages locally
<VITAS[m]>
ill be bussy for 2 -3 weeks starting from the 10th so some ground rules: HebaruSan is in charge of git and alpha/beta he can structure things. I trust AllisTauri to know the best waay to fix bugs in the code and to work on issues (and distribute that work with the help of HebaruSan ). RockyTV Knows SpaceDock for a long time he can help if questions arise "why we did things" (but no rewirtes froms cratch or go/backend project revivals!).
<VITAS[m]>
Darklight has acc to production and can fix things if something goes wrong there (no deployments of new code on prod just keep it running).
<VITAS[m]>
If moduploads dont work again just restart the prod container (its a workaround)
<VITAS[m]>
if a new ksp ver is released: some people on the forums have db acc to the ver table and can add it (just screm in spacedock forum thread)
<VITAS[m]>
scream
<VITAS[m]>
in general: youre free to configure alpha and beta how you like for as long as the end result is 1.) documented b.) makes developement easier c.) is understood by HebaruSan d.) doesnt give random people shell acc
<VITAS[m]>
and yes i should write this down in the wiki or similar when i have time.
<VITAS[m]>
any questions ? crisicism? sugeestions?
<HebaruSa->
I nominate DasSkelett as co-Hebarusan
<VITAS[m]>
what rights would he need and does he want to be your co?
<HebaruSa->
Just for the "understood by Hebarusan" part since I won't be responsive all the time
<VITAS[m]>
understood also means "document things"
<VITAS[m]>
am happy with him getting rights to edit the wiki
<VITAS[m]>
can you give him those or need i to do it?
<HebaruSa->
I think you need to do it
HebaruSan[m] has joined #spacedock
<HebaruSan[m]>
Oh, IRC messages are getting through to Discord again. 👍
<VITAS[m]>
uhh discord fixed things
<VITAS[m]>
ok.
<VITAS[m]>
the idea is that we can give new people the wiki as read and then have someone answer questions about procedures and setups
<VITAS[m]>
AllisTauri: can then tell them everything about the code structure if they want to help with coding.
<VITAS[m]>
i myself might need an update when i return.
<Bart[m]>
@AllisTauri#0000 I'm just pointing out you add complexity for a false sense of security
<Bart[m]>
It doesn't add any real amount of security against the actual attack vectors you're likely to encounter
<Bart[m]>
In the meanwhile the added complexity does increase the risk of configuration error
* VITAS[m]
invites Bart to help with SpaceDock.
<Bart[m]>
@VITAS#0000 I asked a couple of days ago if I should fix the templates lol
<Bart[m]>
Or if you're planning layout works
<VITAS[m]>
im planing to do something in vuejs when i have time (dont know when and if something useable will exist). fixing the current bootstrap based template is a good idea.
<VITAS[m]>
but please only do bugfixing and optimazation. no new layout as such.
<AllisTauri[m]>
Yea, they certainly need more modularization, structuring and better formatting. Those 150+ character long strings...
<VITAS[m]>
this fighting of botstrap instead of using it :)
<VITAS[m]>
im for fixing but not for rewriting because of my hopes towards vuejs
<VITAS[m]>
(dont want to waste your time and energy)
<VITAS[m]>
one feature i would like: is a way to bring the donations more into foreground.
<VITAS[m]>
both as sort of hall of fame of current donators and the amount of donations the site currently has.
<VITAS[m]>
i still feel that the ones who give money to finance things arent praised enough and at the same time i think more peopel should be aware that the site needs them to run (both in participation and money)
<VITAS[m]>
so either a footer thats always visibel with a donation button and indicator showing the current donation amount and/or a reminder in the same popup that suggest subscribing to email alerts after downloads
<VITAS[m]>
if you have a better idea (and maybe also one on what benefits donating people could get that dont restrict the current featurset)
<VITAS[m]>
please tell me
<djerun>
HebaruSan[m]: HebaruSa-: I intend to reserve some time to get an overview of the current state of spacedock between Fri 2019-08-09 19:00 UTC and Sunday 2019-08-11 19:00 UTC so I can actually do something when I have an hour or two every now and then... what time outside of 23:00-07:00 UTC is a good chance you have time to bring me up to speed?
<DasSkelett>
Thanks HebaruSan! Yes I think I can help you a bit review wise. I'll be quite busy the next few weeks myself, but together we should be able to handle the mass of PRs by AllisTauri :)
<VITAS[m]>
oh hi djerun (IRC) :)
<VITAS[m]>
Seh ich dich morgen auch auf deiner Arbeit?
<djerun>
VITAS[m]: yup
<VITAS[m]>
noch tips?
<djerun>
tuer unten ist offen
<VITAS[m]>
nicht die art tip, die ich erwartete, aber danke :D
HebaruSan[m] has quit [Quit: Idle timeout reached: 10800s]
<VITAS[m]>
AllisTauri: did i do it right and you can handle issues and merges yourself now?
<VITAS[m]>
Ive the feeling i should cleanup githubs temas and memebers
<RockyTV>
I missed these two days due to school but everything's good now
<VITAS[m]>
are you good at school?
<RockyTV>
I hope so, it started last week (got back from vacations)
<RockyTV>
😛
<RockyTV>
DasSkelett, what are you using to learn vuejs?
<DasSkelett>
My PC
<DasSkelett>
:P
<DasSkelett>
Otherwise, their documentation. And I watched a quick YT tutorial as quick start.
<DasSkelett>
reomve one of the quicks...
<VITAS[m]>
i use the documentation, searchengines some linux instances, laravel as backend (because it comes with vuejs) and phpstorm
<VITAS[m]>
ive a bunch of questions regarding vuex
<VITAS[m]>
that are showstoppers for me
<VITAS[m]>
so when one of you is far enough in his learning process i hope he can help me out
<VITAS[m]>
btw theres a vuejs and webpack channel on matrix im in
<djerun>
VITAS[m]: lies mal hackint /q
<VITAS[m]>
ups
<RockyTV>
VITAS[m], just wondering (you said no rewrites or go backend): what about converting and adapting the old bootstrap frontend to work with vue?
<RockyTV>
I mean it's not a priority but what are your thoughts?
<VITAS[m]>
as i wrote in irc
<VITAS[m]>
first get prod fixed then think about replacing parts or the whole thing
<VITAS[m]>
so more evolution not revolution
<VITAS[m]>
HebaruSan: can you merge or are there rights missing?
HebaruSan[m] has joined #spacedock
<HebaruSan[m]>
I've already been merging
<VITAS[m]>
oh uh ah cool
<VITAS[m]>
:D
<VITAS[m]>
ill leave it to you then for now insert respectable title here
<VITAS[m]>
Ims till getting used to not having to think about everything myself. a good feeling. Thank you all for it :)
Bart[m] has quit [Quit: Idle timeout reached: 10800s]
<DasSkelett>
I just solved the permission issue with postgres and docker.......
<DasSkelett>
AllisTauri: Postgres couldn't change permission of the directory, because the directory of the git repo was on an mounted NTFS drive. And Linux of course can't chown files/folders on NTFS, so Docker can't, so the postgres container can't...
<DasSkelett>
Didn't think of Docker really trying to change the actual ownership of the folder on the host system.
Gregrox has joined #spacedock
<Gregrox>
yo it's hanging on 100% again
AllisTauri[m] has quit [Quit: Idle timeout reached: 10800s]
<Gregrox>
VITAS
<DasSkelett>
Somebody here who can restart the container?
<DasSkelett>
Of course now they're all gone...
<VITAS[m]>
yes
<VITAS[m]>
dont see any 100% containers
<VITAS[m]>
sd1-6 are green
<DasSkelett>
Well I'm getting a Internal Server Error if I try to upload a test mod via the API.
<DasSkelett>
As last time
<VITAS[m]>
on prod?
<DasSkelett>
Yep
<VITAS[m]>
tried the website?
<DasSkelett>
Well... no
<VITAS[m]>
just to check if its not the api
<DasSkelett>
yeah, will do
<VITAS[m]>
and if that happens ask darklight in #dmp
<VITAS[m]>
he is at australian time
<VITAS[m]>
check and get back to me
<DasSkelett>
It does look like it is not working via the website either.
<VITAS[m]>
k i reboot
<VITAS[m]>
done
<VITAS[m]>
the frequency makes me nervous
<VITAS[m]>
it worked for month and now its weekly or every two?
<VITAS[m]>
can you log the times it happens so that we can find a system?
<DasSkelett>
Confirmed it works again via the website. API still doesn't work, that might be a different issue.
<DasSkelett>
I'll do
<VITAS[m]>
k thx
<VITAS[m]>
a workaround could ba a cronjob that triies to write to the folder and if it doesnt work either tries to remount it or reboots the container (and logs the incendent)
<VITAS[m]>
could you write such a bash script?
<VITAS[m]>
i could put it in place later today
<DasSkelett>
I think I should be able to to that. But someone should have a look at it before we activate it.
<Gregrox>
Is it fixed?
<VITAS[m]>
for now
<Gregrox>
I'll releaswe my two updates soon then
<Gregrox>
before it you know
<Gregrox>
breaks again
<DasSkelett>
But this will have to wait until tomorrow (/today, oh gosh is it late again)
<DasSkelett>
^^
<VITAS[m]>
yes
<VITAS[m]>
night
<DasSkelett>
Night!
HebaruSan[m] has quit [Quit: Idle timeout reached: 10800s]
AllisTauri[m] has joined #spacedock
<AllisTauri[m]>
DasSkelett (IRC): ah, so shifting of the pgdata anywhere inside of the mounted volume couldn't help! Either way, using `PGDATA` is the recommended way if persistence storage is mounted into container.
<DasSkelett>
It's really late over here, so there might be a few typoes and mistakes. I'll reread tomorrow, and maybe extend it more. @everyone, feel free to edit it, append to it, redo it entirely or whatever you want.
<DasSkelett>
But it should be a good start.
<AllisTauri[m]>
<VITAS[m] "AllisTauri: did i do it right an"> I sure can handle those myself; unless they're my own PR's, in which case a peer-review is a must)
<DasSkelett>
It would be nice if especially you AllistaTauri could have a look at it, since you did the the whole code structure and docker rework, and probably know better what you did than I do :)
<AllisTauri[m]>
Sure, was intended to, but not until tomorrow night or at weekend -- too much work right now. Or maybe I could do it from mobile during transits... we'll see/
<AllisTauri[m]>
* Sure, was intended to, but not until tomorrow night or at weekend -- too much work right now. Or maybe I could do it from mobile during transits... we'll see.
<DasSkelett>
Yeah of course, take your time. It's nothing urgent. Most of all since I'm going to bed now :D