<Darklight[m]>
It's normal for free to be nothing but if it lives in buff/cache, I'd be slightly concerned but I don't know anything about LXC and there is no fucking way I'm touching prod
<Darklight[m]>
dmp3 is 512mb allocation with 400mb free
Darklight[m] has quit [Remote host closed the connection]
VITAS[m] has quit [Remote host closed the connection]
im52kde has quit [Remote host closed the connection]
im52kde has joined #spacedock
Darklight[m] has joined #spacedock
<Darklight[m]>
Actually, I'm gonna go get some food - I guess I'll leave you to it, I'll still have my phone on me but I'll be gone for a little bit
<Darklight[m]>
Hrmmm, and there is 300gb worth of mods, were you on the ax41?
VITAS[m] has joined #spacedock
<VITAS[m]>
ive an older model
<VITAS[m]>
with 512gb ssds
<VITAS[m]>
i wish i had 1tb
<VITAS[m]>
but that wouldnt solve my storage problem compleatly
<VITAS[m]>
because ive problems fitting all os drives of all containers on the ssds as is
<VITAS[m]>
cant even think about data storage
<VITAS[m]>
my wet dream would be a second server and 10gbit dc internal network
<VITAS[m]>
ive a fileserver with multiple 8tb drives in another location with 1gbit/s uplink
<VITAS[m]>
but its not a datacenter but our club rooms
<VITAS[m]>
so no theft protection, real ups or climate
<VITAS[m]>
going back to magnetic drives isnt an option either. db would explode in our face
<VITAS[m]>
is slow as is
<VITAS[m]>
investing more money is a no go too as i dont have a job and donations are getting fewer
<VITAS[m]>
sd is back up
<VITAS[m]>
at least rebooting works
<Darklight[m]>
The db isn't on external storage though, only the 300gb of mods which should have nothing to do with the db or gunicorn as it is proxypassed, but yeah that does suck >.<
DasSkelett has joined #spacedock
HebaruSan has joined #spacedock
VITAS has joined #spacedock
<VITAS[m]>
no the db is its own container but all on the internal ssds
djerun has joined #spacedock
<VITAS[m]>
i did it that way so i can have one psql and one mysql instance for all my needs
<VITAS[m]>
e.g. nextcloud and mailserver are using mysql
<VITAS[m]>
some monitoring uses psql
<VITAS[m]>
its funny that neither cpu nor ram are my current bottleneck
<VITAS[m]>
its storage
<VITAS[m]>
size and io delay
<VITAS[m]>
i knew a guy who was THE GUY to go to for all your datacenter needs in my city
<VITAS[m]>
but he recently died so that isnt an option anymore
<VITAS[m]>
i always dreamed of buying some fancy rack servers and putting those itno colocation but colocation is as expensive as renting whole machines
<VITAS[m]>
and you have to spend 1000s to buy the gear in the first place
<VITAS[m]>
and you have to physicly check it regularly
<VITAS[m]>
so craperino in cans
VITAS[m] has quit [Quit: Client limit exceeded: 6]
<Darklight[m]>
I've been thinking about the problem with sd a bit, I can think of a few things that would prevent the site from dying if you allowed it to work from a second storage directory that we keep really small, but I don't know how much free space you have to spare.... I think automatically detecting when storage dies and making it recoverable is the more sane option though, given that you have to work with what you have
<Darklight[m]>
A simple-ish solution is moving ancient mods off of spacedocks main and into a distributed filesystem but that would require volunteers and redirecting the missing mods from storage to that instead
HebaruSan[m] has joined #spacedock
<HebaruSan[m]>
You guys could even redirect to CKAN's archive.org collection if that would help
VITAS[m] has joined #spacedock
<VITAS[m]>
its not that the site is dying because of storage acc
<VITAS[m]>
its going ro because of them moving the storage to different servers with different ips
<VITAS[m]>
archive.org isnt an option. mod authors would rage and its no smb or nfs storage i can acc encrypted
<Darklight[m]>
It is likely the way it is mounted giving you problems in that case, umount won't work with in use directories and your docker will hold the directory open I think, so you would need to signal a shutdown, unmount and remount. lazy unmount (unmount -l) forces it to let go and may help, but when filesystems hang sometimes you get zombie processes >.<
<VITAS[m]>
its not docker im using
<Darklight[m]>
The idea I have in mind is to add a /mnt/storage2 and only keep a small amount of data in /mnt/storage, and if it exists in /mnt/storage you pass that file, if it doesn't you pass it from /mnt/storage2, if /mnt/storage2 breaks then you unmount and remount /mnt/storage2
<Darklight[m]>
I don't know if it's the correct solution, and it's slightly more dangerous as data is moving around
<VITAS[m]>
and i cant remount the share easily
<Darklight[m]>
But it's better than "VITAS spacedock is broken and i can't upload my mod" 😕
<VITAS[m]>
huh? what is?
<Darklight[m]>
When the storage breaks spacedock hangs right?
<VITAS[m]>
no
<VITAS[m]>
you cant upload
<VITAS[m]>
dl is fine
<Darklight[m]>
Ah because ats cache
<VITAS[m]>
no
DasSkelett[m]1 has joined #spacedock
<DasSkelett[m]1>
The drive is still accessible, just read only AFAIK
<Darklight[m]>
What in the fuck
<Darklight[m]>
Does mount -o rw /blah/blah work?
<Darklight[m]>
Does mount -o remount,rw /blah/blah work?
<VITAS[m]>
storage <- smb -> proxmox (physical host) -> vdisk file -> lxc (sees it as normal ext4 hd)
<VITAS[m]>
if storage is moved to another ip and dns is updated (dont ask my why they dont use virtual ips)
<Darklight[m]>
You can flush the DNS cache
<VITAS[m]>
smb tries to use the old ip because dns cache
<VITAS[m]>
during that time lxc has write errors and switches to RO
<VITAS[m]>
because it thinks hd is breaking
<Darklight[m]>
I don't understand how the heck it can be read only if the host is missing >.<
<VITAS[m]>
because it doenst stay missing
<Darklight[m]>
Ok that makes a lot of sense
<Darklight[m]>
So it only downs for a short while?
<VITAS[m]>
its just till smb & dns cache figure out it needs to requery
<VITAS[m]>
yes
<VITAS[m]>
and in addition to that there are limits on parralel writes
<VITAS[m]>
not connections
<Darklight[m]>
mount -o remount,rw /path almost certainly will work
<VITAS[m]>
might
<VITAS[m]>
the same smb share is used by multiple containers
<VITAS[m]>
so doing stuff with it is tricky it might cause other containers to go into RO as well
<VITAS[m]>
just from fixing it
<VITAS[m]>
ah you mean remount inside the container
<Darklight[m]>
Yeah
<VITAS[m]>
yeah i tried that i think
<Darklight[m]>
I've typed this a few times on my pc: sudo mount -o remount,rw /
<VITAS[m]>
because of some other crap rebooting the container doesnt even work
<Darklight[m]>
(If it was root)
<Darklight[m]>
I know you can switch it live but sometimes things get stuck on sync and turn into zombies
<VITAS[m]>
it wont be able to get acc to the vdisk file
<VITAS[m]>
so theres something else going on too
<VITAS[m]>
my guess is parralel write limits
<VITAS[m]>
on the storage
<VITAS[m]>
so you can mount it as often as you want but their fileserver only allows x parralel writes
<VITAS[m]>
and if i reboot the container or try remounting it another container takes up those
<VITAS[m]>
i find their cfg crap and confusing at best
<Darklight[m]>
I think I'd be talking to their support, and if they don't have it give them a zero star review and switch provider
<VITAS[m]>
the problem stems from them designing it as backup storage and marketing selling it as "normal cloud storage"
<VITAS[m]>
i tried they got upsed and yelled at me that its for backup only
<VITAS[m]>
their site says "storage box" and even sells nextcloud instances running on it
<Darklight[m]>
Well, they are retards
<VITAS[m]>
yes and i cant find somethign similar priced
<VITAS[m]>
btw the ram usage from earlier has nothing to do with storage
<VITAS[m]>
maybe you find out why that happened (didnt before)
<Darklight[m]>
Oof if only I lived closer I'd throw you a network block device or nfs... the only problem is I only have 20mbit up >.<
<Darklight[m]>
The advantage of nbd is you could encrypt the contents your side too and I would have no access to it...
<VITAS[m]>
ive a fileserver with 1gbit/s up
<VITAS[m]>
but its not in a controlled place
<VITAS[m]>
so i wouldnt trust it for more than my movie collection
<VITAS[m]>
and encrypted backups
<Darklight[m]>
So encrypt the data :3
<VITAS[m]>
its more about availability
<Darklight[m]>
If it's higher than the nonsense your dealing with it might be viable
<VITAS[m]>
i would trade crap storage for storage without ups or airconditioning
<VITAS[m]>
i cant say
<VITAS[m]>
yet
<VITAS[m]>
either way lets sd apear less professional than it is :D
<DasSkelett[m]1>
We could all setup some storage at our homes and make it a big ceph cluster :D
<VITAS[m]>
would be fun for something non productive :D
<Darklight[m]>
This is how I would do it in an untrusted area possibly, although I can't imagine nbd's performance is anywhere near what you need because it's block level
<Darklight[m]>
This is not a solution for spacedock
<Darklight[m]>
Just saying there is tools out there to do it 😛
<Darklight[m]>
I'd definitely do this for temporary storage while moving providers if I didn't have full ownership of the destination, but that's one hell of an edge case
<VITAS[m]>
crossdress-pictures?
<Darklight[m]>
It's exactly what you think it is :3
<Darklight[m]>
I thought "so I want to encrypt something what should I pick.... needs to be small as I only gave it 1gb of space"
<VITAS[m]>
andbe importnat to you :D
<VITAS[m]>
i knew someone who spend his downtime constantly resorting his porn pics
<Darklight[m]>
Currently those files live in an unencrypted ext4 loop device, but I'm the only one with access to it anyway so *shrug*
<VITAS[m]>
one day after hair color next day after position and so on