wow, TrueNAS SCALE moving from jails to docker/k8s is such a hot mess.

Graham Sutherland / Polynomial

you can install "apps" which are just docker containers from dockerhub. if you want any kind of customisation of that image you're shit outta luck unless you want to roll your own docker image hosted on dockerhub. everything uses the same IP address as the NAS, and just forwards individual ports, so you can't host apps on their own IP / hostname with services on standard ports. awful.

:PUA: Shlee fucked around and

@gsuberland at least they're moving away k8s to just compose...

a tiny bit better

Graham Sutherland / Polynomial

people on the support forums and subreddit have asked about hosting apps on their own IP, and are told it isn't supported and without fail there's always a series of incredulous "why would you want to do that?" bullshit replies as per usual for any kind of linux community support

Erin 💽✨

@gsuberland this isn’t even how docker or k8s normally work wtf are they doing.

Erin 💽✨

@gsuberland heck k8s is an incredibly flexible flexible system with many options but one thing you can rely on is that every Pod will have a unique IPv4 or IPv6 address (potentially both)

Graham Sutherland / Polynomial

@erincandescent the way they've configured it is every container does get its own IP but it's on the internal 172.x.x.x subnet so you can't reach it from anywhere but the host machine. it lets you configure a port forwarding rule from the host IP to the container but you can't give the container a proper IP on the host network.

Graham Sutherland / Polynomial

@erincandescent so for example I've got TrueNAS SCALE running on 10.69.222.1/16. I installed httpd (apache 2.4) as a "custom app" and it was assigned 172.16.1.14. obviously I can't reach that IP from anywhere on the network. the only option I am given is "port forward 10.69.222.1:xxxx to 172.16.1.14:80". which means everything has to be on the same IP and on random high ports, which fucking sucks.

Erin 💽✨

@gsuberland Given they’re running k8s I’d expect them to just run an ingress e.g. Traefik and create ingress rules so you can have nice hostnames >_>

Graham Sutherland / Polynomial

@erincandescent honestly at that point I'd say just throw the lot in the bin and use lxc, 'cos clearly this is way too much complexity for hosting apps

Erin 💽✨

@gsuberland eh I much prefer the “run an ingress and give things hostnames” approach to assigning each service an IP even if it has an additional moving part

But also what you want is something they could easily offer on k8s with some careful CNI config.