New ‘Home-Lab’
My home lab is a bit silly – multiple rack servers, laptops, other computers, and a blade server… most of the time it’s all turned off unless I need it (apart from 2 Lenovo x230 laptops I’ve being using as home server and dev machines). Anyway – time to move forwards, and consolidate.
My new ‘Home Server’ is a Lenovo M710q – 8 x Intel(R) Core(TM) i7-6700T CPU @ 2.80GHz, 16GB RAM, 1TB NVMe storage running Proxmox. It’ll host my home VM’s and also my dev machines that stay on 24/7 (LAMP, LNMP, DNS ect). Power usage was a concern, as is cost so that’s why I when with this choice.
So my ‘Home Lab’ really is now just my desk workstation which is a 12C/24T Xeon with 64GB RAM. It’s only on when I need it so i’m not too concerned about it’s generally higher than modern computer power usage.
And then there’s the ‘Test Bench’ which is all the older server kit – Tho I think I will try and get rid of most of that this year. The HP Z420 with a 12C/24T and 48GB RAM is pretty much the most I really need for alot of my test scenarios now. (End of month pre post review – it’s not enough, two of them aren’t either.)
DDOS issue investigation
Was on a gig and got internally DDOSed by an application, it was an interesting DDOS as it was effectively a slow loris attack, but due to the number of clients it presented as a state table panic / NAT exhaustion event. Have a read here on how they work – https://www.cloudflare.com/en-gb/learning/ddos/ddos-attack-tools/slowloris/ – ironically the endpoint was ‘protected’ by Cloudflare…
In pfSense it appeared to lock up the NIC’s, causing momentary loss of traffic passing, and when we moved to some Cisco hardware, connections would timeout and essentially only connect when a NAT port became available. Looking into a NAT table extraction, 5% of the clients accounted for 91% of the NAT table, and all to one remote host behind Cloudflares DDOS protection.
Didn’t really plan to test for this kind of unusual connection traffic from within the network your looking after, and especially not to an address within Cloudflares blocks. And there isn’t really any obvious way to see if pfSense that it’s occurring – Yes the states summary page exists and shows states per host, but with 300k+ states on an active firewall it takes a while to load (takes a while to load due to the way it pulls the state table (pfctl -s states) into PHP memory then does all the analysis by pulling apart the lines of text). Oh and if the NIC ‘locks up’ while trying, you get no response from the server. https://github.com/pfsense/pfsense/blob/master/src/usr/local/www/diag_states_summary.php
So there’s 3 things to take away
– How does one see this happening before it kills your firewalls?
– how does one replicate this?
– how do I test for this in the future?
I couldn’t find anything that generates connections in this way so… it seems like i’ll need to create something. Time to break out Node.js and websockets.io.
Port exhaustion is an interesting topic from this gig as I actually had to set some static NAT for Teams to work properly, and with 200+ teams clients using only 60 ports outbound I had my concerns. This wasn’t a problem as the endpoints are actually massive subnets rather than one host, and Teams load balances this.
Carbon UI in W3.CSS
I’ve been building a new UI style using W3.CSS but in the IBM Carbon style (ish). This is because a website I look after needs a UI refresh and i was something nice, simple and quick to load. Minimal frameworks in use, the way it should be!! The W3.CSS style sheets make it very easy to make nice looking UI’s with minimal thought from myself, but I do have to add custom CSS to make something’s work.
Storage sort out
Had just under 5TB of files from the gig mentioned above to sort through, things to keep, things not needed, log files of interest (DNS was huge). I should probably make a video on how I mount qcow disk images within my desktop machine so I don’t need to keep the VM’s clogging up my virtual machine manager on my desktop.
PPT puller
Started another interesting project – you upload your powerpoint and it tells you if there’s any audio or video on each slide and which files it is and possibly also de-embeds the files for you also to download. lots of work needed still, but I think it’ll be very interesting and lots of pople have expressed interest in it so far, with lots more feature requests.