fah.crazy-logic.co.uk failing :/

So recently I’ve noticed that the site is struggling quite badly to process the data – the database has grown to just shy of 2.5GB and well it’s not optimised is any way at all.

So there’s 95173 teams from my last data collection and of those 2158 have a score of 0 and 164 have no WU. IE these are unused teams. So in total over the last 5 months, that’s generated an extra ~300k of history records that are not needed at all. Then there are the teams who are not active, also having many records that I don’t really need to worry about creating.

To put things into context – this is a pruning exercise. The DB is too big, so the useless data needs to go. Given that in the 5 months my team has risen to the top 3000 teams with less than 2 machines running for that time, I don’t think this will affect the data validity or usefulness of the site long term.

So to keep the ship from sinking I think it’s time we removed the teams with no WU or Score from the history (after creating a backup). When I say no WU or score I actually mean all teams with less than 10 WU and less than 200 score.

So “SELECT * FROM `teamhistory` where score < 200″ yields 974,625 records and “SELECT * FROM `teamhistory` where wu < 10″ yields 4,459,439 records. Given there’s around 14,000,000 records, removing 30% of them should see a massive speed up in future queries. 

So after deleting some records, the table seems to have shrunk, but I think I will have to review how I keep team history in the future and the size of the fields in the table as it has grown way too fast from January till now.

 

LED LED LED

So this week I’ve been working on two LED projects, one of them is more just a thing I want to do for some pub gigs and that and the other might develop into a sellable product.

So number 1: 

Recycling some old tape into LED panels for lighting bands and stuff at pub gigs, in my workshop for making videos and maybe on gigs ect. It’s a somewhat limited design but, small panels with upcycled LED tape stuck to them, controlled by some LED drivers that I’ve had lying about for some time now doing nothing. After an hour of fiddling, I proved the concept for my needs and I will now order some connectors to make 6 panels for further experimenting.

prototype LED panel

prototype LED panel

and number 2:

So it’s a 5 by 5 PixelPanel, using some recycled ws2801+5050 LED’s, and my idea is to make these panels controlled by Artnet and use PoE to power them. Still very prototype at the moment! Sorry for the crap photo.

5 by 5 LED PixelPanel

5 by 5 LED PixelPanel

Todays trip to the IoT expo 2017 was disappointing.

So I turn up – hop in a lift upto the 2nd floor – the lift has a CAN bus error, still working tho :s. Then I proceed to get my badge, the printer has a jam, and the next kiosk just refuses my barcode…. off to see a real person it is then. so far the things are letting down the team.

So I have a little wonder round, most cubicles are bland and blank, a few were empty, but had signs up for exhibitors who I guess just show up. I was looking for something new, different or simply cleaver, but I don’t really think I saw anything. I was really surprised not to see Arduino there, or the Raspberry PI Foundation, or AWS…. but Ubuntu was which is ok I guess. Anyway it really did feel fragmented and nothing really flowed.

Here’s the things I found interesting enough to snap.

sponsor banner

sponsor banner

A robotic art, controlled by audio not the internet

A robotic art, controlled by audio not the internet

a Raspberry PI based controller

a Raspberry PI based controller

a weird head thing

a weird head thing

Pub en-route

Pub en-route

Installing Transmission BitTorrent web interface on Ubuntu

Here’s how to do it – if you have the desktop version, most of the hard work has been done for you, but here’s how to open the web interface up to the world/your LAN anyway.

I’ve made a new directory in the root of the drive for files, called transmission.

  1. sudo apt-get install transmission-cli transmission-daemon
  2. cd /
  3. sudo mkdir transmission
  4. sudo chgrp -R debian-transmission /transmission 
  5. sudo chmod -R 775 /transmission
  6. sudo service transmission-daemon restart
  7. sudo nano /etc/transmission-daemon/settings.json
  8. make edits to file:
    1. “download-dir”: “/transmission”,
    2. “rpc-password”: “your-password”,
    3. “rpc-port”: abcd
    4. “rpc-whitelist”: “127.0.0.1,192.168.*.*”,
      the password will be hashed on the next restart.
  9. sudo service transmission-daemon reload
  10. sudo service transmission-daemon restart
  11. enjoy vie web interface. 😀
  12. remember to push a port through your firewall….

If you want to do this from the server distro, you’ll need to install ‘transmission-common’ also.

Up-cycling an old server? pt 2

So today I decided it was worth adding in the second GTX 750Ti that was donated to me. this was successful and I managed off a 500w PSU to have the dual core CPU and the 2 graphics cards folding. The cpu has a 65w TDP, and the graphics cards are supposed to be 75w ish each, which means 215w plus the HD all off the 12v rail. Well the PSU is 17A 12v which is a maximum of 204w…. hummmmm. But worked fine.

I did some measurements as I added bits in;

  • 0w unpluged (well durr),
  • 5w standby power consumption,
  • 69w powered on but idling (one GPU installed),
  • 139w with 1 CPU and 1 GPU folding at max,
  • 151w with 2 CPU and 1 GPU folding at max,
  • 81w powered on but idiling (two GPU installed),
  • 152w with 1 CPU and 1 GPU folding at max (2nd GPU installed and idiling),
  • 172w with 2 CPU and 2 GPU folding at max.

So, comparing this for folding to my post here, (based on 24hr period):
the M600 server blade 23000/8.64kWh = 2662 points/kWh,
the QC Core 2 Quad 6500/2.62kWh = 2480 points/kWh,
the QC with 750Ti 38000/3.6kWh = 10500 points/kWh,
the DC with 2 750Ti 76000/4.13kWh = 18402 points/kWh,

Meaning that the dual GPU setup is clearly the winner…

QC with the 750 would generate around 13,870,000 points/year and cost £130 a year to run, the DC with 2 750 would generate around 27,740,000 point/year and cost £150 a year to run… clearly the new setup wins… but… NO.

The problem here is that I want to use the host to run some VM’s doing specific tasks, but with each GPU tying up a CPU core, and only being a dual core machine this would be a hopeless setup. Not to worry – I have a few Quad core machines lying about not doing anything so, I pop in a Core 2 Quad and well… it didn’t work! back to the manuals. So this clearly shows that it does support Core 2 Quad’s, but not the quad core’s I have which are the Q9300. See comparison here between the DC and QC I have and the one I should install. Damn it.

I should really think about the 12v supply a bit more – moving buying and installing a Q6600. It would potentially push me way past the PSU supply limit (75+75+105= 255w, 204w limit). This is one reason why anyone using multiple graphics cards is installing massive PSU’s not for the Wattage, but need for a massive 12v supply rail.

Ah well – time to commit to the dual core with only 4GB RAM for the time being. 🙁

Forgot to mention i found this old diagram for the original motherboard

Original mobo

Original mobo

Up-cycling an old server?

So For many years i’ve had an old RM server sitting under my bench, powered off, gathering dust inside and out. But no more – it’s time for it to either do something or be recycled. So, as a classic hoarder, i’m going to up cycle it rather than let a really good case go to waste.

RM Server Case

RM Server Case

So the case is a “RM systembase TX” by Research Machines, which is actually a “Antec SX1040BII Performance Series II SOHO File Server” that’s been rebadged. Originally the spec would have been something like single or dual processors, sub 1GHz, but I can tell you that the RAM was PC6700, all 256MB of it (RM used to include that on the sticker on the back, that remains today). That also would have been top spec at the time. I vaguely remember there also being a a PCI SCSI card for the 18GB hard drives and the tape drive. The case actually supports EATX motherboards, and had 2*3bay removable hard drive caddies (one slot used by mandatory floppy disk drive – LOL), as well as 4*5+1/4in drives, it has (remaining) 3 case fans too.

But, in the past i had already “upgraded” the system. So out with that rubbish… I removed;

  • MicroATM mobo with,
  • P4 2GHz,
  • 2*1GB RAM,
  • IDE hard drive,
  • IDE CD Drive (left in place but not connected),
  • 3 random PCI ‘fast’ ethernet cards

and i had replaced the PSU to something nice too (500w OCTIGEN jobby).

old mobo

old mobo

old mobo

old mobo

removable hard drive cage

removable hard drive cage

fans

fans

PSU

PSU


and in with the new…. well, 2nd hand but new to me?

  • HP xw4600 ATX motherboard,
  • 2GB of PC2 6400 RAM,
  • Intel Core2Duo processor E6550,
  • brand new Arctic, Alpine 11 CPU cooler (with some light filing mod),
  • 3 old hard SATA drives
    • 500GB Seagate,
    • 1TB Toshiba,
    • 1TB Western Digital Black,
  • EVGA GEFORCE GTX 750Ti (kindly donated to me for folding purposes).

Awesome – so i start some CPU folding using my .deb package from a few posts ago and then i also install lm-sensors by;

sudo apt-get install lm-sensors 
sudo sensors-detect
sensors

Which gives me readings of 29deg for the GPU and 55/56 for the two CPU cores.

loaded CPU temps

loaded CPU temps

So now to install the correct graphics drives and try again with GPU folding. (367.57 driver which now seems to just work from the off, or maybe i just didn’t configure it correctly last time??? probably.) After a restart the sensors command also returns more than just the CPU temps, but now the graphics is missing…. typical. so here’s the sensors data with the VNIDIA X Server Settings showing the temps.

the graphics and sensors data

the graphics and sensors data

Right, best leave this for a while to make sure that the CPU heatsink is installed and dissipating correctly then i’ll start the good work on the machine. Time for dinner!

Installing a GTX 750ti for Folding at Home on ubuntu

So first things first, update the OS to about 16.04 LTS.

Second thing install the graphics driver using Synaptic package manager.

Third install FAH, and then connect to it…..

Folding @ Home error

So back to 14.04.4…. YAY, a working folding machine.

Folding @ home back on 14.04.4

So what about power consumption?

So my M600 blades at full pelt with the 8 cores can do ~23000PPD but use around ~360w (without the chassis).
My QC Core 2 Quad above without the graphics card at full pelt can do ~6500PPD and uses ~110w.
The same QC but now with a GTX750Ti at full pelt on the graphics and 3 cores on the CPU ~38000PPD using ~150w.

So to compare efficiencies (based on 24hr period):
the M600 server blade 23000/8.64kWh = 2662 points/kWh,
the QC Core 2 Quad 6500/2.62kWh = 2480 points/kWh,
the QC with 750Ti 38000/3.6kWh = 10500 points/kWh.

So the QC with the 750 clearly wins and would generate around 13,870,000 points/year and cost £130 a year to run.

 

Better analytics for Folding@Home…

… were needed IMO so, I decided to build my own and they’re now at http://fah.crazy-logic.co.uk. it’s not a nice interface at the moment but the data is being collected in the background 🙂

The home page shows the top 10 teams for score and also for work units, the ten trending (not worked on this algorithm yet) and the ten newest teams with non zero scores/WU. Clicking on a teams name displays the teams info and some historic values.

Here’s how it works, 4 times a day data is pulled from FAH, then added to the database. New rankings for score and work units are then calculated. Once a day a history entry is generated and then in the background in batches (of 500 records) the score differences and that are calculated… basically lots of scripts running in the background, triggered by cron, and theres a table recording time, script and time taken and also if the script gets to the end or times out.

Things I need to do:

  • A nice interface
  • fix the efficiency of the differencing script…. it’s a massive fail atm. (indexing)
  • Need to add some basic site stats, days, records, size… queries per day and database use?
  • I will be adding some graphs in the future,
  • Signature thingy.

Oh and then I found: http://folding.extremeoverclocking.com/team_list.php

edit {hack} a .deb package to remove user options – silent install.

So, with my new Dell M1000e and M600 blades now sitting in their new home in my workshop and having tested all the blades are working (-1) I now need to load test them. I could just use a program designed for this, but i’m thinking that as i’ll be burning through a chunk of electricity i’ll put the compute cycles to good use (the heat will be keeping me warm). So i’m going to do some Folding at Home. If you don’t know about this, then you should check out this great project.

My team id is 232280. Feel free to join me.

Anyway, i’ll be running Ubuntu on the blades for a while (when I can get the latest version installing from USB), so I played around with installing inside a VM. The instructions for this can be found http://folding.stanford.edu/home/guide/linux-install-guide/ but take them with a pinch of salt as a few links are broken/need fixing (i’m letting them know on the forum). The installer asks for user details as part of the package and you can’t do an ‘unattended install’ which is what I was after (for a very specific reason). NVM, i’ll just figure out how to disassemble the .deb package and change it.

I haven’t done this before so I started not know that it was possible, but it’s actually quite simple. I found a post explaining how to open and then recreate the .deb package.

mkdir tmp 
dpkg-deb -R original.deb tmp 
# edit DEBIAN/postinst 
dpkg-deb -b tmp fixed.deb 

…and then had a bit of a read up of what I found inside – read this, sec 7.6. After a bit of head scratching and a few edits that didn’t work I finally worked out what I needed to do to remove the user interaction. (some of this helped.) Rename the templates file and comment out the lines referencing db_get then add in the variables for user, team, passkey, power and autostart. (One thing I would like to do is add some sort of reference to different machines MAC address in the user but i’ll look into that later.) After doing this i’ve hacked the .deb into something that I can script silently – which means remote deployment :D.

I also came across https://www.linux.com/learn/writing-simple-bash-script while wondering the web – it’s really well written for beginners like me.

You can find my hacked .deb here. My team id is baked in. Install using:

wget http://www.crazy-logic.co.uk/wp-content/uploads/2017/01/FAHinstaller.deb
sudo dpkg -i --force-depends FAHinstaller.deb

ver 2 now includes the mac address in the username so i can see how much each blade will have done.