hyperscale.at

Just got back from International CES. Nice to see some familiar faces and meet many new people! McDevOps makes computers for DevOps. The newest computer we’re working on is called Dynamic-Periphery^TM. Unsatisfied with the one-to-one constraint of personal computing, we decided that a workstation isn’t a personal computer. One workstation, powered by supercomputers, could be accessed by many tablets. But we couldn’t just use any tablets, we needed dynamic periphery. This means that one user may use several tablets in order to have a more tailored user experience, and be able to send their user experience to another user. Portable user experience is one of the most exciting features of cloud-based virtual desktop infrastructure. So we took it a step further and began designing specialized tablets for use with desktop supercomputing workstations. It just makes more sense in today’s software engineering, video production, and enthusiast gaming environments. After all, if DevOps culture doesn’t constrain what-could-be by what-is, then why should hardware constrain platform service software? We think it’s also better to have a consistent user experience in development and production and we think a common yet flexible software framework (for example prototype-friendly structure programming in Dart or open PaaS frameworks like CloudFoundry and OpenShift) facilitates this efficiency in many software engineering practices.

Dog-fooding the Supercomputer

But honestly, localized computation is only half of the fun of cloud VDI. We really wanted to rock the portable UX over the internet globally. And that’s doable with a McDevOps microcloud account (contact me if you want an invite), whether or not you roll your own microcloud. Microcloud accounts will be free for engineers, developers, designers, and devops culturists… and in general free for anyone looking for work or something to hack on. But it’s not just a SaaS model, it’s a PaaS model from a software perspective. From the hardware perspective it’s a gateway appliance taking you through the pearly gates to supercomputing heaven in the cloud. Desktops are a heavy workload in and of themselves, especially in the aggregate. The problem with all the cloud hype in consumer electronics or “personal cloud” is that they’ve gotten away from cloud computing’s future value. The future value of cloud computing is that it offers scalability. As Dave Nielsen says Cloud computing is OSSM (On Demand, Scalable, Self-serviceable, and Measureable), and I say it’s OSSAM (adding Automation which is implied in every letter of OSSM)…. consumer electronics manufacturers haven’t really delivered the scalability components, but rather what seems to be an overprovisioned appliance or box. The cloud is not a box, nor a puppet show, but maybe more like a vending machine. Get served.

We might be engineers or developers but we’re often not a this-or-a-that we’re often both. And I think in DevOps culture this is the case. I think it’s also the case that a desktop hybrid microcloud can handle heavier video production workloads much better than a beefed up mac (request demo), due to parallel elastic provision at hyperscale supporting rendering workloads for example. And that’s just one example because rendering is just one video production workload. And when these guys get bored they play LAN parties which works really nicely with a desktop microcloud in your cube farm or wherever.

So think how software engineers play with supercomputers while video producers play 3-D shooters. It’s a competition, but for practical purposes the same infrastructure is used to prove the concept that collaboration is like competition on steroids… especially when you can use the same tools and share the same big data insights.

So at CES this year it really seemed as though cloud either meant wireless or SAN or NAS… but I think cloud storage is a nice low hanging fruit. Cloud persistence is the other benefit of microcloud. It’s a gateway to public utility persistence of files. So it takes the load off your tablets and keeps things locally accessible via ultra high speed bandwidth while it slowly persists remotely in heaven… eventually consistent and redundantly persistent… You can take it with you.

In a previous post, I described an experimental method of mounting S3 as a virtual file system within a cloud instance. I’m still in the process of doing spring cleaning… although fall is basically here… but anyway cleanliness is generally overrated until it comes to the idea of getting web files organized properly in the cloud. So before I take a shower this morning I think I’ll finish moving some static content into content distribution networked storage bit buckets.

#!/bin/bash
# Asher Bond 2010
# http://www.asherbond.com/blog/2010/09/23/sizeup-sh-for-cleaning-house-i-mean-cloud/
# sizeup.sh [dir]
# sizes up the present working directory or some other directory
# by summarizing the directories inside. I use this script to
# make sure my cloud compute instances are storing files properly
# in walrus and s3 filesystems or google storage instead of cluttering
# the compute instance's internal file systems

if [ "$#" -gt 0 ] ; then
        cd $1;
fi

find . -maxdepth 1 -type d -exec du -hs {} \;

A Content Distribution Network is often more efficient than one point of delivery such as a single or centralized web server / VPS.

Please note this is risky business down here…
Here’s how a thrill seeker could try to move a whole bunch of stuff into a cdn real fast:

sizeup.sh
# oh wow what’s up here? that one is like fulla mp3s and videos
s3-mount.sh cdn.somedomain.com /mnt/cdn.somedomain.com
mv /var/some-directory/some-big-podcast-2010* /mnt/cdn.somedomain.com
echo “hey web master dude I just moved all ur files into the cdn, so update ur links.” | wall
# oh no I’m still the web master…
tar -cvf backup-in-case-my-links-dont-get-updated.tar /var/some-web-site
gzip *.tar
umount fuse
s3-mount.sh backups.somedomain.com /mnt/backups.somedomain.com
mv backup-in-case-my-links-dont-get-updated.tar.gz /mnt/backups.somedomain.com
cd /var/some-web-site
find . -type f -exec perl -p -i -e ‘s/some-old-links/some-new-cdn-links/g’ {} \;
# whoa i hope that worked… LOL!

If you’re wondering if there’s a difference between backups in the Service-oriented Cloud and backups that the rest of the world is familiar with… well… there are 5 key differences mostly stemming from the fact that cloud backups are service-based (surprise-surprise)… but the rest of the world is probably not familiar with backing things up anyway, so let’s continue.

In SOA, a “cloud backup” is done by taking a snapshot outside the virtual device being backed up. It’s not really new. This can be done using XEN, Veritas, Amazon Elastic Block Storage snapshots, etc, etc…

Some people don’t like the idea of backing up an entire instance with it’s binaries, log files, and duplicate data. I believe that redundancy is useful if not necessary for reliable backups, so I take the big snapshots once per day or so, but I also back up smaller files more frequently for added roll-back-ability or whatever you want to call it. Here’s how I back up scripts from cloud appliances to my S3 bit buckets. Remember that a backup is only as good as your ability to restore it and automatic backups should be tested often. You might also want to periodically delete old backups that you don’t need, but this is optional and could be hasty. Redundancy can help ensure better data integrity for backups, but it’s at the cost of disk space and some network bandwidth… and you have to keep backups safe!

#!/bin/bash
# Asher Bond 2010
# http://www.asherbond.com/blog/2010/09/15/service-oriented-backups-from-ec2-to-s3/
# backup-scripts.sh
# backup scripts every hour
# slightly tested on Debian Lenny
# put this in your /etc/cron.hourly

# no trailing slashes
local_backup_dir='/var/backups';
remote_backup_dir='/mnt/backups.asherbond.com';

# script directories to recursively back up
script_dirs='/etc'

# learn the date in rfc-3339 format
date=`date --rfc-3339=seconds | cut -d: -f1 | tr ' ' '-'`;

hostname=`hostname`;

file_prefix="$hostname-backup-scripts-";

# files look like: myhostname-backup-scripts-YYYY-MM-DD-HH.tar.gz when they're done
filename="$file_prefix$date.tar";

cd $local_backup_dir

# delete any local backups older than 7 days
echo "Deleting backups older than 7 days..."
find . -type f -ctime +7 -name "$file_prefix*.tar.gz" -exec rm -f {} \;

echo "Archiving files..."
tar -cvf $filename $script_dirs
gzip $filename

# mount s3 backup bit bucket using FUSE
# http://www.asherbond.com/blog/2010/09/14/mount-an-amazon-s3-bit-bucket-as-a-drive-in-unix-using-fuse/
/etc/asher-bond-cloud/s3-mount.sh backups.asherbond.com

# copy to remote bit bucket
echo "Copying backup to Amazon S3 bit bucket..."
cp $filename.gz $remote_backup_dir

# unmount fuse when done
echo "Dismounting from S3, what a trusty workhorse..."
sleep 30 && umount fuse

echo "FIN."

The output will look something like this:

Archiving Files…
/etc/
/etc/mysql/
/etc/mysql/debian.cnf
/etc/mysql/my.cnf
/etc/asher-bond-cloud/
/etc/asher-bond-cloud/loadavg.py
/etc/asher-bond-cloud/backup-scripts.sh
/etc/asher-bond-cloud/s3-mount.sh
/etc/etc/etc/etc/lol
Deleting backups older than 7 days…
Copying backup to Amazon S3 bit bucket…
Getting object list from S3 …
Validating cache …
Setup complete
Dismounting from S3, what a trusty workhorse.
Fin.

It’s a long way down if your head is in the CLOUD.
- Asher Bond

You can put this in your startup scripts, but I just run mine when I want to upload files to S3 for content distribution or for backup. A couple of my backup scripts invoke this script, then umount fuse when they’re done backing things up to s3. It hasn’t fully been tested yet, but let me know how it works out. Remember to store your AWS credentials in a safe place where only trusted people can read them. It’s also a good idea to expire and rotate them frequently.

#!/bin/bash
# Asher Bond 2010
# http://www.asherbond.com/blog/2010/09/14/mount-an-amazon-s3-bit-bucket-as-a-drive-in-unix-using-fuse/
#
# USAGE: s3-mount.sh bucket-name [/mnt/point/optional/if/different]
#
# mount s3 using s3-simple-fuse
# This is released under GPL
# you might need to:
# See http://code.google.com/p/s3-simple-fuse/ for s3-simple-fuse
# apt-get install python-fuse
# apt-get install python-dateutil
# apt-get install python-boto

# if you don't specify a mount point it just assumes /mnt/bit-bucket-name
function s3-mount ()
{

        if [ "$#" == 1 ] ; then
                # allow for custom /mnt/points
                mnt="/mnt/$1"
        else    mnt="$2"
        fi

        # keep this safe
        aws_access_key_id='AKUIAS7YOURMOM8XUTA4GA'
        aws_secret_access_key='sa09idontask2hsdfkjh34tnotrealna5p8'

        mkdir $mnt
        s3-simple-fuse $mnt -o AWS_ACCESS_KEY_ID=$aws_access_key_id,AWS_SECRET_ACCESS_KEY=$aws_secret_access_key,bucket=$1

}

s3-mount $1 $2

# to-do: mount google storage
# http://code.google.com/apis/storage/
#function google-storage-mount ()
#{
#}

Now with cloud computing you can delete entire computers with a touch of a button! So what if it happens by accident? No problem, you have a backup… right? Here is a script you may find useful for when you want to set instances up to automatically be backed up by a central server (or you can have them back themselves up individually, but don’t copy your AWS credentials to somewhere unsafe!)

Data Integrity Prerequisites and Other Considerations

You probably should lock your DBMS first, to avoid data corruption. If certain files are moved around while users are writing to the MYSQL or Oracle or some other database… well let’s just say it’s possible that the DBMS could get confused. The script will attempt to do this if you are using MYSQL, but you will need your mysql account information. Remember to back things up in a way you can restore them. A backup is only as good as one’s ability to restore it. Remember to store this script and all your credentials in a safe place. (chmod 700 is probably also a good idea). This is intended for backing up small instances, but you might use EBS for persistant storage, take EBS snapshots, and make copies if your instance is larger than 10 GB.

Account Credentials Needed

You might need to access your Account credentials page via aws.amazon.com. You’ll need the pem keys and the X.509 certs, etc.

Once you have all that, just edit and put this script into your cron.daily or schedule it for a good time [:

#!/usr/bin/python
# Asher Bond 2010
# Backup your EC2 instances to S3 every day
# an enhanced script based on backup.py from Paul Kenjora of Aware Labs
# http://blog.awarelabs.com/2009/painless-amazon-ec2-backup/

# apt-get install python-mysqldb if you don't have this, but
# it's not needed if you don't have MYSQL or a DBMS.
import MySQLdb

# you will need this to make system calls
import os

from datetime import date

# get this from aws.amazon.com (under security credentials you can find real credentials to replace the examples)
pem_file = '/root/.ssh/pk-APKAI6XFAKE1RVFIDNDA.pem'
cert_file = '/root/.ssh/cert-1J7DNJST6ET0YERMOMSREAL897B26UMS.pem'
user_id = '163265130428'
platform = 'i386'
bucket = 'backups.asherbond.com'

access_key = 'AKIAJ4QMQYJ5MXUTA4GA'
secret_key = '1WQjBA8i45CW3mnzzB/59jAFTuPl32d6u1YDgHvF'
ec2_path = '/etc/ec2/bin/' #use trailing slash

# it's a good idea to lock your DBMS so that people aren't writing to your DBMS files while they snapshot is taken.
# again, you can comment this whole database section out if you don't have a DBMS on the instance you're imaging.
conn = MySQLdb.connect (host = "localhost",
                        user = "backup",
                        passwd = "ordie",
                        db = "mysql")

cursor = conn.cursor ()
cursor.execute ("FLUSH TABLES WITH READ LOCK")
cursor.close ()

# 3 steps to back up the files to S3
days = ('monday', 'tuesday', 'wednesday', 'thursday', 'friday', 'saturday', 'sunday')
manifest = days[date.today().weekday()]

# remove some files
step_1 = ‘nice rm -f /mnt/%s*’ % (manifest,)

# bundle whatever’s here into an image
step_2 = ‘nice %sec2-bundle-vol -p %s -d /mnt -k %s -c %s -u %s -r %s’ % (ec2_path, manifest, pem_file, cert_file, user_id, platform)

# unlock the DBMS
cursor = conn.cursor ()
cursor.execute (“UNLOCK TABLES”)
cursor.close ()

# close this script’s connection to the dbms
conn.close ()

# upload the bundle to s3
step_3 = ‘nice %sec2-upload-bundle -b %s -m /mnt/%s.manifest.xml -a %s -s %s’ % (ec2_path, bucket, manifest, access_key, secret_key)

print step_1
os.system(step_1)
print step_2
os.system(step_2)
print step_3
os.system(step_3)

You should expect the process to take about 30 min on an m1.small, but t1.mini or large instances may back up faster (unless of course they have more stuff on disk, lol.)

Here is what my output looks like:

asherbond:~$ /etc/cron.daily/backup.py
Copying / into the image file /mnt/thursday...
Excluding:
/sys
/proc/bus/usb
/proc
/dev/pts
/dev
/media
/mnt
/proc
/sys
/mnt/thursday
/mnt/img-mnt
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied, 0.001796 s, 584 MB/s
mke2fs 1.41.3 (12-Oct-2008)
cBundling image file...
Splitting /mnt/thursday.tar.gz.enc...
Created thursday.part.000
Created thursday.part.001
Created thursday.part.002
Created thursday.part.003
Created thursday.part.004
.... etc ....
Created thursday.part.150
Generating digests for each part...
Digests generated.
Creating bundle manifest...
ec2-bundle-vol complete.
Uploading bundled image parts to the S3 bucket backups.asherbond.com ...
Uploaded thursday.part.000
Uploaded thursday.part.001
Uploaded thursday.part.002
Uploaded thursday.part.003
.... etc ....
Uploaded thursday.part.150
Uploading manifest ...
Uploaded manifest.
Bundle upload completed.

Be sure to log into your S3 bit bucket to make sure the backups are there. You should probably also restore each backup to a test instance and see if things look right.

I got switched to a Xen VPS (debian linux) recently and discovered that some applications couldn’t write to /dev/null. It turns out that /dev/null was actually only readable and writable by root. I logged in as root and changed the permissions so that /dev/null was readable and writable by all.

chmod 666 /dev/null

This solved the problem.

I was still having trouble with some shell accounts, though, because no users could access screen (virtual terminals). The error message was that no PTYs were available. I solved this problem by giving these users access to the /dev/ptmx directory. This directory was initially only readable, writable, and executable by root, but I made it so that anyone in the admin group could read write and execute this directory.

chown root:adm /dev/ptmx

chmod 770 /dev/ptmx

If you want any user to be able to use screen, you will have to do this:

chmod 777 /dev/ptmx