RSync Files to a Unix/BSD Backup Device from your Mac Laptop

My photo-taking workflow while on vacation usually involves taking a lot of photos daily, dumping them to a laptop, processing, then backing them up once I have returned home.
Previously, I accomplished this manually using BeyondCompare for Windows, as that would run on Windows Home Server.
Since moving to ZFS-based storage, however, this is no longer an option as BeyondCompare only has a Linux client (nothing for Unix/BSD).
There are other ways to get around this:

  • SSHFS and Meld – Complicated, somewhat bloated, but great BeyondCompare alternative
  • *Commander Utilities – Midnight Commander derivatives can accomplish similar tasks using the ctrl+x,d shortcut
  • Rsync – typically installed by default, easy to script

I chose Rsync as I wanted something more automated, but I do find myself using Midnight Commander from time-to-time to simply “get things done” when syncing files other than my images.

Here’s how I did it:

rsync -a -e ssh /volumes/PICTURES/ 'username@mymac:/Volumes/BIGRAID/'

Let’s break this down into smaller pieces:

rsync – this is the command that will do our heavy lifting and file comparison

-a – archive mode

-e – specify an RSH replacement

ssh  – use SSH

/volumes/PICTURES/ – this specifies the “Volumes” folder on my Mac, and the “PICTURES” drive within it. Replace this with the location of your items to backup

 – note the use of single quotes here. We’re using these in case there are spaces in the folder names, and we could have done the same above.

username@mymac – We’re logging on to the host “mymac” with the username “username”. You’ll probably want to change these. I use a hostname here, but you could just as easily use an IP address if you use static IP addresses.

:/volumes/BIGRAID/ – the colon denotes a subfolder on the server we are backing up to, and /volumes/BIGRAID in this case refers to a ZFS pool called “BIGRAID”.

Do you have a similar backup strategy for BSD/Unix targets that you would like to share?

Clear All ZFS Snapshots

If you’ve been running snapshots for a while and have already backed them up, you might occasional need to delete all zfs snapshots for your pool.
Typically, you’d do this as part of your backup script, assuming that they have been written correctly.

First, to find the used snapshot space, run this command:
zfs list -o space
This will give you a detailed readout of your pools and snapshot space used.

Here’s my script to wipe ZFS snap shots, but I am certainly open to suggestion:
zfs list -H -o name -t snapshot | xargs -n1 zfs destroy
Again, caution is needed as this will remove ALL SNAPS from your pools.

ZFS Build Checklist

I’ve decided to replace the Windows Home Server Vail server with something capable of handling newer builds of ZFS and the inherent deduploication.

Here’s a quick kit list and build diary I’ll try to keep up-to-date as I go along.

Kit:

  • Dell Perc6i – this is essentially a port multiplier. I scored it from eBay on the cheap, though it was delivered from Israel, took awhile, and had neither cables nor mounting bracket.
  • OCZ RevoDrive 120GB – Though the RAID controller on this card is not supported in Linux/Solaris, the drives show up as two separate devices as long as you make sure to put it in the right PCIe slot. That means it’s perfect for both ZIL (log) and L2ARC (cache).
  • 2x Intel 80GB X25-M SSDs – these will house the virtual machine files to be deduped. Very reliable drives, and though they might not be the fastest in terms of writes, the speeds are relatively constant which is quite handy compared to solutions that attempt compression like SandForce controllers. ZFS will take care of that, thanks.
  • (IN TRANSIT) 2x Dual Port 1gbit Intel PCIe NICs – I’ll use these for the direct connection to the virtual machine host. Currently one link is used, but when reading from the SSD drives the line is saturated.
  • (IN TRANSIT) 32 Pin SAS Controller To 4x SATA HDD Serial Cable Cord – This is needed to plug in 8 drives to the LSI controller.
  • 5x 1.5TB Seagate hard drives – These will be the bread-and-butter storage running in RAID-Z2 (similar to RAID 6).
  • 3x 3TB Seagate hard drives – These might simply be a large headache, but the plan was to have an extra 3TB RAID-Z2 for backups in another machine. Unfortunately there seem to be issues with drives that are 4k presenting themselves as 512b. I may be able to get around this by hacking or waiting as they become more popular. For now 2 of them are in software RAID1 on a Windows 7 host, and the other remains in the external USB 3 case and is used as a backup drive.
  • NetGear GS108T Switch – A cheap VLAN-capable switch should I decide to use more than 2 bonded ports (I doubt it), currently running the lab.

EMC Buys Into Cloud Computing with Pi Acquisition

From TechTarget:

EMC Corp. has acquired a startup still in stealth called Pi Corp. in
an all-cash transaction for an undisclosed amount.

Pi Corp. founder and CEO Paul Maritz will join EMC as president and
general manager of EMC’s newly created best cloud storage infrastructure and
Services Division, reporting directly to EMC CEO Joe Tucci. The new
division will also include the EMC Fortress SaaS infrastructure, the
Mozy online backup service and “other upcoming EMC cloud
infrastructure systems and software offerings under development.”

This is interesting to me as I see it as the “new next big thing”.

Currently the only people really loud in this space are Google and Amazon.

I highly doubt we will see much come out of either party until the very soonest Q2 2008 (Mozy update), but expect a product by year-end.

Disco Disc Burner App Public Beta for Mac OS X

Why is it that I always find myself ponying up the small amounts of cash it takes to purchase Mac shareware applications? Black fire and a disco ball? I gotta have it.

Today I fell victim to MacZot‘s current promotion (ends in 23 hours!) that saw me gobbling up the public beta of the Disco CD/DVD burning application for Mac OS X, and boy am I content with this lil 700KB number.

Firstly, the hype: when you purchase the beta at the silly-low price, you also get the opportunity to send a free license to someone else. So here goes – the first person to leave an intelligent comment gets the license! (I’ll need a real email address) This is a great marketing idea that see people not only using the beta of a terrifically designed piece of software, but also spreading the word to others so they can try it as well. Good show.

Disco Windows

Next, the features: Disco copies discs in two clicks. Say that ten times fast. Disco can span CDs and DVDs automagically. They call it Spandex, I call it dead sexy. This is extremely handy when backing up iPhoto or iTunes, trust me. Disco can burn VIDEO_TS folders by just dragging them onto the application, which makes it great for burning your DVD backups. Disco has realtime 3D interactive smoke that actually wafts up from the application as you burn discs. If you blow into your microphone, the smoke will dissipate, and if you move your cursor around the smoke itself, it reportedly moves accordingly – but I didn’t get to see this on the low-end test box. Disco has an amazingly simplistic interface that is very easy to follow as it prompts you at every step. Last, but certainly not the least in terms of features, and should really be in caps: Disco remembers every single burn and keeps a catalog of the burn sessions using what it calls “discography” so that you can easily do things like print labels, disk jackets, or simply keep handy for when you need that dot release or the pre-altered code that you can never find. Let Disco be your personal sherpa.

Disco runs very well for a beta application. I have yet to see it crash under Panther, Tiger, or even Leopard (WWDC or the 9a ADC build). Oh, and it worked fine in Tiger server as well. Unfortunately the test box is a 1st generation Mac Mini (PPC), so there was no “smoke” to speak of (anyone have screenshots?), so that will need to be tested later. I burned some mix audio CDs, a backup copy of the disk image for Disco public beta itself, an ISO image of Ubuntu Edgy Eft Beta, and a bunch of Gentoo releases, for good measure. No hiccups. I erased a very old scratched CDRW. I even inserted a scratched blank CDR to see what would happen… Well it wouldn’t burn correctly but Disco kept on keepin’ on – like some sort of Disco inferno – no bouncing beachballs of death here. And you call this a beta? I expected flames, not smoke. No crash and burn here.

picture-1.pngpicture-2.pngpicture-3.pngpicture-4.pngpicture-5.pngpicture-6.pngpicture-7.pngpicture-8.pngpicture-9.pngpicture-10.pngpicture-10.pngpicture-11.png

In these screenshots you see the Disco app in full swing, from installation to a test burn, and a view of all the windows it provides. I really like the design of this app, and it was well worth the 15$ pricetag. If you’re new to MacZot, and need a referrer, feel free to use “blandname”. Now go write some optical media and dance around like a schoolgirl.

Oh and on the topic of Disco, and in the name of Halloween here’s a joke:

“Why didn’t the skeleton dance at the disco?”

“He had no body to dance with”

9 Great, Free Applications that Work with Vista

This is a list of my favorite Windows applications that I honestly couldn’t live without that also work on Microsoft Windows Vista RC as well, much to my surprise. All of the software is free to download and use, and in fact most of them are open source.

ConTEXT – ConTEXT is a free and lightweight editor for programming or can be used as a notepad replacement. ConTEXT supports find and replace in multiple files at once so changing one method in loads of files is no longer an issue. Works in Vista with no problems whatsoever.

Synergy – Synergy is like a software KVM, but only shares keyboard and mouse capabilities. Synergy is multiplatform, and I currently use it to have my keyboard and mouse work in Ubuntu, Vista, and my Mac Mini running Leopard with no problem at all! This way I get to use my favorite keyboard and mouse and get to free up some space on my desk at the same time.

VLC – VLC has been my favorite media player for years now. VLC comes with most of the codecs you will need to watch videos on your PC already. VLC is very lightweight, and JUST WORKS, something that can’t be said about many media players. The only issue with VLC in Vista is that it turns Aero Glass off while it is playing.

FileZilla – I use FileZilla to interface with clients that still haven’t moved to SCP. FileZilla is an open source FTP client that gets the job done, supporting drag and drop, SSL, and NAT to NAT connections. The only thing it is missing is FXP support, but that’s not really a big deal in my case. Works fine in Vista with no problems at all.

IMGBurn – I love IMGBurn. This is hands-down the easiest way to burn .IMG, .ISO and BIN/CUE disk images to backup CDs or DVDs. Free, open source, and awesome. Works a treat in Microsoft Vista Beta 2 as well.

WinSnap – I use WinSnap to make many of the screen shots you see on blandname. WinSnap supports full screen and windowed screen shots, and also allows for rotation and drop shadows if you feel the need.

Electric Sheep – I often refer to Electric Sheep as “the best screensaver ever”, but truth be told, it’s really a collection of computer-generated screensavers that allow users to vote on them using a Digg-like system. On Windows, Electric Sheep uses bit torrent to transfer the sheep data. Again, working just fine in Vista!

Xming – Xming is my prefered interface to remote Linux boxes. Xming is for Linux what RDP is for Windows – you get a local X server and acceleration that displays data from remote applications running on Linux machines. Tested more than a few times to a remote Ubuntu computer, and one Gentoo box with no caveats.

WinSCP3 – SCP is now my preferred file transfer method. Luckily my favorite client also works on Vista, or I may not have used it at all. Much like FileZilla, WinSCP3 has a very simple, streamlined interface that is feature-rich and gets the job done, even on Vista.

So there you have it: 9 free applications I couldn’t live without that work just fine on Vista, and make it easy for me to do my day-to-day tasks. Hopefully someday this list will include F-Spot and Amarok, but I’m happy with this as a start considering neither of them work on Windows XP SP2 either!