Getting Started with Salt Stack Windows Minions

This how-to on Windows Salt minions will just scratch the surface of the power behind Salt, and will cover remote execution, installation and management.

Please note: I’m new to Salt, and I’m a recovering Windows syasdamin from the 2000-2013 era. This means I probably have a bunch of out-moded ways of working with Windows.

To this end, this first question would be: why? Active Directory and associated policies do a good job of managing Wintel already. I’ve had luck in the past managing hundreds, thousands of servers, desktops and laptops using the tools Microsoft provides without too much issue, and I’ve heard things only got better since I left for greener pastures.

The easy answer is that many organisations may now find themselves with a combination of Linux and Windows servers, Windows and macOS workstations, coupled with the odd BSD system here and there, Linux desktops and Unix servers.

What Salt allows for is a management platform that covers all of these platforms with a common language that is easy to read and hack away at (YAML) and very extensible using a programming language with a relatively gentle learning curve (Python).

In my case, most things running at home are either running OSX or some version of Linux/BSD, but there are a few of oddball Windows 10 computers doing things Microsoft-based computers do pretty well; gaming, spreadsheets, touchscreen-enabled full-desktops.

The idea of managing Windows computers with Salt came to me pretty late into the journey through the docs, but hit me like a freight-train as these are often the computers that require the most maintenance for me personally, meaning reinstalls, reconfiguration and general “care and feeding”. Having gone down the path of running my own Active Directory at home a few times and finally settling on a couple of Apple “servers”, this seemed like a pretty good work-around that wouldn’t introduce much pain.

The first caveat, however, was that I was unable to get Windows-based Salt minions to be accepted by OSX-based Salt masters. Not the end of the world, but it meant running Docker or a Vagrant box to handle the Windows 10 computers. I ended up simply installing Salt on an Ubuntu-based storage server I already had running that had recently been upgraded to Xenial instead, but I’m most likely going to be revisiting this architecture in the long-term.

Now that we have that out of the way, here’s the general architecture of what I’ll be going over:

Windows Salt Minions

A simple Windows Salt Minions example with one Salt Stack master running Ubuntu Linux

The Nitty Gritty – Deploying Salt Minion Services to All of Your Windows Nodes

I’ll come right out and say that I cheated when it came to installing the Minion service on the nodes at home in that I already had Chocolatey installed, and that I simply used the default salt hostname for the master by adding it the the static DNS entries on my DNS server.

This last bit, the part about the DNS server, might be a luxury depending on your install location. I also can’t expect that Chocolatey is installed everywhere, but just quickly, here’s how that scenario works:

  1. Install Salt Master and start the service on a computer in your network
  2. Ensure that there is a DHCP reservation for the IP of this node
  3. Add a DNS entry salt for that DHCP-reserved IP
  4. Install the Salt minions via choco install -y saltminion on the Windows hosts
  5. Accept the keys on the Linux master via salt-key -A

Should you have to go about this a different way, you can point an alias (CNAME) to the hostname in your network, OR add a manual host entry to your nodes (I’ve used Group Policy to do this in the past, with network scripts), or finally, you can simply change the Salt Minion config to point to the new hostname.

As for the deployment of the Salt Minion service and binaries, you could do this over RDP for a few hosts, Group Policy, or via psexec:

Salt-Minion-2016.11.5-x64-Setup.exe /S /master=SALTMASTER

This snippet will install the Salt Minion service, configure it to point at `SALTMASTER` (replace this with the hostname you’d prefer) then start the service. Note: as mentioned above, I haven’t tested this myself!

The most up-to-date Salt Minion binaries can be found here: https://docs.saltstack.com/en/latest/topics/installation/windows.html Should the version change, you’ll want to update that one-liner as well.

Accepting the Minions

Assuming you’ve got your minions now sending requests to the Salt master node, you need to approve them.

On the master node running Linux, this can be done like so:

sudo salt-key -A

You’ll be greeted with a list of nodes to be accepted, and you can take a moment to note the names and accept them:

user@ubuntu:/srv/salt# sudo salt-key -A
[sudo] password for user: 
The following keys are going to be accepted:
Unaccepted Keys:
WINDOWSGAMES
Proceed? [n/Y] 
Key for minion WINDOWSGAMES accepted.
user@ubuntu:/srv/salt#

Testing the Minions

Now that you have minions installed and working, let’s run a quick test that proves we can run remote commands on them. Note that this is just scratching the surface, and that “push commands” are just one way of working with Salt Stack, you can also have the nodes “pull” from the Salt master on a regular basis.

Sample run command:

sudo salt -G 'os:windows` cmd.run 'dir'

This should give you a directory listing on each of the nodes, like so:

WINDOWSGAMES:
 Volume in drive C has no label.
 Volume Serial Number is 3FF7-A973
 
 Directory of C:\Windows\system32\config\systemprofile
 
 07/16/2016 04:47 AM <DIR> .
 07/16/2016 04:47 AM <DIR> ..
 07/16/2016 04:47 AM <DIR> AppData
 0 File(s) 0 bytes
 3 Dir(s) 66,483,965,952 bytes free

Package Management on the Master Node

You’ve got a few options for package management for Windows Salt minions:

  • Chocolatey packages
  • Cygwin Packages
  • Master-shared packages (WinRepo)

Personally, I’m currently using Chocolatey package management because it’s what I’m used to, but I certainly do still use Cygwin for development purposes from time-to-time since Windows Bash mode has still yet to become commonplace. The third option, however, is probably the most scalable, and more user-friendly.

To get started with Windows repositories on the Linux Salt master, you’ll need to run:

sudo salt-run winrepo.update_git_repos

This means that you now have a repository for Windows packages on the Master node, but also note that these don’t come with binaries – they will fetch and send them as needed. I can see this as both good and bad, in a corporate setting you’ll want to create your own SLS files and point them to known-good sources.

To install a package on all of your Windows remote nodes using the Linux Salt master repository:

sudo salt -G 'os:windows' package.install 'firefox'

Note that if ever you want to specify a specific action for a node in particular, that’s done like so:

sudo salt 'WINDOWSGAMES' pkg.install 'firefox'

I’m trying to follow the “pets vs cattle” paradigm here, though you can also use -G to specify groups of servers/workstations/laptops which can overlap in order to manage things in a more granular approach (bit of a pun here, the G is for Grains).

Useful Commands for Windows Salt Minions

In no particular order, here’s a quick dump of the commands I used during my first weekend managing Salt minions at home running Windows 10.

Show installed applications:

sudo salt -G 'os:windows' pkg.list_pkgs

List current Salt minion keys:

sudo salt-key -L

Delete an old Salt minion key:

sudo salt-key -d WINDOWSGAMES

Check if a service is available:

sudo salt -G 'os:windows' service.available 'salt-minion'

Restart a service:

sudo salt -G 'os:windows' service.restart 'salt-minion'

Flashing an AMJPad with QMK

Thanks to the hard work of /u/TerryMatthews on Reddit, there’s now a port of QMK for the AMJ numpad!

In order to get QMK on the number pad, however, a bit of work will need to be done, though none of it is insurmountable.

First off, I’ve compiled the AMJ Pad QMK default layout hex file for you to download.

The default layout is set up like this:

 /* Keymap _BL: (Base Layer) Default Layer
 * ,-------------------.
 * |Esc |TAB |BS  | =  |
 * |----|----|----|----|
 * | NL | /  | *  | -  |
 * |----|----|----|----|
 * | 7  | 8  | 9  |    |
 * |----|----|----| +  |
 * | 4  | 5  | 6  |    |
 * |----|----|----|----|
 * | 1  | 2  | 3  |    |
 * |----|----|----| En |
 * | 0       |./FN|    |
 * `-------------------'
 */

 /* Keymap _FL: Function Layer
 * ,-------------------.
 * |Esc |TAB |BS  | =  |
 * |----|----|----|----|
 * | NL | /  | *  | -  |
 * |----|----|----|----|
 * | 7  | 8  | 9  |    |
 * |----|----|----|RST |
 * | 4  | 5  | 6  |    |
 * |----|----|----|----|
 * | 1  | 2  | 3  |    |
 * |----|----|----| En |
 * | 0       |./FN|    |
 * `-------------------'
 */

Legend:

  • Esc – escape
  • BS – backspace
  • NL – Number lock
  • En – Enter
  • Fn – Function key, hold this and press keys in the second layout to activate them
  • RST – Reset – put the board in DFU mode

Programming the board can be done like other Atmel PCBs – just remember that this is an Atmel ATmega32u4.

As such, we can follow The Van Keyboards programming guide, which covers Linux, Windows and Mac.

On a macOS system, for example:

sudo dfu-programmer atmega32u4 erase --force
sudo dfu-programmer atmega32u4 flash ~/Downloads/amjpad_default.hex
sudo dfu-programmer atmega32u4 reset

Building Zeal60 Keymaps on OSX

Before starting, this guide requires Homebrew, which you can install from here: https://brew.sh

Open Terminal, and paste in the following. It will take a while to build, but you should be able to get it all done in one fell swoop:

cd
git clone https://github.com/Wilba6582/qmk_firmware.git
cd qmk_firmware
git branch zeal60
cd keyboards/zeal60
git submodule update --init --recursive
brew tap osx-cross/avr
brew install avr-gcc avr-libc dfu-programmer

You can then make your keymap, like so:

  • make keymap=default
  • make keymap=poker2_ansi
  • make keymap=poker2_iso
  • make keymap=hhkb_ansi

You can run them all if you want, but you’ll probably only need one ūüôā

Once you have your firmware, you can flash like so (for HHKB ANSI):

sudo dfu-programmer atmega32u4 erase --force
sudo dfu-programmer atmega32u4 flash hhkb_ansi.hex
sudo dfu-programmer atmega32u4 reset

Getting Started with the ZealPC Zeal60 Keyboard PCB

Recently I picked up a Zeal 60 printed circuit board from BC-based ZealPC, and I couldn’t be happier!

The board supports a few layouts for now, though it’s open source, and you can either fork it to add your own, or add a pull request to share with the community.

The first thing you’ll want to do after getting a Zeal60 PCB is to flash it with the “default.hex” file. I used Windows for this, though you can achieve similar results on Linux and MacOS (OSX).

Initial Downloads

  1. Download Java Runtime Environment (JRE) here: http://www.oracle.com/technetwork/java/javase/downloads/jre8-downloads-2133155.html
  2. Download the Atmel flashing utility “FLIP”:¬†http://www.atmel.com/tools/flip.aspx
  3. Grab the most recent Zeal 60 keyboard zip: https://cdn.shopify.com/s/files/1/0490/7329/files/zeal60_v0_3.zip
  4. Unzip the zeal60_v0_3.zip file to your desktop

Install the Driver

  1. Plug in your Zeal60 to your computer via USB
  2. Hit “Windows” and “r” keys together, then type “devmgmt.msc”
  3. Right click the device with the exclamation point next to it
  4. Click “browse for driver”
  5. Navigate to C:\Program Files\FLIP\Drivers
  6. Hit “next”

Flashing the Board

  1. Install the driver –
  2. If you’ve not started assembling anything yet, press the golden button between the “A” and “S” pads on the board (if you’ve already assembled it, unplug it and hold the “space” and “esc” keys while plugging in the USB cable)
  3. Open FLIP
  4. In FLIP, hit connect to the device, and select “ATMega32U4”
  5. Hit “ctrl” + “U” keys to connect to the Zeal 60 PCB
  6. Hit “ctrl” + “L” to load a file
  7. Select the “default.hex” file in the Zeal 60 folder you extracted to your desktop
  8. On the left-hand window, click “Run”
  9. Once completed, then click “Start Application” in the right window

You’re all set!

Top 10 Virtual Appliances, Revisited

It’s been almost 4 years since I’ve rounded up VMs used on a daily basis, so it’s high time I take another kick at the can and make an update list.

My workflows have changed quite a bit over the years, with more focus being on the Windows side of things. That said, I havent stopped using Linux and still have a keen interest in both storage and management, which should be reflected here.

FreeBSD 9 – I’ve made the switch to this as my go-to server OS. The jails functionality and ports collections are amazing! This could run many of the functions listed herein, but at the very least is a great ZFSv28 test box for the uninitiated.

Astaro – ¬†I’m still using Astaro after all these years, and Sophos purchasing them has not stopped the love. By far the easiest way to start using Squid, Quagga and OpenVPN.

GNS3 Workbench – I use this for testing Cisco configurations on my way to certification. Load up an IOS image, configure, test away!

Nexenta Community Edition – My ZFS primer was done a few years ago using Nexenta, and it is still the easiest way to get into ZFS, so it deserves the nod. The first time you see the¬†speedometers you’ll be in love.

Solaris 11 11/11 – For newer versions of ZFS, you’re stuck with Solaris 11.11.11. You can download this for free, but won’t be able to get support and updates without a license, so I wouldn’t consider it production-ready.

Bactrack 5 – Time to test your wifi security. I’d recommend plugging an Alfa USB wifi device into ESX, sharing the device with the VM and scanning your access point in order to do quick audits.

Windows Server 2008 R2 – Not free, per se, but a good trial that should be enough to get you going on your road to certification. I use the Core install for DHCP and DNS when Windows integration is important.

Citrix XenApp – You’ll need to convert this one, but combined with the developer license, you’ll get 2 concurrent users, include the web access gateway. Click this for a tutorial on getting a hostgator coupon or the developer license, which has always been a bit of a pain.

Plop Boot Manager – Great ISO for booting and testing USB sticks.

Amahi – Easy as pie mDNS and uPnP autodiscovery. I’ve written about this being a poor Windows Home Server replacement before, but to be fair they get an awful lot right at this point in time.

OSX Lion – A $20 purchase, and well worth it! Follow the guide here to get it all working in ESXi

Ubuntu LTS¬†–¬†Ubuntu is currently the most popular Linux distribution, can run a wealth of software. Finally took over OpenSuSE as my go-to distribution. The only thing I would mention is that unity does not work so well in ESXi, and if you require the whole desktop experience, you might be better off with Xubuntu or Mint.

 

Top 10 Chrome Applications I Can’t Live Without

I’ve been trying to become as agile as possible when it comes to personal data in order to be able to run and test any platform of my choosing. To that end, I usually resort to web applications.
That said, Chrome and its open source cousin Chromium can only be run on many (but not all) platforms:
CHROME
  • Windows XP/Vista/2008/7/8
  • Mac OS 10.5.6+
  • Flavours of Linux – Debian, RedHat, OpenSuSE, Ubuntu, Fedora, CentOS
  • ChromeOS
CHROMIUM
  • All of the above
  • BSD: FreeBSD, OpenBSD, NetBSD, DragonflyBSD
  • Solaris and Open Indiana
  • ChromiumOS
  • Gentoo
Unfortunately one OS left out is HP-UX 11i. This is typically a server install, and while I know it’s no excuse, no longer has the mindshare it once enjoyed.
On to the apps!
  • Google Music – free sync for up to 20000 songs. My iTunes replacement that allows offline play for iOS and Android
  • Adblock – A must. I block adverts on everything.
  • Google Reader – It took a very long time for me to get used to the way Google reader works, but it might actually be the best there is at the moment especially considering the aggregation of many feeds into one.
  • Mint – Personal finance application I could not live without.
  • IMO – Goodbye Adium, Pidgin and MSN Messenger! IMO.im is not only a multi-instance web chat client that runs everywhere, it also runs on iOS!
  • Kindle Cloud Reader – Never lose your place. The web client knows where you were on your Kindle, iOS device and syncs it up for you.
  • Google Finance – For stock checking and even watching mutuals. Find out when the next dividend is, sort companies by financials and even display candle graphs.
  • Aviary – Has just surpassed Picnik as my only photo editor, and is now also integrated with flickr. Note that there are many Aviary editors ranging from vector to audio and even video.
  • Netflix Instant Queue – I’m sure you’ve heard of this, but did you also realize that it will resume from PS3, XBOX 360, iPhone/iPad on the web? Outside of the US, we’re not able to use “Instant Queue” but this app brings it back.
  • Offline Gmail – Spotty wifi? Don’t worry, Gmail offline has you covered.
  • Google Voice – This doesn’t get as much hype as it should, but is a great app that can not only make calls for you, but also send SMS.Soon they will make loans app and pay day loans apps.

 

Clear All ZFS Snapshots

If you’ve been running snapshots for a while and have already backed them up, you might occasional need to delete all zfs snapshots for your pool.
Typically, you’d do this as part of your backup script, assuming that they have been written correctly.

First, to find the used snapshot space, run this command:
zfs list -o space
This will give you a detailed readout of your pools and snapshot space used.

Here’s my script to wipe ZFS snap shots, but I am certainly open to suggestion:
zfs list -H -o name -t snapshot | xargs -n1 zfs destroy
Again, caution is needed as this will remove ALL SNAPS from your pools.

ZFS Build Checklist

I’ve decided to replace the Windows Home Server Vail server with something capable of handling newer builds of ZFS and the inherent deduploication.

Here’s a quick kit list and build diary I’ll try to keep up-to-date as I go along.

Kit:

  • Dell Perc6i – this is essentially a port multiplier. I scored it from eBay on the cheap, though it was delivered from Israel, took awhile, and had neither cables nor mounting bracket.
  • OCZ RevoDrive 120GB – Though the RAID controller on this card is not supported in Linux/Solaris, the drives show up as two separate devices as long as you make sure to put it in the right PCIe slot. That means it’s perfect for both ZIL (log) and L2ARC (cache).
  • 2x Intel 80GB X25-M SSDs – these will house the virtual machine files to be deduped. Very reliable drives, and though they might not be the fastest in terms of writes, the speeds are relatively constant which is quite handy compared to solutions that attempt compression like SandForce controllers. ZFS will take care of that, thanks.
  • (IN TRANSIT) 2x Dual Port 1gbit Intel PCIe NICs – I’ll use these for the direct connection to the virtual machine host. Currently one link is used, but when reading from the SSD drives the line is saturated.
  • (IN TRANSIT) 32 Pin SAS Controller To 4x SATA HDD Serial Cable Cord – This is needed to plug in 8 drives to the LSI controller.
  • 5x 1.5TB Seagate hard drives – These will be the bread-and-butter storage running in RAID-Z2 (similar to RAID 6).
  • 3x 3TB Seagate hard drives – These might simply be a large headache, but the plan was to have an extra 3TB RAID-Z2 for backups in another machine. Unfortunately there seem to be issues with drives that are 4k presenting themselves as 512b. I may be able to get around this by hacking or waiting as they become more popular. For now 2 of them are in software RAID1 on a Windows 7 host, and the other remains in the external USB 3 case and is used as a backup drive.
  • NetGear GS108T Switch – A cheap VLAN-capable switch should I decide to use more than 2 bonded ports (I doubt it), currently running the lab.

A Biased Review of Amahi Home Server

The PR-savvy folks at Amahi recently chimed in on the Vail-Fail fiasco by presenting Amahi as an alternative to the Windows Home Server (Vail) solution, and I thought I should give it a run for the money to see how it stacks up.

In short: not well at all.

First, the good things: when configured properly, Amahi offers DLNA/uPNP streaming and the ability to send h264 streams to iPod/iPad/iPhone devices. It also supports backups, disk spanning, remote access, dynamic DNS (*.yourhda.com) and a slew of other features that should have you salivating by now.

The bad? None of it works out of the box.

In order to set up an Amahi server, you must first complete a Fedora 12 install. That’s right, Fedora 12. Not 13 or 14, don’t be confused. Just like most open source software, Amahi suffers from circular dependency issues if you choose the wrong version and the wrong repository – be warned. (Note: if you want to use current versions of Fedora, make sure to change the repository to either f13.amahi.org or f14.amahi.org and realize that there are no plugins for either).

Fedora 12 is a relatively easy install, but when you’re manually setting IP configurations, you lose most of the WHS market in one fell swoop. Fail?

After the install has completed, you logon to Fedora and run the Amahi installer. You’re met with a logon screen. What username and password to use? Pick anything and you’ve just been made an admin. Security by obscurity or brainless UI design? I think it’s the latter.

This install will take almost as long as the Fedora build, which is counter-intuitive. Why not simply chain the install? Why not build a freaking fork that contains Amahi? I’m ranting here, but I find this bit incredibly odd, especially since Amahi has specific OS requirements.

OK, we’ve survived, we’ve realized that eth0 is the only card available to Amahi by now, and through process of elimination we’ve figured out which port Fedora has decided this is. We’ve realized the firewall has been disabled, we’ve entered the activation code and received an email letting us know that we now own http://im.yourhda.com

Huzzah.

Let’s start packing it full of media, eh? We’ll need disks for that, but they are in the case so we should be OK – let’s add a disk to Amahi and let the good times roll. Oh wait, you can’t do that. Why not? It needs to be done via commandline. OK, getting the sleeves rolled up is fun once in a while, disks added.

Let’s add some media to the disks. Done. the transfer speed is a good 10% faster than WHS, and 50% faster than Vail. Good news. But you have to use SCP to do it… The Samba sharing doesn’t actually work out of the box (fixed later in Fedora). More fail.

Alright, media is on the device, let’s play some. Pop on the TV, have a look for uPNP or DLNA devices. None. Hmm. Oh yeah, it’s not even added yet.

Civilization 5 Tactics and Strategies

This post will outline and detail strategies, tips, tricks, cheats and tactics for Sid Meier’s Civilization V (Civ5/CivV). Call it an FAQ of sorts.

Let’s get on with it, shall we?

Some quick observations:

  • Open borders allows you to surround an enemy, then attack. This works wonders later on.
  • The trebuchet seems a bit overpowered, and is definitely the best way to attack cities.
  • The easiest way I know of to skip the first intro movie is to simply rename “Civ5_Opening_Movie_en_US.wmv” to something else.
  • Alternatively, edit the “usersettings.ini” file found in “My Documents\My Games\Sid Meiers Civilization 5\”. Make sure to change “SkipIntroVideo=0” to “=1” on line 15.
  • For a copy of the English manual for Civ5, head here
  • How to turn on workers automatically keeping improvements
    In the My Documents folder go to the Civ V folder and then look for the “usersettings.ini” file.

    There is a setting called AutoWorkersDontReplace. Change the value from 0 to 1.