Migrate from AWS to a hybrid OSX environment

Leverage cheaper alternatives to AWS for OSX where possible

10 min read time

#aws #osx

Towards the end of 2020 AWS introduced new Mac OSX instance types. These require a Dedicated Host to run which are charged by the hour like most AWS services. Whilst a great addition to the AWS family, there are a number of aspects which make them a bit prohibitive to work with. One of these aspects is cost.

They are quite expensive to run, with the cheapest at the time of writing being the (outdated) M1 chip at $0.65/hr or $473.2/mo. That's almost the cost of a new modern Mac Mini every few months.

In addition, they are not always available and in our experience there have been a number of times where we have received an "Insufficient Capacity" error when trying to launch a new instance. This is not uncommon with AWS and can be frustrating when you are trying to get work done.

For one client one of their primary cost drivers was OSX dedicated hosts. The workload was isolated and did not depend on any other AWS services and we started to explore options outside of AWS. We settled on MacStadium which offered a 60% cost saving compared to AWS for the same spec instances. This post details the migration process to MacStadium, but the concepts should be appicable to any OSX bare metal hosting provider.

Why MacStadium?

MacStadium is a bare metal hosting provider that specializes in OSX hosting. They offer a number of different options for hosting OSX instances, including Mac Mini, Mac Pro and Mac Studio. They also offer a number of different configurations for each instance type, so you can choose the one that best fits your needs.

A quick comparison on cost compared to AWS:

Migration Process

For a successful migration we set out the following goals:

• Ensure the OSX instances in MacStadium are sufficiently locked down
• Maintain the CloudWatch monitoring and metrics in AWS which means ensuring the CloudWatch agent is installed and configured outside AWS
• Maintain the ability to issue commands from SSM using AWS and manage the servers in the same fleet
• Maintain a working AWS AMI for reverting back to AWS if quick scaling is required

This would allow a hybrid approach where our client does not need to change their current deployment and monitoring processes.

[!NOTE] This is to be used as a guide for configuring a Bare Metal OSX server and you would typically automate these steps using an OSX provisioning tool such as Ansible

Ensure the OSX instances in MacStadium are sufficiently locked down

The first step was to lock down the OSX instances in MacStadium. It's easy to forget the luxury of security groups and "built in" firewalls in modern cloud computing solutions

MacStadium, like most bare metal providers, provide an OSX host connected to the internet that you connect to remotely.

MacStadium do provide a managed Cisco firewall at additional cost and this would be the recommended approach for adding additional security. However there are some steps you can take yourself running standard Bare Metal servers to add some protection.

By default, you can connect to your instance either via SSH or using a VNC client with the remote desktop protocol.

You don’t want to leave it this open for long, so a few recommended steps are as follows:

Enable the Firewall

Go to System Settings -> Network -> Firewall -> Turn On

Enable Stealth Mode by clicking "Options" under the Turn On Firewall button.

Disable Unneeded Services

• Disable Location Services: System Settings -> Privacy And Security -> Location Services
• Remove unneeded Services from Sharing: System Settings -> Sharing
  • Usually all except Remote Management and Remote Login - These should be disabled by MacStadium already but just to be sure)
• Disable Apple Intelligence if you don’t intend to use this.

Disable Passwords for SSH

It is good practice to disable password authentication for SSH. This means that only users with a private key can access the instance.

There are many guides to enabling SSH key authentication and generating your keys.

Add the public key you will be connecting from to ~/.ssh/authorized_keys.

Update the SSH config file /etc/ssh/sshd_config to disable password authentication.

PasswordAuthentication no
ChallengeResponseAuthentication no
UsePAM no

Restart the SSH service to apply the changes.

sudo launchctl stop com.openssh.sshd
sudo launchctl start com.openssh.sshd

Test that you can login via SSH OK.

Disable Remote Management when not in use

If you aren’t intending to be connected to your servers all the time, you can disable Remote Management when not in use making the primary method of connecting via your SSH key. This will greatly reduce the attack vectors on the instance.

To make it easy to enable and disable, create a stop_remote_management.sh script with the following contents:

#!/bin/zsh
sudo /System/Library/CoreServices/RemoteManagement/ARDAgent.app/Contents/Resources/kickstart -deactivate

And a start_remote_management.sh script to re-start it.

#!/bin/zsh
sudo /System/Library/CoreServices/RemoteManagement/ARDAgent.app/Contents/Resources/kickstart -activate

Make both executable:

chmod +x stop_remote_management.sh start_remote_management.sh

Then via your SSH user you can start and stop the remote management service as needed.

Enable the Cloudwatch Agent

If you want to keep your external OSX servers logging metrics and logs the same way as the rest of your AWS infrastructure you will need to enable the Cloudwatch agent.

Download the agent:

curl -sL 'https://s3.amazonaws.com/amazoncloudwatch-agent/darwin/arm64/latest/amazon-cloudwatch-agent.pkg' -O\
    amazon-cloudwatch-agent.pkg

Install the agent:

sudo installer -pkg amazon-cloudwatch-agent.pkg -target /

As we can't use instance roles to authenticate to AWS, we need to create an IAM user.

You should attach the CloudWatchAgentServerPolicy and the a new custom policy which with the following permissions:

{
    "Statement": [
        {
            "Action": [
                "cloudwatch:Describe*",
                "cloudwatch:GetMetric*",
                "cloudwatch:PutMetric*",
                "cloudwatch:GenerateQuery",
                "cloudwatch:GetDashboard",
                "cloudwatch:List*",
                "cloudwatch:SetAlarmState",
                "ec2:DescribeRegions",
                "logs:DescribeLogGroups",
                "oam:ListSinks"
            ],
            "Effect": "Allow",
            "Resource": [
                "*"
            ]
        }
    ],
    "Version": "2012-10-17"
}

Create an Access Key and Secret for this user and put it in /var/root/.aws/credentials under [AmazonCloudWatchAgent].

[AmazonCloudWatchAgent]
region = us-east-1
aws_access_key_id = <your_access_key>
aws_secret_access_key = <your_secret_key>

Then rebuild your clouwatch config with the following command:

sudo /opt/aws/amazon-cloudwatch-agent/bin/amazon-cloudwatch-agent-ctl -a fetch-config -m onPrem -c file:/opt/aws/amazon-cloudwatch-agent/bin/config.json -s

If you want to collect metrics a few extra steps will be needed.

Install Homebrew if you have not done so already:

/bin/bash -c "$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/HEAD/install.sh)"

Install collectd to collect the metrics and awscli:

brew install collectd awscli

Edit your config in /opt/aws/amazon-cloudwatch-agent/bin/config.json to add collectd. In this example we are getting the disk used percent but add other mesaurements as you wish:

{
  "agent": {
    "metrics_collection_interval": 60,
    "logfile": "/opt/aws/amazon-cloudwatch-agent/logs/amazon-cloudwatch-agent.log"
  },
  "metrics": {
    "metrics_collected": {
      "collectd": {
        "collectd_security_level": "none",
        "collectd_typesdb": [
          "/opt/homebrew/opt/collectd/share/collectd/types.db"
        ],
        "metrics_aggregation_interval": 60
      },
      "disk": {
        "measurement": [
          "used_percent"
        ],
        "metrics_collection_interval": 60,
        "resources": [
          "/"
        ]
      }
    }
  }
}

Everytime you edit the config you need to re-run the fetch-config:

sudo /opt/aws/amazon-cloudwatch-agent/bin/amazon-cloudwatch-agent-ctl -a fetch-config -m onPrem -c file:/opt/aws/amazon-cloudwatch-agent/bin/config.json -s

For any measurements you will need the equivalent plugins enabled in collectd.conf. You can find the config file at /opt/homebrew/etc/collectd.conf.

At a minimum you will need to enable the following plugins:

LoadPlugin unixsock
LoadPlugin network
LoadPlugin syslog

And for our example with disk used percent:

LoadPlugin df
LoadPlugin disk

And restart collectd:

sudo brew services restart collectd

All being well, you should start to see the metric going to cloudwatch.

Enable SSM Agent

If you manage your fleet of servers using SSM, or want to send remote commands to your OXX servers from within AWS, you can enable the SSM agent.

Installing the SSM Agent

For updates in the ci/cd pipeline, the SSM agent needs to be installed.

sudo wget https://s3.us-east-1.amazonaws.com/amazon-ssm-us-east-1/latest/darwin_arm64/amazon-ssm-agent.pkg
sudo installer -pkg amazon-ssm-agent.pkg -target /

In AWS create a new user for the agent or use the same user you created for cloudwatch if that makes sense for you and attach the AmazonSSMManagedInstanceCore policy to it.

Generate an access key and secret key for the user and add them to the agent startup script in /Libray/LaunchDaemons/com.amazon.aws.ssm.plist:

    <key>EnvironmentVariables</key>
    <dict>
        <key>AWS_ACCESS_KEY_ID</key>
        <string>KEY</string>
        <key>AWS_SECRET_ACCESS_KEY</key>
        <string>SECRET</string>
    </dict>

Register a new hybrid activation in the SSM console by going to Systems Manager -> Hybrid Activations -> New Activation.

Use the key to register the instance:

sudo /opt/aws/ssm/bin/amazon-ssm-agent -register -code <activation-code> -id <activation-id> -region us-east-1

The OSX instance should now appear in SSM for you.

Some other General Settings

• Ensure correct Energy Settings - System Settings -> Energy
  • Disable Low Power Mode
  • Disable Prevent Automatic sleeping when the display is off
  • Disable Put hard disks to sleep when possible
  • Enable Wake for Network Access

Server Provisioning & Orchestration

If you are setting up multiple servers, there are a number of options available to make this setup process more automated.

MacStadium cover the more popular options in their docs, such as Ansible in their docs.

In addition, they offer a Kubernetes style OSX orchestration tool called Orka which we are excited to try out in the future when the use case presents itself.

In our case, we also use Packer to create a base AMI for OSX images on AWS and we keep this up-to-date with any changes made on MacStadium so that we can revert back to AWS if needed. This also allows us to test changes in AWS before deploying them to MacStadium for example major OS version upgrades.

Conclusion

So if you find that your Dedicated Host AWS bill is becoming a drag on your business, it is worth exploring whether other providers are able to provide the functionality at a lower cost.

You can also still rely on many of the other AWS services such as Cloudwatch even though you are running a hybrid cloud.

If you would like to discuss migrating your OSX workloads to MacStadium don't hesitate to reach out to us.