---

Installation and Best Practices

Because every setup of Mac OS X Server is different, you should understand as much as possible about the process before deciding on a plan for implementation. This article discusses and recommends accepted best practices that should apply to many, but certainly not all, situations.

This article is intended as an overview of the best practices for system administrators responsible for large-scale deployments of Mac OS X Server. This article is not a step-by-step manual. Where appropriate or available, references are provided for more information on a particular topic. Also, special attention is paid to large-scale enterprise and institutional needs, because smaller-scale needs are already well covered by existing documentation.

Contents:

Initial Installation of Mac OS X Server
Configuration of Mac OS X Server
Drive Configuration
Cloning the Boot Volume
Performing Ongoing Maintenance

Initial Installation of Mac OS X Server

Even though Server Setup Assistant provides a very robust and useful interface for setting up a server, a better methodology is required when deploying tens or hundreds of servers at any one time.

Many sites are using deployment techniques for Mac OS X Server that are very similar to Mac OS X client systems. Typically a "golden master" server image is created. This golden master sever image includes all relevant site-specific modifications, such as software installs and configuration, but has a dynamic DHCP-based IP address and no directory services configured.

The image is installed on the machines by hand imaging them with tools such as Apple's Disk Utility, which can do disk image restores. This process can easily be run from either an external bootable device, such as a FireWire hard drive, or a bootable DVD. The golden master image can either be stored on the bootable media, or hosted on a remote web or NFS server. For more information about installing images on computers, see Mac OS X Server System Image and Software Update Administration.

If you need to image more than a few systems at any one time, the use of a NetBoot or NetRestore server is common practice. Once booted from the NetBoot server, the systems are imaged with the golden master. Many times the imaging is actually accomplished by automated scripts so that no human intervention is required after the machine has NetBooted.

The new multicast Apple Software Restore (ASR) server that is present in Mac OS X Server v10.4 and Mac OS X v10.4 is being used for large scale system imaging. This tool allows the golden master image to be multicasted to as many machines as your network hardware can support at any one time. For more information about the ASR server see the ASR man page in the Mac OS X Man Pages.

Current deployments are imaging up to 2000 machines per day with a combination of NetBoot and multicast ASR with minimal administrative staff working on the task.

Regardless of whether you choose a full NetBoot or ASR setup or just use a firewire drive to restore an image onto the new server, you should not have to do full installs from the Mac OS X Server installation DVDs.

Configuration of Mac OS X Server

After the system has been installed, even if coming from the Mac OS X Server installation DVDs, you need to configure the system.

To facilitate mass deployments Apple has created a configuration file framework that allows administrators to pre-create setup files that a newly installed server will use to configure itself.

The Server Setup Assistant allows you to do this on systems installed from the installation media and to create text configuration files that can either be stored in the LDAP directory or connected to the machine by external storage.

Often you will be interested in more customized configurations, so it is common to use a golden master where the initial admin user and other settings have already been configured.

Note that when performing single machine installs through the Server Setup Assistant you should always configure the machine to be a Stand Alone system in regards to directory services. Log in to the machine first, and ensure that your settings for DNS and networking in general are working properly, before you changing the Open Directory configuration.

Also, if you are going through the Server Setup Assistant, you will be given the option to enable specific services that will be immediately available after the machine reboots for the last time during the configuration. With the exception of Apple Remote Desktop, used when the target machine is configured in a "headless" configuration without a video card, you should not enable any of the services.

Since you can enable individual services at a later time with no loss of functionality, it is best to ensure that your server has been correctly set up and then to configure each of the services before enabling them.

Drive Configuration

You should configure all servers with some form of RAID for the boot volume. If all of your data storage for the system is being handled by an external RAID device, considered best practice, then you can easily use the built-in software RAID of Mac OS X Server to mirror the boot drive.

If you need internal data storage, consider a hardware RAID card that will allow for RAID 5 and more configuration options than the software RAID alone will allow.

If you use an Xserve server computer with external storage, you want to fill the first two drive bays with smaller drive modules and then mirror the data for the boot volume. Then configure the third bay with a larger drive. Use it for admin storage, clones of the boot volume, and staging of backups.

While it is possible to install OS X on a UFS-formatted volume, it’s highly recommend you use an HFS+ Journaled format.

It’s common in many Unix server configurations to map the temp folder, virtual memory space, and /var directory to other partitions. This is typically done to either speed up the system or to prevent the boot drive from being filled up with either errant log files or ballooning virtual memory swap files. With the large amount of performance tuning that has been done with OS X to maximize the caching of the RAM and the kernel itself in addition to the massive increase in hard disk capacity in recent years, there are rarely, if ever, any gains to be had by implementing the remappings. Additionally, it is more typically a machine’s processor or network connection that is the bottleneck and not system drive I/O. In fact, you may be putting yourself at a disadvantage by doing this since very few admin tools or third party solutions for OS X expect this kind of configuration.

Cloning the Boot Volume

To account for system-update regressions and potential catastrophic damage to your boot volume, you should regularly clone the boot system onto other media.

In the case of the Xserve with external storage, the third drive bay is commonly configured with a large drive and then partitioned into three volumes. These volumes are used for a nightly backup, a known good backup, and a pre-update backup. Each of these is discussed in more detail below. The boot volume is then cloned to this drive using either rsync, psync, Apple System Restore, or a number of third-party utilities.

Every night the system clones the boot volume to the "Nightly" partition. This partition is used as a staging area for backups. It is also used as a primary restore volume in the case of a failure or corruption of the operating system. In these cases you can usually boot from the Nightly volume and have your server accessible to clients in a matter of minutes after the disaster. Then, when it is more convenient, you can either clone the Nightly volume back to the boot volume, or repair the boot volume so that it is useable again.

If you do clone your boot volume on a nightly basis, you can then run your backup software against the cloned system instead of your boot drive. Doing so lessens the performance impact of a backup on your server.

The boot volume is then cloned to a "Known Good" partition whenever the system has been running well for some time and has not had recent configuration changes. This volume is used less for restoration purposes and more for an online archive of configuration files.

Finally, the system should be cloned to a "Pre-Update" partition immediately before you apply software updates to the system. This provides a high degree of protection in case the update procedure produces unexpected results. In the same way that you booted from the Nightly volume, you can use the Pre-Update volume to restore a machine to functionality within minutes of detecting an abnormality with your system.

It's important to note that the cloning methodology described here is not a solution to backing up client data. It is preferred to have all dynamic client data be stored on an external RAID system and backed up to tape using any of a variety of backup solutions. The use of the boot volume clones is entirely to ensure the highest availability possible for your system volume, not your client data.

Performing Ongoing Maintenance

Software updates to production systems should be undertaken only after sufficient regression testing. Although the Nightly volume will help in this matter, with large installations it is common to use a network-based file management solution.

Apple's Apple Remote Desktop (ARD) allows an administrator to push software changes to hundreds of machines at the same time, helping you to ensure that they are all maintained in a similar fashion. With the introduction of ARD 3.0, machines that are offline during a software or configuration update are updated or configured when they come online..

If you desire a more automatic solution for this you can look into the open source Radmind, popular in a large number of higher education installations, or commercial options such as FileWave, Casper, and Net Octopus. All of these solutions allow you to do checksums on every file on your server and automatically replace any file that has been modified without permission.

Although these solutions usually have a fairly steep learning curve and infrastructure requirements, using them virtually guarantees that all of your systems are identical in configuration and prevents configuration decay over time.

---