Hot Fixing Base SAS and SAS Viya

Base SAS and SAS Viya are powerful frameworks for analytics that evolve constantly. Major new releases come out about every 5-10 years, while maintenance releases are about every year or two. Hot Fixes are the most granular and timely, and are issued in between these major upgrades and/or maintenance releases, to keep things patched and up-to-date.

Base SAS 9.4

The most recent major release of SAS is 9.4 M7 (as of the Summer of 2021). If you are not on this release yet, then it would be wise to start planning an upgrade to it.

With that said, every release of Base SAS and the variety of SAS products and solutions designed around SAS Foundation, such as BI, AML, MA, RTDM, EM –all have sets of hotfixes that get released. Hot fixes provide security patches, feature enhancements, and bug fixes to your existing analytics environment that you can apply quickly. These fixes come between maintenance releases or the larger major releases of the software are published.

In a SAS environment, you control what hotfixes you apply and when they get applied. In fact, many SAS environments get installed and never get a single hotfix applied afterward. The ZenGuard service that Zencos provides is an excellent way of having a professional and knowledgeable consultant review your SAS environment, recommend, and apply hotfixes for you.

The SAS Technical Support Hot Fixes webpage is the starting point for all information related to patching your system. There are various ways of staying up-to-date, including subscribing to an email notification service, checking the SAS Communities Announcement Board, and even subscribing to an RSS feed. But in many cases, you may be requested to apply a hotfix by SAS Tech Support if you’ve opened a ticket to troubleshoot and resolve an issue you’ve come across.

At a high level, you need to do the following to hotfix/patch your SAS environment:

1. Run the SAS Deployment Registry utility to generate the DeploymentRegistry.txt report. This report lists all the current hotfixes (or lack of) for your SAS environment. If you have a multi-tier SAS environment that has a Metadata Tier, Compute Tier, and Mid Tier then you will need to run this utility multiple times – once for each tier. You will then need to perform the following steps for each copy of the DeploymentRegistry.txt that you’ve produced:
2. Download and execute the SAS Hot Fix Analysis, Download, and Deployment (SASHFADD) Tool. Always work with the most recent version. The download link for this tool is always available from the SAS Technical Support Hot Fixes webpage. Note that you can download the SASHFADD Tool for Windows and still get the hot fixes for a Linux environment.
3. Use the DeploymentRegistry.txt file generated earlier, in conjunction with the SASHFADD tool, to produce a report of hotfixes available for your environment. The SASHFADD tool looks for the DeploymentRegistry.txt file specifically, so make sure you don’t rename it. Also, the SASHFADD tool wants you to specify some sort of unique identifier – either a machine name or tier name before it auto-generates some subdirectories – you can leave it blank, but it’s best to put some sort of name in – for example, “PROD_COMPUTE” or “META_CLUSTER_NODE3”, or better yet, the actual machine name like “awsprodmid01.sas.com”. This allows you to run the SASHFADD tool multiple times.

You must have an internet connection to SAS to run the SASHFADD Tool. If you don’t, or if your firewall is blocking the FTP port, then you will need to download the most recent copy of the database of all known SAS hotfixes (an XML file) that the SASHFADD tool relies on, to run the SASHFADD Tool Hot Fix report generator and be able to download the list of hotfixes identified for your SAS installation. The SASHFADD Usage Guide document provides a link to this database (which means you need to read the documentation!). You will also need to modify the SASHFADD configuration file to use this locally downloaded database file. Be aware that it is very large (currently about 6-7MB).

1. Review the hot fixes from the generated SASHFADD HTML report. There will be links to every hot fix listed, including the corresponding documentation and installation instructions. You can download each hot fix manually at this point, or simply switch to the DOWNLOAD_<machine>_<datetimestamp> directory as shown below and run one of the Powershell scripts to download ALL of the hot fixes to the DEPLOY directory. The Powershell script uses FTP to connect to a SAS website and downloads the correct hot fixes for you automatically.
2. If you are downloading SAS hotfixes for a Linux environment, you’ll need to then manually FTP or Secure-FTP (sftp) these hotfixes from your Windows machine to the corresponding Linux machines using an FTP client like WinSCP or FileZilla. SAS defaults and assumes you will put your hot fixes in SASHOME/InstallMisc/HotFixes/New, but a best practice is to create a dated subdirectory under your SAS Depot to store the hotfixes that you downloaded. example: /sasdepot/Hotfixes_2021-07-21
3. Lastly, if patching the SAS environment affects multiple users, then schedule an outage. You would typically need to coordinate and submit a Change Control or some other communication at your organization, and inform the SAS user community that there will be a brief outage, since you will need to bring down various SAS services before applying the hotfixes.
4. You apply almost all SAS Hot Fixes using the SAS Deployment Manager utility. Choose “Apply Hot Fixes” and point to the directory where you’ve stored the hot fixes. Again, always confirm that all SAS services have been shutdown before applying SAS hot fixes.

SAS Hot Fixes come in two flavors: those that have no configuration or pre/post installation steps, and those that do. The former are really simple to implement and require no extra manual work after the SAS Deployment Manager utility applies the hot fixes to the machine, other than restarting the SAS services. The latter typically require some post-installation steps such as using the SAS Deployment Manager utility again (multiple times) to re-build and deploy web apps, before restarting the SAS services. Regardless, it is very important to review each and every hot fix installation document from the SASHFADD HTML Report, as there may be some surprises which you may want to contact SAS Tech Support about.

SAS Viya

SAS Viya hot fixes are applied completely differently from the traditional SAS hotfixes discussed above. Firstly, SAS Viya environments require that you must be connected to the internet and allow traffic to the SAS download servers. This is because each update of a SAS Viya environment requires the environment to pull the latest version of all the software you’re licensed for. In other words, every update of a Viya installation is, in essence, a reinstallation of all the out-of-date components to the latest version. The SAS documentation explains this best: “An update replaces some or all of your deployed software with the latest version of the software. You perform the update with the same software order that was used to install SAS Viya.” Like SAS 9, Viya has minor and major upgrades. However, unlike SAS 9, there are no maintenance releases such as SAS 9.4 M1 – M7. Instead, you either update the individual components, or you upgrade the version of Viya.

The basic process for applying the latest hotfixes to a Viya environment is as follows:

1. If the environment was installed using a mirror repository, it will need to be synchronized with the latest hotfixes. If the original script to download the mirror repository is available, it can be reused to update the mirror repository. Though unnecessary, it’s good practice to back up the old mirror repository before synchronizing it if disk space allows. Synchronizing the mirror repository, downloads a local copy of the latest versions of all the software the environment is licensed for. This is the only step that requires an external internet connection and can be done prior to the update period as it does not require any downtime of the environment.
2. The next step is to generate a new and up-to-date playbook. This step is not always required, as the ansible playbook is not always updated. However, if the playbook is updated, the Viya update(s) won’t work properly using the old playbook. Furthermore, it’s difficult to tell when SAS has updated the playbook, so it’s best to include this step in every update.
   • To generate a new playbook, recreate it using the SAS Orchestration CLI. Your environment should already have this installed from its original deployment. If the SAS Orchestration CLI script has been removed, then the original Software Order Email (SOE) will contain a link to download it.
   • Once the CLI is downloaded, create the new playbook using the “./sas-orchestration“ command, pointing it to the downloaded repository zip file, and including the appropriate operating system parameter. Instructions on formulating the command can be found on the SAS documentation website. As a best practice, extract the new playbook to a different folder than the current playbook in use to avoid confusion and not to overwrite the old one.
3. Now it’s time to prepare the environment for updating. To successfully run the update, all but a few services need to be shutdown. The only micro-services that should be up prior to starting the update are the Consul service, the Vault, any Local Consuls, and the HA Postgres Consul Template. The HA Postgres Consul Template will have a naming convention that follows this naming convention: “sas-viya-sasdatasvrc-servicename–postgresnode-consul-template-consul-template-service-name”.
4. If the environment is multi-tenanted, you’ll need to shut down each of the tenants as well. A good pre-check to run before running the update script is to run the system-assessment playbook. This runs a series of checks to make sure the environment is in a good state for executing an installation or update.
5. Once this multi-tenant check is complete (if applicable), the last step is to verify that the configuration files haven’t been changed in the new playbook. The vars.yml, inventory.ini, and ansible.cfg files need to be double-checked for any discrepancies, as a new playbook can sometimes add or modify various parameters. The easiest way to do this is to run a diff command between the old/current vars.yml and the new one after accounting for the environment specific inputs. If there are any differences, the two files will need to be merged to account for the additions.
6. After all the files are ready, it’s time to start the update. Go to the new playbook directory and run the ansible-playbook site.yml command to initiate the process. This will initiate a lengthy ansible process, but the update should be completed once it finishes.
7. Finally, start the environment back up and validate the functionality. At this point, the update process is completed.

To upgrade from a previous Viya version, the process is more complex. There are a significant number of manual steps involved, especially if upgrading from a version below Viya 3.4. It may only require a small subset of them, depending on which solutions the environment is licensed for. To account for all of these steps, it’s best to reference the SAS documentation directly, especially if upgrading to 3.5 from a version before Viya 3.4. Lastly, if the Viya-ARK playbook has been downloaded to the environment, it contains the viya-pre-upgrade.yml playbook that automates several steps, including disk space checks, health validations, tenant shutdowns, and several others. Once all of the manual steps are completed however, the process is very similar to the SAS Viya hotfixing process listed above.

Running an update to Viya is very similar to the initial installation process. Fortunately, since the software was already installed and configured in the environment, the future path is now paved for seamless updates. The above steps are a helpful reference to the overall process involved in updating a Viya environment to allow for any new security updates or features and enhancements that SAS releases for Viya.