blog

Using the SAS Visual Analytics AutoLoad Feature

11/18/2015 by Jaime D’Agord

Self-driving cars are a few years away, but you can use self-loading data today with SAS Visual Analytics. The SAS AutoLoad process moves or removes data tables from a defined server directory to the SAS Visual Analytics LASR Server.

The data can be a delimited text file, a spreadsheet, or a SAS data set. The administrator schedules the AutoLoad process to run at regular intervals, such as every 10-15 minutes. By default, the AutoLoad process is attached to each SAS Analytic LASR server defined during the installation. This process can be used to supplement your daily batch process or as a self-service feature for your users.

The following figure shows some key elements of the AutoLoad process you need to understand. The host file system has two directories: one directory contains the AutoLoad scripts and the second a drop location where users put the data tables. SAS Management Console contains the LASR Library definition, which customizes the process with the LASR Server. With the host file system and metadata properly enabled, you can use the SAS Visual Analytics Administrator to verify the tables were loaded.

sas auto loader

Understanding the SAS Visual Analytics AutoLoad Process

It’s easier to understand how these elements work together once you understand the overall process.  The AutoLoad Drop location has three directories: AutoLoad, Append, and Unload. Users place data in each directory based on the desired action and the AutoLoad process evaluates the data based on its name and its timestamp to determine if it is loaded or unloaded.  The following figure shows a basic overview of what happens to the data when placed in each folder.

The following figure shows how the directory appears in Linux for the Public LASR Server. The EVDMLA and VALIBLA are the AutoLoad folders for the other LASR server libraries.

The default location for the Public LASR Server AutoLoad directory is:

<sasconfig>/ Lev1/AppData/SASVisualAnalytics/VisualAnalyticsAdministrator/AutoLoad

Understanding the SAS VA AutoLoad Scripts

The AutoLoad scripts allow you to control the schedule, configuration and execute the process. The schedule.sh script calls the RUNSAS.SH script, which in turn calls the AutoLoad.sas program.  The AutoLoad.sas program points to a set of SAS macros that control how the data is managed with the LASR Server. The administrator runs the schedule.sh (or schedule.bat) script to enable the AutoLoad process.

The following figure shows how the directory appears in Linux. The Logs directory is where each log is placed after the AutoLoad process runs.

The path is:

 <sasconfig>/Lev1/Applications/SASVisualAnalytics/VisualAnalyticsAdministrator

Understanding the SAS Metadata Library Attributes

In the SAS Management Console you can find a metadata definition for the LASR Analytic Servers. Each LASR Library must have the AutoLoad extended attribute defined. These attributes define where to register the tables in the metadata, if the AutoLoad is enabled, where the host file scripts are located and so on. Refer to the SAS Administrators manual for detailed information about each attribute.

By default, the libraries defined in the installation are already configured, as shown in the following figure. This is the Extended Attributed tab from the Visual Analytics Public LASR library. Perhaps you noticed the host location attributed is the path used above for the data.

Verifying the SAS VA AutoLoad Worked

There are two ways to determine if the AutoLoad process completed successfully. The easiest way is to log in to SAS Visual Analytics Administration tab. You can review what tables are loaded and when. In the following figure, you can see two tables were loaded with the green dots. The Loaded and Modified columns show the dates. The table with the red square was not loaded.  Sometimes users get confused when they don’t see the table getting updated every 15 minutes. You have to remind them that the table is only reloaded when the timestamp on the file changes.

If a table failed to load then check the Logs sub-directory in the scripts area. Each time the AutoLoad process runs a log is generated. The log has the timestamp on it so you can sort to find the last one loaded.

Tips for Enabling the SAS AutoLoad Feature

To enable the AutoLoad feature, the SAS administrator runs the schedule script (schedule.sh/schedule.bat).

Typically the LASRADM account schedules this job because:

  • Can start/stop the LASR server
  • Has the ability to schedule jobs
  • Read and write access to the AutoLoad directories
  • Has a metadata user ID

If you are running LINUX, check crontab to see that the job was scheduled.  From Windows, check the Task Scheduler. You can run the unschedule script to disable the AutoLoad process.