Paper 258-2013
Data Set Compression using COMPRESS=

ABSTRACT
DUE TO AN INCREASED AWARENESS ABOUT DATA MINING, TEXT MINING AND BIG DATA APPLICATIONS ACROSS ALL DOMAINS THE VALUE OF DATA HAS BEEN REALIZED AND IS RESULTING IN DATA SETS WITH A LARGE NUMBER OF VARIABLES AND INCREASED OBSERVATION SIZE. OFTEN IT TAKES A LOT OF TIME TO PROCESS THESE DATASETS WHICH CAN HAVE AN IMPACT ON DELIVERY TIMELINES. WHEN THERE IS LIMITED PERMANENT STORAGE SPACE, STORING SUCH LARGE DATASETS MAY CAUSE SERIOUS PROBLEMS. BEST WAY TO HANDLE SOME OF THESE CONSTRAINTS IS BY MAKING A LARGE DATASET SMALLER, BY REDUCING THE NUMBER OF OBSERVATIONS AND/OR VARIABLES OR BY REDUCING THE SIZE OF THE VARIABLES, WITHOUT LOSING VALUABLE INFORMATION. IN THIS PAPER WE WILL SEE HOW A SAS DATA SET CAN BE COMPRESSED USING THE COMPRESS= SYSTEM OPTION AND ALSO SOME TECHNIQUES TO MAKE THIS OPTION MORE EFFECTIVE.

Introduction
THE PROCESS OF REDUCING THE NUMBER OF BYTES REQUIRED TO REPRESENT EACH OBSERVATION IS KNOWN AS COMPRESSION. SOME OF THE ADVANTAGES OF COMPRESSING A DATASET INCLUDE BUT ARE NOT LIMITED TO, A COMPRESSED FILE REDUCES THE STORAGE REQUIREMENTS AND ALSO REDUCES THE NUMBER OF I/O OPERATIONS NECESSARY TO READ OR WRITE THE DATA DURING PROCESSING. IN A COMPRESSED FILE, THE DELETED OBSERVATION SPACE CAN BE REUSED USING REUSE= OPTION, WHEREAS IN AN UNCOMPRESSED DATA SET THE DELETED OBSERVATION SPACE IS NEVER REUSED.

To create a compressed data set we use the COMPRESS= output data set option or system option.

Syntax: SAS-data-set(Compress=NO| YES| CHAR| BINARY)

OPTIONS COMPRESS=NO|YES|CHAR|BINARY

Two data sets are being used here, one contains character data and the other binary data.

Compressing text data
THE OPTION COMPRESS=YES|CHAR IS EFFECTIVE WITH CHARACTER DATA THAT CONTAINS REPEATED CHARACTERS SUCH AS BLANKS. IT USES THE RUN-LENGTH ENCODING (RLE) COMPRESSION ALGORITHM, WHICH COMPRESSES REPEATING CONSECUTIVE BYTES SUCH AS TRAILING BLANKS OR REPEATED ZEROES.

options fullstimer;

PROC IMPORT OUT= WORK.charcompress(compress=yes)

DATAFILE= "H:\compress.csv"

DBMS=CSV REPLACE;

GETNAMES=YES;

DATAROW=2;

RUN;

Measuring Efficiency
THE FOLLOWING ARE THE STATISTICS FOR COMPRESSED DATA SET

[pic]

The size of the data set decreased by 36.11 percent and the number of pages decreased are 130, which significantly decreases the number of I/O operations. There will be increase in CPU time but the savings in I/O operations greatly outweigh the increase in CPU time.

Before Compressing:

[pic]

After Compressing:

[pic]

From the above performance statistics we can see that the processing time for compressed data set is 1.1 seconds which is 0.19 seconds less compared to the uncompressed version of the file. There is also significant decrease in memory due to data set compression.

Compressing Numeric Data
THE OPTION COMPRESS=BINARY USES THE ROSS DATA COMPRESSION (RDC) ALGORITHM, WHICH COMBINES RUN-LENGTH ENCODING AND SLIDING WINDOW COMPRESSION. THIS OPTION IS MORE EFFICIENT WITH OBSERVATIONS GREATER THAN A THOUSAND BYTES IN LENGTH AND CAN BE VERY EFFECTIVE WITH NUMERIC DATA AND CHARACTER DATA THAT CONTAIN PATTERNS RATHER THAN SIMPLE REPETITIONS. IT TAKES SIGNIFICANTLY MORE CPU TIME TO UNCOMPRESS THAN COMPRESS=YES|CHAR.

options fullstimer;

PROC IMPORT OUT= WORK.bindata(compress=binary)

DATAFILE= "H:\binary.csv"

DBMS=CSV REPLACE;

GETNAMES=YES;

DATAROW=2;

RUN;

Measuring Efficiency
THE FOLLOWING ARE THE STATISTICS FOR DATA COMPRESSED USING COMPRESS=BINARY OPTION

[pic]

The size of the data set decreased by 81.82 percent after compressing and also the number of pages required by compressed data set decreased to 10 from 55.

Before Compression:

[pic]

After Compression

[pic]

The processing time for compressed data set is 1 second where as for uncompressed is 1.2, our sample data set contains only 425 observations but in many practical applications the data set contains millions of observations so the difference in processing time will be very significant.

Making Compress= Option more effective
SOME OF YOU MIGHT BE WONDERING WHY SAS DOES NOT, FOR THE SAKE OF MAKING ALL DATA SETS AS SMALL AS POSSIBLE, AUTOMATICALLY COMPRESS THEM DURING DATA STEP PROCESSING. THERE ARE SEVERAL REASONS WHY WE DON’T FAVOR THE “BLIND” USE OF THIS SAS SYSTEM OPTION IN ALL SITUATIONS. ONE OF THE MAIN REASONS IS INCREASE IN CPU-TIME. ADDITIONAL CPU RESOURCES ARE REQUIRED TO BOTH CREATE A COMPRESSED DATA SET AND TO APPLY SAS TASKS TO THE COMPRESSED DATA SET. THE SAS SYSTEM MUST UNCOMPRESS THE DATA SET EVERY TIME IT APPLIES A DATA STEP OR PROCEDURE STEP TO IT.

Following are few techniques which can be used along with the compress= option to make the process of decreasing the size of a large data set more effective.

1. LENGTH STATEMENT: The default length of numeric variables is 8 bytes. The Length statement can be used to declare the byte lengths of variables in the Program Data Vector

More observations can be placed in each page of a dataset memory by appropriately adjusting the byte lengths of variables, resulting in a smaller data set and also reducing the number of input-output operations required to move the data set in and out of the CPU.

2. REUSE= OPTION: When you create a compressed file, you can also specify REUSE=YES (as a data set option or a system option) in order to track and reuse space. With REUSE=YES, new observations are inserted in space which becomes freed when other observations are updated or deleted, which results in optimal usage of the space. Note that when using Compress=yes and reuse=yes system options, the observations cannot be addressed using observation numbers. This is because the REUSE= option has higher precedence over POINTOBS= option and they are mutually exclusive.

3. OPTIMIZING CPU PERFORMANCE: The major drawback of compressing data set is increase in CPU time, by setting the CGOPTIMIZE= option to 0 CPU time can be saved when processing large data sets. So Optimizing CPU performance while compressing, using techniques like these can improve the overall performance.

CONCLUSION
DATA SET COMPRESSION REDUCES THE SIZE OF THE DATA SETS ONLY IF THE MAXIMUM SIZE OF THE OBSERVATION IS MORE THAN THE 12-BYTE (32-BIT SYSTEMS) OR 24-BYTE (64-BIT SYSTEM) OVERHEAD INTRODUCED BY COMPRESSION. A COMPRESSED DATA SET REQUIRES LESS NUMBER OF PAGES WHICH RESULTS IN LESS NUMBER OF I/O OPERATIONS AND INCREASED PERFORMANCE. NEVERTHELESS, IT SHOULD BE NOTED THAT THERE IS AN INCREASE IN CPU TIME FOR PROCESSING THE COMPRESSED FILE, SO IF YOU HAVE LIMITED CPU RESOURCES THEN COMPRESSING A DATA SET MIGHT NOT BE A GOOD OPTION.

References

“HTTP://SUPPORT.SAS.COM/DOCUMENTATION/CDL/EN/LRDICT/64316/HTML/DEFAULT/VIEWER.HTM#A000202890.HTM”

“HTTP://SUPPORT.SAS.COM/DOCUMENTATION/CDL/EN/LRDICT/64316/HTML/DEFAULT/VIEWER.HTM#A000202894.HTM”

ANDREW H. KARP “INDEXING AND COMPRESSING SAS®DATA SETS: HOW, WHY AND WHY NOT” SUGI28 PAPER 3-28

“HTTP://SUPPORT.SAS.COM/DOCUMENTATION/CDL/EN/LRCON/62955/HTML/DEFAULT/VIEWER.HTM#A001091115.HTM”

CONTACT INFORMATION
YOUR COMMENTS AND QUESTIONS ARE VALUED AND ENCOURAGED. CONTACT THE AUTHOR AT:
Srinivas Busi Reddy, Email: srinivas.busi_reddy@okstate.edu
Srinivas Busi Reddy is Master’s student in Management Information Systems at Oklahoma State University. He is an ADVANCED SAS® 9 certified Programmer.

Srikar Rayabaram, Email: rayabar@okstate.edu
Srikar Rayabaram is Master’s student in Management Information Systems at Oklahoma State University. He is a BASE SAS® 9 and a certified SAS® predictive modeler using Enterprise Miner 6. He will be receiving his SAS® and OSU Data Mining Certificate in December 2012.

Musthan Meeran Mohideen, Email: musthan@okstate.edu
Musthan Meeran Mohideen is Master’s student in Management Information Systems at Oklahoma State University. He is a BASE SAS® 9 and a certified SAS® predictive modeler using Enterprise Miner 6.

SAS and all other SAS Institute Inc. product or service names are registered trademarks or trademarks of SAS Institute Inc. in the USA and other countries. ® indicates USA registration.

Other brand and product names are trademarks of their respective companies.