Version 2 extends compression up to 512 bytes.

 1. In td512.c, for 128 and more values, text and extended string modes
    are called for checked data. For other data, and for any remaining values
    from calls for 128 or more values, td64 is called.
    A minimum of 16 characters are compressed.
 2. In tdString.c, extended string mode was modified to stop on the 65th
    unique value. This value is the last value output and the number
    of values at that point is returned.
 3. In main.c, after decompression, the input file is verified against
    the decompressed output file.

Author: lsleonard

README.md

@@ -7,9 +7,9 @@ td512 filename [loopCount]
 
 	loopCount (default 1) is the loop count to use for performance testing. Also see BENCHMARK_LOOP_COUNT macro in main.c.
 
-Tiny data compression is not usually supported by compression programs. Now with td512 you can compress data from 6 to 512 bytes. td512 is available under the GPL-3.0 License at https://github.com/lsleonard/tiny-data-compression. Although for some types of data, programs QuickLZ, Zstandard and Snappy can get better compression at 512 bytes than td512, the performance of td512 is very close. All these programs steadily decline in compression ratio as the number of bytes decreases to 128. At 64 bytes, none of these programs produces compression. td512 combines the compressed output of td64 for each block of 64 bytes in the input, meaning that the compression achieved at 512 bytes is the same as that for 64 bytes. The td512 algorithm emphasizes speed, and based on data in this paper, gets 26.5% average compression at 272 Mbytes per second on the Squash benchmark test data (see https://quixdb.github.io/squash-benchmark/#) running on a 2 GHz processor.
+Tiny data compression is not supported by standard compression programs. Now with td512 you can reasonably compress data from 16 to 512 bytes. td512 is available under the GPL-3.0 License at https://github.com/lsleonard/tiny-data-compression. Compared with QuickLZ, a fast compression program that is designed to compress smaller data sets, td512 gets as good or better compression for 512-byte blocks of most data types. QuickLZ steadily declines in compression ratio as the number of bytes decreases to 128, and at 64 bytes, produces no compression. td512 has good compression at 64 bytes with the td64 interface. td512 combines extended text and string modes for 128 to 512 bytes with the td64 interface to compress any remaining bytes in the input. The td512 algorithm emphasizes speed, and based on data in this paper, gets 31.6% average compression for 512-byte blocks at 250 Mbytes per second on the Squash benchmark test data (see https://quixdb.github.io/squash-benchmark/#) running on a 2 GHz quad-core processor. For 64-byte blocks on this benchmark data, td512 gets 25.3% average compression at 250 MBytes per second.
 
-You can call the td512 and td512d functions to compress and decompress 1 to 512 bytes. The td512 interface performs compression of 6 to 512 bytes, but accepts 1 to 5 bytes and stores them without compression. td512 acts as a wrapper that uses the td64 interface to compress blocks of 64 bytes until the final block of 64 or fewer bytes is compressed. Along with the number of bytes processed, a pass/fail bit is stored for each 64-byte (or smaller) block compressed, and the compressed or uncompressed data is output.
+You can call the td512 and td512d functions to compress and decompress 1 to 512 bytes. The td512 interface performs compression of 16 to 512 bytes, but accepts 1 to 15 bytes and stores them without compression. Along with its extended text and string modes, td512 acts as a wrapper that uses the td64 interface to compress blocks of 64 bytes until the final block of 64 or fewer bytes is compressed. Along with the number of bytes processed, a pass/fail bit is stored for each block compressed, and the compressed or uncompressed data is output.
 
 With td64, you can call the td5 and td5d functions to compress and decompress 1 to 5 values. This interface is not used by td512 because the number of bytes generated is often more than the number of values to compress. Or you can call td64 and td64d functions to compress and decompress 6 to 64 values. The td64 interface returns pass (number of compressed bits) or fail (0) and outputs only compressed values. Decompression requires input of the number of original values.
 

Tiny Data Compression with td512.docx

@@ -1,39 +1,43 @@
 //
 //  main.c
 //  high-speed lossless tiny data compression for 1 to 512 bytes based on td512
-//  version 1.1
 //
 //  file-based test bed outputs .td512 file with encoded values
 //  then reads in that file and generates .td512d file with
 //  original values.
 //
-//  Created by Stevan Leonard on 10/30/21.
-//  Copyright © 2021 Oxford House Software. All rights reserved.
-//
+//  Created by L. Stevan Leonard on 10/31/21.
+//  Copyright © 2021-2022 L. Stevan Leonard. All rights reserved.
 /*
-    This program is free software: you can redistribute it and/or modify
-    it under the terms of the GNU General Public License as published by
-    the Free Software Foundation, either version 3 of the License, or
-    (at your option) any later version.
-
-    This program is distributed in the hope that it will be useful,
-    but WITHOUT ANY WARRANTY; without even the implied warranty of
-    MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
-    GNU General Public License for more details.
-
-    You should have received a copy of the GNU General Public License
-    along with this program.  If not, see <https://www.gnu.org/licenses/>.//
-*/
+ This program is free software: you can redistribute it and/or modify
+ it under the terms of the GNU General Public License as published by
+ the Free Software Foundation, either version 3 of the License, or
+ (at your option) any later version.
+ 
+ This program is distributed in the hope that it will be useful,
+ but WITHOUT ANY WARRANTY; without even the implied warranty of
+ MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+ GNU General Public License for more details.
+ 
+ You should have received a copy of the GNU General Public License
+ along with this program. If not, see <https://www.gnu.org/licenses/>.
+ */
 #include "td512.h" // td512 functions
 
 #include <stdio.h>
 #include <stdlib.h>
 #include <time.h>
 #include <string.h>
 
-#define BENCHMARK_LOOP_COUNT // // special loop count for benchmarking
+#define BENCHMARK_LOOP_COUNT // special loop count for benchmarking
 //#define TEST_TD512 // invokes test_td512_1to512
 
+#ifdef TD512_TEST_MODE
+extern uint32_t gExtendedStringCnt;
+extern uint32_t gExtendedTextCnt;
+extern uint32_t gtd64Cnt;
+#endif
+
 int32_t test_td512_1to512(void)
 {
     // generate data then run through compress and decompress and compare for 1 to 512 values
@@ -63,7 +67,7 @@ int32_t test_td512_1to512(void)
     }
     if (textData[0] == 'i')
     {
-        // set all values to 0x83 and run again
+        // set all values to same value and run again
         memset(textData, 0x91, sizeof(textData));
         goto RUN_512;
     }
@@ -90,7 +94,7 @@ int main(int argc, char* argv[])
     int loopCnt; // argv[4] option: default is 1
     uint32_t blockSize=512; // block size to use when iterating through file
 
-    printf("tiny data compression td512 %s   block size: %d\n", TD64_VERSION, blockSize);
+    printf("tiny data compression td512 %s   block size: %d\n", TD512_VERSION, blockSize);
 #ifdef TEST_TD512
     int32_t retVal;
     if ((retVal=test_td512_1to512()) != 0) // do check of 1 to 512 values
@@ -124,7 +128,7 @@ int main(int argc, char* argv[])
     fclose(ifile);
 
     // allocate "uncompressed size" + 3 bytes per block for the destination buffer
-    dst = (unsigned char*) malloc(len + 3 * (len / blockSize + 1));
+    dst = (unsigned char*) malloc(len + 4 * (len / blockSize + 1));
     if (argc >= 3)
     {
         sscanf(argv[2], "%d", &loopCnt);
@@ -135,9 +139,9 @@ int main(int argc, char* argv[])
     {
         loopCnt = 1;
     }
-#ifdef BENCHMARK_LOOP_COUNT // // special loop count for benchmarking
+#ifdef BENCHMARK_LOOP_COUNT // special loop count for benchmarking
     loopCnt = 100000000 / len;
-    loopCnt = (loopCnt < 20) ? 20 : loopCnt;
+    loopCnt = (loopCnt < 20) ? 10 : loopCnt;
     loopCnt = (loopCnt > 2000) ? 2000 : loopCnt;
 #endif
     loopNum = 0;
@@ -154,7 +158,7 @@ int main(int argc, char* argv[])
     while (nBytesRemaining > 0)
     {
         uint32_t nBlockBytes=(uint32_t)nBytesRemaining>=blockSize ? blockSize : (uint32_t)nBytesRemaining;
-            nCompressedBytes = td512(src+srcBlockOffset, dst+dstBlockOffset, nBlockBytes);
+        nCompressedBytes = td512(src+srcBlockOffset, dst+dstBlockOffset, nBlockBytes);
         if (nCompressedBytes < 0)
             exit(nCompressedBytes); // error occurred
         nBytesRemaining -= nBlockBytes;
@@ -173,12 +177,17 @@ int main(int argc, char* argv[])
     }
     timeSpent = minTimeSpent;
     printf("compression=%.02f%%  %.00f bytes per second inbytes=%lu outbytes=%u\n", (float)100*(1.0-((float)totalCompressedBytes/(float)len)), (float)len/(float)timeSpent, len, totalCompressedBytes);
+#ifdef TD512_TEST_MODE
+    double totalBlocks=gExtendedTextCnt+gExtendedStringCnt+gtd64Cnt;
+    printf("TD512_TEST_MODE\n   Extended text mode=%.01f%%   Extended string mode= %.01f%%   td64 =%.01f%%\n", (float)gExtendedTextCnt/totalBlocks*100, (float)gExtendedStringCnt/totalBlocks*100, (float)gtd64Cnt/totalBlocks*100);
+#endif
 
     fwrite(dst, totalCompressedBytes, 1, ofile);
     fclose(ofile);
     free(src);
     free(dst);
 
+    // **********************
     // decompress
     ifile = fopen(ofileName, "rb");
     strcpy(ofileName, argv[1]);
@@ -194,6 +203,7 @@ int main(int argc, char* argv[])
     fread(src, 1, len3, ifile);
     len2 = len; // output==input
     dst = (unsigned char*) malloc(len2);
+    fclose(ifile);
 
     minTimeSpent=600;
     loopNum = 0;
@@ -208,6 +218,8 @@ int main(int argc, char* argv[])
     {
         int32_t nRetBytes;
         nRetBytes = td512d(src+srcBlockOffset, dst+dstBlockOffset, &bytesProcessed);
+        if (nRetBytes != 512)
+            nRetBytes = nRetBytes;
         if (nRetBytes < 0)
             return nRetBytes;
         nBytesRemaining -= bytesProcessed;
@@ -227,7 +239,27 @@ int main(int argc, char* argv[])
     timeSpent = minTimeSpent;
     printf("decompression=%.00f bytes per second inbytes=%lu outbytes=%lu\n", (float)len/(float)timeSpent, len3, len);
     fwrite(dst, len, 1, ofile);
-    fclose(ifile);
     fclose(ofile);
+    free(src);
+    // verify original input file with decompressed output
+    ifile = fopen(argv[1], "rb");
+    if (!ifile)
+    {
+        printf("td512 error: file not found to verify with decompressed output file: %s\n", argv[1]);
+        return 9;
+    }
+    // allocate source buffer and read file
+    src = (unsigned char*) malloc(len);
+    fread(src, 1, len, ifile);
+    fclose(ifile);
+    if ((memcmp(src, dst, len)) != 0)
+    {
+        printf("td512 error: decompressed file differs from original input file\n");
+        free(src);
+        free(dst);
+        return 11;
+    }
+    free(src);
+    free(dst);
     return 0;
 }