输入n个字符串字典序排序_FSST - 快速字符串压缩算法

VLDB： Fast Static Symbol Table (FSST): fast text compression that allows random access

这篇paper提出一种快速实现对“短字符串”进行压缩和解压缩的算法，并且可以提供“Random Access”的能力。

通过对比lz4（当前最优秀的通用压缩算法），发现LZ4在几k数据的情况下，压缩较好。而对于较短的字符串，处理并不好。一般都是将字符串列的多列数据放在一起组成一个block，然后交给lz4进行统一压缩。采用字典压缩。

此外，业界在int类型压缩上研究较为成熟，对于字符串的压缩，没有做到特别好。一般都是按照字典压缩，将字符串转换为int数据，然后可以对int数据再进一步压缩。但很少对字符串字典本身做压缩处理。并且也不支持能够随机访问，即，需要将整个block解压缩后进行访问，而对于非连续访问的情况，通常仅仅访问其中少量的数据，效率不高。

字符串压缩也有一些提出的方案，比如order-preserve的可以保证序，这在一篇2009年的论文中被提出，使用了tree，具体还没去读原论文。

这篇文章提到的方案是，对压缩的字符串进行采样抽取片段，然后对片段进行静态编码为一个1-255的byte，而这个片段也不长，最多8个字符串长度。这些小片段为编码单位。然后使用这个表对字符串集合进行逐个的编码，解码也一样，使用这个表去查表就能解码。从这个方案可以看到，得确具备随机访问的能力。

充分使用AVX-512指令，使压缩并行化进行，可以同时处理8个压缩：

压缩/解压分2步3个API：

/* Calibrate a FSST symboltable from a batch of strings (it is best to provide at least 16KB of data). */
fsst_encoder_t*
fsst_create(size_t n,         /* IN: number of strings in batch to sample from. */size_t lenIn[],   /* IN: byte-lengths of the inputs */unsigned char *strIn[],  /* IN: string start pointers. */int zeroTerminated       /* IN: whether input strings are zero-terminated. If so, encoded strings are as well (i.e. symbol[0]=""). */
);

这一步根据输入的采样字符串集合构造静态编码表，编码表的构造在论文中有大致的算法，在github上有完整的代码。

编码表结构如下：

/* Data structure needed for decompressing strings - read-only and thus can be shared between multiple decompressing threads. */
typedef struct {unsigned long long version;     /* version id */unsigned char zeroTerminated;   /* terminator is a single-byte code that does not appear in longer symbols */unsigned char len[255];         /* len[x] is the byte-length of the symbol x (1 < len[x] <= 8). */unsigned long long symbol[255]; /* symbol[x] contains in LITTLE_ENDIAN the bytesequence that code x represents (0 <= x < 255). */
} fsst_decoder_t;

然后可以进行压缩了：

/* Compress a batch of strings (on AVX512 machines best performance is obtained by compressing more than 32KB of string volume). */
/* The output buffer must be large; at least "conservative space" (7+2*inputlength) for the first string for something to happen. */
size_t                      /* OUT: the number of compressed strings (<=n) that fit the output buffer. */
fsst_compress(fsst_encoder_t *encoder, /* IN: encoder obtained from fsst_create(). */size_t nstrings,         /* IN: number of strings in batch to compress. */size_t lenIn[],          /* IN: byte-lengths of the inputs */unsigned char *strIn[],  /* IN: input string start pointers. */size_t outsize,          /* IN: byte-length of output buffer. */unsigned char *output,   /* OUT: memory buffer to put the compressed strings in (one after the other). */size_t lenOut[],         /* OUT: byte-lengths of the compressed strings. */unsigned char *strOut[]  /* OUT: output string start pointers. Will all point into [output,output+size). */
);

解压缩一样，要传入静态编码表:

/* Decompress a single string, inlined for speed. */
inline size_t /* OUT: bytesize of the decompressed string. If > size, the decoded output is truncated to size. */
fsst_decompress(fsst_decoder_t *decoder,  /* IN: use this symbol table for compression. */size_t lenIn,             /* IN: byte-length of compressed string. */unsigned char *strIn,     /* IN: compressed string. */size_t size,              /* IN: byte-length of output buffer. */unsigned char *output     /* OUT: memory buffer to put the decompressed string in. */
)

源代码链接

https://github.com/cwida/fsst