标签

PostgreSQL , 全文检索 , 词频统计 , ts_stat , madlib

背景

TF(Term Frequency 词频)/IDF(Inverse Document Frequency 逆向文本频率)是文本分析中常见的术语。

《PostgreSQL结合余弦、线性相关算法 在文本、图片、数组相似 等领域的应用 - 1 文本(关键词)分析理论基础 - TF(Term Frequency 词频)/IDF(Inverse Document Frequency 逆向文本频率)》

PostgreSQL支持全文检索，支持tsvector文本向量类型。

如何在一堆文本中，找到热词，或者对词频进行分析呢？

方法1，ts_stat

第一种方法来自PostgreSQL的内置函数，ts_stat，用于生成lexeme的统计信息，例如我想知道某个问答知识库中，出现最多的词是哪个，出现在了多少篇文本中。

ts_stat介绍如下

https://www.postgresql.org/docs/devel/static/functions-textsearch.html

https://www.postgresql.org/docs/devel/static/textsearch.html

https://www.postgresql.org/docs/devel/static/textsearch-features.html

12.4.4. Gathering Document Statistics

The function ts_stat is useful for checking your configuration and for finding stop-word candidates.

ts_stat(sqlquery text, [ weights text, ]  OUT word text, OUT ndoc integer,  OUT nentry integer) returns setof record

sqlquery is a text value containing an SQL query which must return a single tsvector column. ts_stat executes the query and returns statistics about each distinct lexeme (word) contained in the tsvector data. The columns returned are

• word text — the value of a lexeme

• ndoc integer — number of documents (tsvectors) the word occurred in

• nentry integer — total number of occurrences of the word

If weights is supplied, only occurrences having one of those weights are counted.

For example, to find the ten most frequent words in a document collection:

SELECT * FROM ts_stat('SELECT vector FROM apod')
ORDER BY nentry DESC, ndoc DESC, word
LIMIT 10;

The same, but counting only word occurrences with weight A or B:

SELECT * FROM ts_stat('SELECT vector FROM apod', 'ab')
ORDER BY nentry DESC, ndoc DESC, word
LIMIT 10;

测试

1、创建生成随机字符串的函数

create or replace function gen_rand_str(int) returns text as $$    select substring(md5(random()::text), 4, $1);
$$ language sql strict stable;

2、创建生成若干个随机词的函数

create or replace function gen_rand_tsvector(int,int) returns tsvector as $$    select array_to_tsvector(array_agg(gen_rand_str($1))) from generate_series(1,$2);
$$ language sql strict;

postgres=# select gen_rand_tsvector(4,10);  gen_rand_tsvector
-----------------------------------------------------------------------  '21eb' '2c9c' '4406' '5d9c' '9ac4' 'a27b' 'ab13' 'ba77' 'e3f2' 'f198'
(1 row)

3、创建测试表，并写入测试数据

postgres=# create table ts_test(id int, info tsvector);
CREATE TABLE
postgres=# insert into ts_test select generate_series(1,100000), gen_rand_tsvector(4,10);
INSERT 0 100000

4、查看词频，总共出现了多少次，在多少篇文本（多少条记录中出现过）

postgres=# SELECT * FROM ts_stat('SELECT info FROM ts_test')
ORDER BY nentry DESC, ndoc DESC, word
LIMIT 10;  word | ndoc | nentry
------+------+--------  e4e6 |   39 |     39  9596 |   36 |     36  a84c |   35 |     35  2b44 |   32 |     32  5146 |   32 |     32  92f6 |   32 |     32  cd56 |   32 |     32  fd00 |   32 |     32  4258 |   31 |     31  5f18 |   31 |     31
(10 rows)

5、再写入一批测试数据，查看词频，总共出现了多少次，在多少篇文本（多少条记录中出现过）

postgres=# insert into ts_test select generate_series(1,100000), gen_rand_tsvector(2,10);
INSERT 0 100000
postgres=# SELECT * FROM ts_stat('SELECT info FROM ts_test')
ORDER BY nentry DESC, ndoc DESC, word
LIMIT 10;  word | ndoc | nentry
------+------+--------  30   | 4020 |   4020  a7   | 4005 |   4005  20   | 3985 |   3985  c5   | 3980 |   3980  e6   | 3970 |   3970  f1   | 3965 |   3965  70   | 3948 |   3948  5e   | 3943 |   3943  e4   | 3937 |   3937  2b   | 3934 |   3934
(10 rows)

方法2，madlib

实际上MADlib也提供了词频统计的训练函数

http://madlib.apache.org/docs/latest/group__grp__text__utilities.html

Term frequency

Term frequency tf(t,d) is to the raw frequency of a word/term in a document, i.e. the number of times that word/term t occurs in document d. For this function, 'word' and 'term' are used interchangeably. Note: the term frequency is not normalized by the document length.

    term_frequency(input_table,  doc_id_col,  word_col,  output_table,  compute_vocab)

Arguments:

input_table

TEXT. The name of the table storing the documents. Each row is in the form <doc_id, word_vector> where doc_id is an id, unique to each document, and word_vector is a text array containing the words in the document. The word_vector should contain multiple entries of a word if the document contains multiple occurrence of that word.

id_col

TEXT. The name of the column containing the document id.

word_col

TEXT. The name of the column containing the vector of words/terms in the document. This column should of type that can be cast to TEXT[].

output_table

TEXT. The name of the table to store the term frequency output. The output table contains the following columns:

• id_col: This the document id column (same as the one provided as input).

• word: A word/term present in a document. This is either the original word present in word_col or an id representing the word (depending on the value of compute_vocab below).

• count: The number of times this word is found in the document.

compute_vocab

BOOLEAN. (Optional, Default=FALSE) Flag to indicate if a vocabulary is to be created. If TRUE, an additional output table is created containing the vocabulary of all words, with an id assigned to each word. The table is called output_table_vocabulary (suffix added to the output_table name) and contains the following columns:

• wordid: An id assignment for each word

• word: The word/term

参考

《PostgreSQL结合余弦、线性相关算法 在文本、图片、数组相似 等领域的应用 - 1 文本(关键词)分析理论基础 - TF(Term Frequency 词频)/IDF(Inverse Document Frequency 逆向文本频率)》

《一张图看懂MADlib能干什么》

http://madlib.apache.org/docs/latest/group__grp__text__utilities.html

https://www.postgresql.org/docs/devel/static/functions-textsearch.html

https://www.postgresql.org/docs/devel/static/textsearch.html

https://www.postgresql.org/docs/devel/static/textsearch-features.html

http://madlib.incubator.apache.org/