02_clickhouse安装，卸载，启动/关闭服务，交互式命令（数据库创建，数据导入，查询），批模式数据导入，MySQL接口操作ClickHouse，DBeaver可视化工具安装与使用（学习笔记）

1 ClickHouse安装

安装文件清单

clickhouse-client-${version}.noarch.rpm
clickhouse-common-static-dbg-${version}.x86_64.rpm
clickhouse-common-static-${version}.x86_64.rpm
clickhouse-server-${version}.noarch.rpm

clickhouse的安装只需要上面4个文件

安装命令

命令1: rpm -ivh *.rpm
命令2： yum install *.rpm

卸载
查看

rpm -qa | grep clickhouse

卸载命令：

rpm -e  rpm包名

如果卸载不了，使用：

# 卸载及删除安装文件（需root权限）
yum list installed | grep clickhouseyum remove -y clickhouse-common-staticyum remove -y clickhouse-server-commonrm -rf /var/lib/clickhouserm -rf /etc/clickhouse-*rm -rf /var/log/clickhouse-server

卸载报错时强制删除：

# 删除rpm包的时候不调用卸载脚本
sudo rpm -e clickhouse-server.x86_64 --noscripts

启动/关闭服务

systemctl start clickhouse-server
systemctl status clickhouse-server
systemctl restart clickhouse-server

交互式命令

clickhouse-client

select 1

测试数据
下载下面文件，使用wget方式下载

wget https://clickhouse-datasets.s3.yandex.net/hits/tsv/hits_v1.tsv.xz
wget https://clickhouse-datasets.s3.yandex.net/visits/tsv/visits_v1.tsv.xzyum install xz xz -d hits_v1.tsv.xz
xz -d visits_v1.tsv.xz

创建数据库

clickhouse-client --query "CREATE DATABASE IF NOT EXISTS tutorial"

建表
创建hits_v1表

clickhouse-client  -m

CREATE TABLE tutorial.hits_v1
(`WatchID` UInt64,`JavaEnable` UInt8,`Title` String,`GoodEvent` Int16,`EventTime` DateTime,`EventDate` Date,`CounterID` UInt32,`ClientIP` UInt32,`ClientIP6` FixedString(16),`RegionID` UInt32,`UserID` UInt64,`CounterClass` Int8,`OS` UInt8,`UserAgent` UInt8,`URL` String,`Referer` String,`URLDomain` String,`RefererDomain` String,`Refresh` UInt8,`IsRobot` UInt8,`RefererCategories` Array(UInt16),`URLCategories` Array(UInt16),`URLRegions` Array(UInt32),`RefererRegions` Array(UInt32),`ResolutionWidth` UInt16,`ResolutionHeight` UInt16,`ResolutionDepth` UInt8,`FlashMajor` UInt8,`FlashMinor` UInt8,`FlashMinor2` String,`NetMajor` UInt8,`NetMinor` UInt8,`UserAgentMajor` UInt16,`UserAgentMinor` FixedString(2),`CookieEnable` UInt8,`JavascriptEnable` UInt8,`IsMobile` UInt8,`MobilePhone` UInt8,`MobilePhoneModel` String,`Params` String,`IPNetworkID` UInt32,`TraficSourceID` Int8,`SearchEngineID` UInt16,`SearchPhrase` String,`AdvEngineID` UInt8,`IsArtifical` UInt8,`WindowClientWidth` UInt16,`WindowClientHeight` UInt16,`ClientTimeZone` Int16,`ClientEventTime` DateTime,`SilverlightVersion1` UInt8,`SilverlightVersion2` UInt8,`SilverlightVersion3` UInt32,`SilverlightVersion4` UInt16,`PageCharset` String,`CodeVersion` UInt32,`IsLink` UInt8,`IsDownload` UInt8,`IsNotBounce` UInt8,`FUniqID` UInt64,`HID` UInt32,`IsOldCounter` UInt8,`IsEvent` UInt8,`IsParameter` UInt8,`DontCountHits` UInt8,`WithHash` UInt8,`HitColor` FixedString(1),`UTCEventTime` DateTime,`Age` UInt8,`Sex` UInt8,`Income` UInt8,`Interests` UInt16,`Robotness` UInt8,`GeneralInterests` Array(UInt16),`RemoteIP` UInt32,`RemoteIP6` FixedString(16),`WindowName` Int32,`OpenerName` Int32,`HistoryLength` Int16,`BrowserLanguage` FixedString(2),`BrowserCountry` FixedString(2),`SocialNetwork` String,`SocialAction` String,`HTTPError` UInt16,`SendTiming` Int32,`DNSTiming` Int32,`ConnectTiming` Int32,`ResponseStartTiming` Int32,`ResponseEndTiming` Int32,`FetchTiming` Int32,`RedirectTiming` Int32,`DOMInteractiveTiming` Int32,`DOMContentLoadedTiming` Int32,`DOMCompleteTiming` Int32,`LoadEventStartTiming` Int32,`LoadEventEndTiming` Int32,`NSToDOMContentLoadedTiming` Int32,`FirstPaintTiming` Int32,`RedirectCount` Int8,`SocialSourceNetworkID` UInt8,`SocialSourcePage` String,`ParamPrice` Int64,`ParamOrderID` String,`ParamCurrency` FixedString(3),`ParamCurrencyID` UInt16,`GoalsReached` Array(UInt32),`OpenstatServiceName` String,`OpenstatCampaignID` String,`OpenstatAdID` String,`OpenstatSourceID` String,`UTMSource` String,`UTMMedium` String,`UTMCampaign` String,`UTMContent` String,`UTMTerm` String,`FromTag` String,`HasGCLID` UInt8,`RefererHash` UInt64,`URLHash` UInt64,`CLID` UInt32,`YCLID` UInt64,`ShareService` String,`ShareURL` String,`ShareTitle` String,`ParsedParams` Nested(Key1 String,Key2 String,Key3 String,Key4 String,Key5 String,ValueDouble Float64),`IslandID` FixedString(16),`RequestNum` UInt32,`RequestTry` UInt8
)
ENGINE = MergeTree()
PARTITION BY toYYYYMM(EventDate)
ORDER BY (CounterID, EventDate, intHash32(UserID))
SAMPLE BY intHash32(UserID)
SETTINGS index_granularity = 8192;

CREATE TABLE tutorial.visits_v1
(`CounterID` UInt32,`StartDate` Date,`Sign` Int8,`IsNew` UInt8,`VisitID` UInt64,`UserID` UInt64,`StartTime` DateTime,`Duration` UInt32,`UTCStartTime` DateTime,`PageViews` Int32,`Hits` Int32,`IsBounce` UInt8,`Referer` String,`StartURL` String,`RefererDomain` String,`StartURLDomain` String,`EndURL` String,`LinkURL` String,`IsDownload` UInt8,`TraficSourceID` Int8,`SearchEngineID` UInt16,`SearchPhrase` String,`AdvEngineID` UInt8,`PlaceID` Int32,`RefererCategories` Array(UInt16),`URLCategories` Array(UInt16),`URLRegions` Array(UInt32),`RefererRegions` Array(UInt32),`IsYandex` UInt8,`GoalReachesDepth` Int32,`GoalReachesURL` Int32,`GoalReachesAny` Int32,`SocialSourceNetworkID` UInt8,`SocialSourcePage` String,`MobilePhoneModel` String,`ClientEventTime` DateTime,`RegionID` UInt32,`ClientIP` UInt32,`ClientIP6` FixedString(16),`RemoteIP` UInt32,`RemoteIP6` FixedString(16),`IPNetworkID` UInt32,`SilverlightVersion3` UInt32,`CodeVersion` UInt32,`ResolutionWidth` UInt16,`ResolutionHeight` UInt16,`UserAgentMajor` UInt16,`UserAgentMinor` UInt16,`WindowClientWidth` UInt16,`WindowClientHeight` UInt16,`SilverlightVersion2` UInt8,`SilverlightVersion4` UInt16,`FlashVersion3` UInt16,`FlashVersion4` UInt16,`ClientTimeZone` Int16,`OS` UInt8,`UserAgent` UInt8,`ResolutionDepth` UInt8,`FlashMajor` UInt8,`FlashMinor` UInt8,`NetMajor` UInt8,`NetMinor` UInt8,`MobilePhone` UInt8,`SilverlightVersion1` UInt8,`Age` UInt8,`Sex` UInt8,`Income` UInt8,`JavaEnable` UInt8,`CookieEnable` UInt8,`JavascriptEnable` UInt8,`IsMobile` UInt8,`BrowserLanguage` UInt16,`BrowserCountry` UInt16,`Interests` UInt16,`Robotness` UInt8,`GeneralInterests` Array(UInt16),`Params` Array(String),`Goals` Nested(ID UInt32,Serial UInt32,EventTime DateTime,Price Int64,OrderID String,CurrencyID UInt32),`WatchIDs` Array(UInt64),`ParamSumPrice` Int64,`ParamCurrency` FixedString(3),`ParamCurrencyID` UInt16,`ClickLogID` UInt64,`ClickEventID` Int32,`ClickGoodEvent` Int32,`ClickEventTime` DateTime,`ClickPriorityID` Int32,`ClickPhraseID` Int32,`ClickPageID` Int32,`ClickPlaceID` Int32,`ClickTypeID` Int32,`ClickResourceID` Int32,`ClickCost` UInt32,`ClickClientIP` UInt32,`ClickDomainID` UInt32,`ClickURL` String,`ClickAttempt` UInt8,`ClickOrderID` UInt32,`ClickBannerID` UInt32,`ClickMarketCategoryID` UInt32,`ClickMarketPP` UInt32,`ClickMarketCategoryName` String,`ClickMarketPPName` String,`ClickAWAPSCampaignName` String,`ClickPageName` String,`ClickTargetType` UInt16,`ClickTargetPhraseID` UInt64,`ClickContextType` UInt8,`ClickSelectType` Int8,`ClickOptions` String,`ClickGroupBannerID` Int32,`OpenstatServiceName` String,`OpenstatCampaignID` String,`OpenstatAdID` String,`OpenstatSourceID` String,`UTMSource` String,`UTMMedium` String,`UTMCampaign` String,`UTMContent` String,`UTMTerm` String,`FromTag` String,`HasGCLID` UInt8,`FirstVisit` DateTime,`PredLastVisit` Date,`LastVisit` Date,`TotalVisits` UInt32,`TraficSource` Nested(ID Int8,SearchEngineID UInt16,AdvEngineID UInt8,PlaceID UInt16,SocialSourceNetworkID UInt8,Domain String,SearchPhrase String,SocialSourcePage String),`Attendance` FixedString(16),`CLID` UInt32,`YCLID` UInt64,`NormalizedRefererHash` UInt64,`SearchPhraseHash` UInt64,`RefererDomainHash` UInt64,`NormalizedStartURLHash` UInt64,`StartURLDomainHash` UInt64,`NormalizedEndURLHash` UInt64,`TopLevelDomain` UInt64,`URLScheme` UInt64,`OpenstatServiceNameHash` UInt64,`OpenstatCampaignIDHash` UInt64,`OpenstatAdIDHash` UInt64,`OpenstatSourceIDHash` UInt64,`UTMSourceHash` UInt64,`UTMMediumHash` UInt64,`UTMCampaignHash` UInt64,`UTMContentHash` UInt64,`UTMTermHash` UInt64,`FromHash` UInt64,`WebVisorEnabled` UInt8,`WebVisorActivity` UInt32,`ParsedParams` Nested(Key1 String,Key2 String,Key3 String,Key4 String,Key5 String,ValueDouble Float64),`Market` Nested(Type UInt8,GoalID UInt32,OrderID String,OrderPrice Int64,PP UInt32,DirectPlaceID UInt32,DirectOrderID UInt32,DirectBannerID UInt32,GoodID String,GoodName String,GoodQuantity Int32,GoodPrice Int64),`IslandID` FixedString(16)
)
ENGINE = CollapsingMergeTree(Sign)
PARTITION BY toYYYYMM(StartDate)
ORDER BY (CounterID, StartDate, intHash32(UserID), VisitID)
SAMPLE BY intHash32(UserID)
SETTINGS index_granularity = 8192

数据导入
–max_insert_block_size=100000:表示每次批量导入100000行。

date
clickhouse-client --query "INSERT INTO tutorial.hits_v1 FORMAT TSV" --max_insert_block_size=100000 < hits_v1.tsv
dateclickhouse-client --query "INSERT INTO tutorial.visits_v1 FORMAT TSV" --max_insert_block_size=100000 < visits_v1.tsv

查询一下数据（如果想按照好的格式显示，可以在最后加上WG）：
select * from hits_v1 limit 1; \G

查询

SELECTStartURL AS URL,AVG(Duration) AS AvgDuration
FROM tutorial.visits_v1
WHERE StartDate BETWEEN '2014-03-23' AND '2014-03-30'
GROUP BY URL
ORDER BY AvgDuration DESC
LIMIT 10;SELECTsum(Sign) AS visits,sumIf(Sign, has(Goals.ID, 1105530)) AS goal_visits,(100. * goal_visits) / visits AS goal_percent
FROM tutorial.visits_v1
WHERE (CounterID = 912887) AND (toYYYYMM(StartDate) = 201403) AND (domain(StartURL) = 'yandex.ru');

2 clickhouse-client

交互式模式：clickhouse-client
非交互式模式：指定了–query参数或将数据发送至标准的stdin。

测试表创建：

create database if not exists test;
use test;
drop table test;
create table test(id UInt8, text String, created DateTime) ENGINE=TinyLog;

批模式数据导入：

echo -ne "1, 'some text', '2016-08-14 00:00:00'\n2, 'some more text', '2016-08-14 00:00:01'" | clickhouse-client --database=test --query="INSERT INTO test FORMAT CSV";cat <<_EOF | clickhouse-client --database=test --query="INSERT INTO test FORMAT CSV";
3, 'some text', '2016-08-14 00:00:00'
4, 'some more text', '2016-08-14 00:00:01'
_EOFselect * from test.test;\G

总结：
（1）、批模式下的默认数据格式为TabSeparated，可使用FORMAT指定格式。
（2）、使用–multiquery参数，可同时执行多个查询（除了INSERT），每个查询语句使用分号(;)分隔。
（3）、使用–multiline或-m参数，允许执行多行的查询。
（4）、在查询语句的分号之后指定\G或使用\G替换分号，数据以垂直的格式展示。
（5）、历史的执行语句保存在操作系统文件：~/.clickhouse-client-history。
（6）、退出客户端的方式，按Ctrl + D或Ctrl + C，或执行命令：exit、quit、logout、q、Q、:q。

查询中指定参数：

clickhouse-client --param_myid=3 --database=test --query="select * from test where id>{myid:UInt8}"
clickhouse-client --database=test --query="select * from test where id>3"

查询语句指定参数，参数值通过客户端参数传递。
使用大括号占位要传递的参数值，格式为： {:}
name：占位标识符。通过clickhouse-client参数指定，格式为–param_ = value 。
data type：指定参数值的数据类型，例如：UInt8、String等。

clickhouse-client --param_parName="[1, 2]"  -q "SELECT * FROM table WHERE a = {parName:Array(UInt16)}"clickhouse-client --query="select * from test.test FORMAT TabSeparated" > file.tsv

clickhouse-client查找配置文件的顺序：
（1）、通过–config-file指定的配置文件。
（2）、./clickhouse-client.xml
（3）、~/.clickhouse-client/config.xml
（4）、/etc/clickhouse-client/config.xml

{host}
{port}
{user}
{database}
{display_name}

3 MySQL接口操作ClickHouse

安装MySQL的客户端

tar -xzf MySQL-client-5.6.38-1.el6.x86_64.tar.gz
rpm -ivh MySQL-client-5.6.38-1.el6.x86_64.rpm

配置MySQL协议

vim /etc/clickhouse-server/config.xml
<mysql_port>9004</mysql_port>

重启clickhouse

systemctl restart clickhouse-server

命令行连接到ClickHouse：

mysql --protocol tcp -u default -P 9004

为了兼容所有MySQL客户端，建议使用SHA1加密用户密码(即修改访问密码，下面的命令直接在服务器上执行)：

PASSWORD=$(base64 < /dev/urandom | head -c8); echo "$PASSWORD"; echo -n "$PASSWORD"  \
| sha1sum | tr -d '-' | xxd -r -p | sha1sum | tr -d '-'

0FeOfptD                                    这个作为用户名
e6f07fa4a54a8d81a3fd318378311d8fd5ae8e6c    密码

修改配置文件：vim /etc/clickhouse-server/users.xml

将<password></password>注释，然后添加如下：
<password_double_sha1_hex>e6f07fa4a54a8d81a3fd318378311d8fd5ae8e6c</password_double_sha1_hex>

执行命令：

mysql --protocol tcp -u default -P 9004 --password=0FeOfptD

4 DBeaver可视化工具安装与使用

安装DBeaver （软件下载地址：https://dbeaver.io/download/ ；安装方式是直接下一步下一步的方式进行安装：）
下载DBeaver的JDBC驱动
连接ClickHouse，注意，配置CH的网络

/etc/clickhouse-server/config.xml
<listen_host>0.0.0.0</listen_host>

使用DBeaver连接ClickHouse

编辑如下内容：

点击"编辑驱动设置"

添加阿里云的源

URL:http://maven.aliyun.com/nexus/content/groups/public/
然后点击"应用"，“Apply and Close”，一直点击到最后。

在"SQL编辑器”里，可以点击"SQL编辑器"。在编辑器中可以编写sql语句。

设置参数
在URL模板中，设置ClickHouse的参数

jdbc:clickhouse://{host}:{port}[/{database}]?socket_timeout=600000&max_bytes_before_external_group_by=20000000

配置参数：

timeout
socket_timeout

外部聚合，如果用于聚合内存不足，10G

max_bytes_before_external_group_by=5G