k8s部署Archery-SQL审核查询平台

介绍：

Archery是archer的分支项目，定位于SQL审核查询平台，旨在提升DBA的工作效率，
支持多数据库的SQL上线和查询，同时支持丰富的MySQL运维功能，所有功能都兼容手机端操作

项目地址：https://github.com/hhyo/Archery

部署archery :

项目本身不支持以环境变量的方式传值识别，可以拉取到本地进行适当的修改再重新进行打包，或者进行将文件挂载出来也可以，皆可。

需修改文件位置：

https://github.com/hhyo/Archery/blob/master/archery/settings.py

# -*- coding: UTF-8 -*-# Build paths inside the project like this: os.path.join(BASE_DIR, ...)
import osBASE_DIR = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))# SECURITY WARNING: keep the secret key used in production secret!
SECRET_KEY = 'hfusaf2m4ot#7)fkw#di2bu6(cv0@opwmafx5n#6=3d%x^hpl6'# SECURITY WARNING: don't run with debug turned on in production!
DEBUG = TrueALLOWED_HOSTS = ['*']# 解决nginx部署跳转404
USE_X_FORWARDED_HOST = True# 请求限制
DATA_UPLOAD_MAX_MEMORY_SIZE = 15728640# Application definition
INSTALLED_APPS = ('django.contrib.admin','django.contrib.auth','django.contrib.contenttypes','django.contrib.sessions','django.contrib.messages','django.contrib.staticfiles','django_q','sql','sql_api','common',
)MIDDLEWARE = ('django.contrib.sessions.middleware.SessionMiddleware','django.middleware.common.CommonMiddleware','django.middleware.csrf.CsrfViewMiddleware','django.contrib.auth.middleware.AuthenticationMiddleware','django.contrib.messages.middleware.MessageMiddleware','django.middleware.clickjacking.XFrameOptionsMiddleware','django.middleware.security.SecurityMiddleware','django.middleware.gzip.GZipMiddleware','common.middleware.check_login_middleware.CheckLoginMiddleware','common.middleware.exception_logging_middleware.ExceptionLoggingMiddleware',
)ROOT_URLCONF = 'archery.urls'TEMPLATES = [{'BACKEND': 'django.template.backends.django.DjangoTemplates','DIRS': [os.path.join(BASE_DIR, 'common/templates')],'APP_DIRS': True,'OPTIONS': {'context_processors': ['django.template.context_processors.debug','django.template.context_processors.request','django.contrib.auth.context_processors.auth','django.contrib.messages.context_processors.messages','common.utils.global_info.global_info',],},},
]WSGI_APPLICATION = 'archery.wsgi.application'# Internationalization
LANGUAGE_CODE = 'zh-hans'TIME_ZONE = 'Asia/Shanghai'USE_I18N = TrueUSE_TZ = False# 时间格式化
USE_L10N = False
DATETIME_FORMAT = 'Y-m-d H:i:s'
DATE_FORMAT = 'Y-m-d'# Static files (CSS, JavaScript, Images)
STATIC_URL = '/static/'
STATIC_ROOT = os.path.join(BASE_DIR, 'static')
STATICFILES_DIRS = [os.path.join(BASE_DIR, 'common/static'), ]
STATICFILES_STORAGE = 'common.storage.ForgivingManifestStaticFilesStorage'# 扩展django admin里users字段用到，指定了sql/models.py里的class users
AUTH_USER_MODEL = "sql.Users"# 密码校验
AUTH_PASSWORD_VALIDATORS = [{'NAME': 'django.contrib.auth.password_validation.UserAttributeSimilarityValidator',},{'NAME': 'django.contrib.auth.password_validation.MinimumLengthValidator','OPTIONS': {'min_length': 9,}},{'NAME': 'django.contrib.auth.password_validation.CommonPasswordValidator',},{'NAME': 'django.contrib.auth.password_validation.NumericPasswordValidator',},
]############### 以下部分需要用户根据自己环境自行修改 #################### SESSION 设置
SESSION_COOKIE_AGE = 60 * 300  # 300分钟
SESSION_SAVE_EVERY_REQUEST = True
SESSION_EXPIRE_AT_BROWSER_CLOSE = True  # 关闭浏览器，则COOKIE失效# 该项目本身的mysql数据库地址
DATABASES = {'default': {'ENGINE': 'django.db.backends.mysql','NAME': os.environ.get('MYSQL_NAME','archery'),'USER': os.environ.get('MYSQL_USER','archery'),'PASSWORD': os.environ.get('MYSQL_PWD','password'),'HOST': os.environ.get('MYSQL_HOST'),'PORT': os.environ.get('MYSQL_PORT','3306'),'OPTIONS': {'init_command': "SET sql_mode='STRICT_TRANS_TABLES'",'charset': 'utf8mb4'},'TEST': {'NAME': 'test_archery','CHARSET': 'utf8mb4',},}
}# Django-Q
Q_CLUSTER = {'name': 'archery','workers': 4,'recycle': 500,'timeout': 60,'compress': True,'cpu_affinity': 1,'save_limit': 0,'queue_limit': 50,'label': 'Django Q','django_redis': 'default','sync': False  # 本地调试可以修改为True，使用同步模式
}# 缓存配置
CACHES = {"default": {"BACKEND": "django_redis.cache.RedisCache","LOCATION": os.environ.get('REDIS_LOCATION1'),"OPTIONS": {"CLIENT_CLASS": "django_redis.client.DefaultClient","PASSWORD": os.environ.get('REDIS_PWD'),}},"dingding": {"BACKEND": "django_redis.cache.RedisCache","LOCATION": os.environ.get('REDIS_LOCATION2'),"OPTIONS": {"CLIENT_CLASS": "django_redis.client.DefaultClient","PASSWORD": os.environ.get('REDIS_PWD'),}}
}# LOG配置
LOGGING = {'version': 1,'disable_existing_loggers': False,'formatters': {'verbose': {'format': '[%(asctime)s][%(threadName)s:%(thread)d][task_id:%(name)s][%(filename)s:%(lineno)d][%(levelname)s]- %(message)s'},},'handlers': {'default': {'level': 'DEBUG','class': 'logging.handlers.RotatingFileHandler','filename': 'logs/archery.log','maxBytes': 1024 * 1024 * 100,  # 5 MB'backupCount': 5,'formatter': 'verbose',},'django-q': {'level': 'DEBUG','class': 'logging.handlers.RotatingFileHandler','filename': 'logs/qcluster.log','maxBytes': 1024 * 1024 * 100,  # 5 MB'backupCount': 5,'formatter': 'verbose',},'console': {'level': 'DEBUG','class': 'logging.StreamHandler','formatter': 'verbose'}},'loggers': {'default': {  # default日志'handlers': ['console', 'default'],'level': 'WARNING'},'django-q': {  # django_q模块相关日志'handlers': ['console', 'django-q'],'level': 'WARNING','propagate': False},'django_auth_ldap': {  # django_auth_ldap模块相关日志'handlers': ['console', 'default'],'level': 'WARNING','propagate': False},# 'django.db': {  # 打印SQL语句，方便开发#     'handlers': ['console', 'default'],#     'level': 'DEBUG',#     'propagate': False# },# 'django.request': {  # 打印请求错误堆栈信息，方便开发#     'handlers': ['console', 'default'],#     'level': 'DEBUG',#     'propagate': False# },}
}

注：本配置缺少 openLDAP相关配置，因为公司内openLDAP是一成一变的，不需要再次修改，而数据库和Redis 是因为为了之后迁移时会方便很多，故以变量形式。

打包时的需要的 Dockerfile 文件：

drifter@localhost archery % more Dockerfile
FROM hhyo/archery:1.8.1WORKDIR /opt/archeryCOPY . /opt/archery/
COPY settings.py /opt/archery/archery/
COPY soar.yaml /etc/
#port
EXPOSE 9123#start service
ENTRYPOINT bash /opt/archery/startup.sh && bash

注：此Dockerfile 是基于原有的archery 进行文件覆盖式的.
你也可以根据官方已经自己重新打基础镜像
https://github.com/hhyo/Archery/tree/master/src/docker

more  settings.py
# -*- coding: UTF-8 -*-# Build paths inside the project like this: os.path.join(BASE_DIR, ...)
import osBASE_DIR = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))# SECURITY WARNING: keep the secret key used in production secret!
SECRET_KEY = 'hfusaf2m4ot#7)fkw#di2bu6(cv0@opwmafx5n#6=3d%x^hpl6'# SECURITY WARNING: don't run with debug turned on in production!
DEBUG = TrueALLOWED_HOSTS = ['*']# 解决nginx部署跳转404
USE_X_FORWARDED_HOST = True# 请求限制
DATA_UPLOAD_MAX_MEMORY_SIZE = 15728640# Application definition
INSTALLED_APPS = ('django.contrib.admin','django.contrib.auth','django.contrib.contenttypes','django.contrib.sessions','django.contrib.messages','django.contrib.staticfiles','django_q','sql','sql_api','common',
)MIDDLEWARE = ('django.contrib.sessions.middleware.SessionMiddleware','django.middleware.common.CommonMiddleware','django.middleware.csrf.CsrfViewMiddleware','django.contrib.auth.middleware.AuthenticationMiddleware','django.contrib.messages.middleware.MessageMiddleware','django.middleware.clickjacking.XFrameOptionsMiddleware','django.middleware.security.SecurityMiddleware','django.middleware.gzip.GZipMiddleware','common.middleware.check_login_middleware.CheckLoginMiddleware','common.middleware.exception_logging_middleware.ExceptionLoggingMiddleware',
)ROOT_URLCONF = 'archery.urls'TEMPLATES = [{'BACKEND': 'django.template.backends.django.DjangoTemplates','DIRS': [os.path.join(BASE_DIR, 'common/templates')],'APP_DIRS': True,'OPTIONS': {'context_processors': ['django.template.context_processors.debug','django.template.context_processors.request','django.contrib.auth.context_processors.auth','django.contrib.messages.context_processors.messages','common.utils.global_info.global_info',],},},
]WSGI_APPLICATION = 'archery.wsgi.application'# Internationalization
LANGUAGE_CODE = 'zh-hans'TIME_ZONE = 'Asia/Shanghai'USE_I18N = TrueUSE_TZ = False# 时间格式化
USE_L10N = False
DATETIME_FORMAT = 'Y-m-d H:i:s'
DATE_FORMAT = 'Y-m-d'# Static files (CSS, JavaScript, Images)
STATIC_URL = '/static/'
STATIC_ROOT = os.path.join(BASE_DIR, 'static')
STATICFILES_DIRS = [os.path.join(BASE_DIR, 'common/static'), ]
STATICFILES_STORAGE = 'common.storage.ForgivingManifestStaticFilesStorage'# 扩展django admin里users字段用到，指定了sql/models.py里的class users
AUTH_USER_MODEL = "sql.Users"# 密码校验
AUTH_PASSWORD_VALIDATORS = [{'NAME': 'django.contrib.auth.password_validation.UserAttributeSimilarityValidator',},{'NAME': 'django.contrib.auth.password_validation.MinimumLengthValidator','OPTIONS': {'min_length': 9,}},{'NAME': 'django.contrib.auth.password_validation.CommonPasswordValidator',},{'NAME': 'django.contrib.auth.password_validation.NumericPasswordValidator',},
]############### 以下部分需要用户根据自己环境自行修改 #################### SESSION 设置
SESSION_COOKIE_AGE = 60 * 300  # 300分钟
SESSION_SAVE_EVERY_REQUEST = True
SESSION_EXPIRE_AT_BROWSER_CLOSE = True  # 关闭浏览器，则COOKIE失效# 该项目本身的mysql数据库地址
DATABASES = {'default': {'ENGINE': 'django.db.backends.mysql','NAME': os.environ.get('MYSQL_NAME','archery'),'USER': os.environ.get('MYSQL_USER','archery'),'PASSWORD': os.environ.get('MYSQL_PWD','password'),'HOST': os.environ.get('MYSQL_HOST'),'PORT': os.environ.get('MYSQL_PORT','3306'),'OPTIONS': {'init_command': "SET sql_mode='STRICT_TRANS_TABLES'",'charset': 'utf8mb4'},'TEST': {'NAME': 'test_archery','CHARSET': 'utf8mb4',},}
}# Django-Q
Q_CLUSTER = {'name': 'archery','workers': 4,'recycle': 500,'timeout': 60,'compress': True,'cpu_affinity': 1,'save_limit': 0,'queue_limit': 50,'label': 'Django Q','django_redis': 'default','sync': False  # 本地调试可以修改为True，使用同步模式
}# 缓存配置
CACHES = {"default": {"BACKEND": "django_redis.cache.RedisCache","LOCATION": os.environ.get('REDIS_LOCATION1'),"OPTIONS": {"CLIENT_CLASS": "django_redis.client.DefaultClient","PASSWORD": os.environ.get('REDIS_PWD'),}},"dingding": {"BACKEND": "django_redis.cache.RedisCache","LOCATION": os.environ.get('REDIS_LOCATION2'),"OPTIONS": {"CLIENT_CLASS": "django_redis.client.DefaultClient","PASSWORD": os.environ.get('REDIS_PWD'),}}
}# LOG配置
LOGGING = {'version': 1,'disable_existing_loggers': False,'formatters': {'verbose': {'format': '[%(asctime)s][%(threadName)s:%(thread)d][task_id:%(name)s][%(filename)s:%(lineno)d][%(levelname)s]- %(message)s'},},'handlers': {'default': {'level': 'DEBUG','class': 'logging.handlers.RotatingFileHandler','filename': 'logs/archery.log','maxBytes': 1024 * 1024 * 100,  # 5 MB'backupCount': 5,'formatter': 'verbose',},'django-q': {'level': 'DEBUG','class': 'logging.handlers.RotatingFileHandler','filename': 'logs/qcluster.log','maxBytes': 1024 * 1024 * 100,  # 5 MB'backupCount': 5,'formatter': 'verbose',},'console': {'level': 'DEBUG','class': 'logging.StreamHandler','formatter': 'verbose'}},'loggers': {'default': {  # default日志'handlers': ['console', 'default'],'level': 'WARNING'},'django-q': {  # django_q模块相关日志'handlers': ['console', 'django-q'],'level': 'WARNING','propagate': False},'django_auth_ldap': {  # django_auth_ldap模块相关日志'handlers': ['console', 'default'],'level': 'WARNING','propagate': False},# 'django.db': {  # 打印SQL语句，方便开发#     'handlers': ['console', 'default'],#     'level': 'DEBUG',#     'propagate': False# },# 'django.request': {  # 打印请求错误堆栈信息，方便开发#     'handlers': ['console', 'default'],#     'level': 'DEBUG',#     'propagate': False# },}
}

  more  soar.yaml # 是否允许测试环境与线上环境配置相同
allow-online-as-test: false
# 是否清理测试时产生的临时文件
drop-test-temporary: true
# 语法检查小工具
only-syntax-check: false
sampling-data-factor: 100
sampling: false
sampling-statistic-target: 100
profiling: false
trace: false
# 日志级别，[0:Emergency, 1:Alert, 2:Critical, 3:Error, 4:Warning, 5:Notice, 6:Informational, 7:Debug]
log-level: 3
log-output: /opt/archery/logs/soar.log
# 优化建议输出格式
report-type: markdown
ignore-rules:
- ""
# 启发式算法相关配置
max-join-table-count: 5
max-group-by-cols-count: 5
max-distinct-count: 5
max-index-cols-count: 5
max-total-rows: 9999999
spaghetti-query-length: 2048
allow-drop-index: false
# EXPLAIN相关配置
explain-sql-report-type: pretty
explain-type: extended
explain-format: traditional
explain-warn-select-type:
- ""
explain-warn-access-type:
- ALL
explain-max-keys: 3
explain-min-keys: 0
explain-max-rows: 10000
explain-warn-extra:
- ""
explain-max-filtered: 100
explain-warn-scalability:
- O(n)
query: ""
list-heuristic-rules: false
list-test-sqls: false
verbose: true

以上是 Dockerfile 文件所需的文件。

archery-deployment.yaml

more   archery-deployment.yaml
apiVersion: apps/v1
kind: Deployment
metadata:name: archerynamespace: mysql-config
spec:replicas: 1selector:matchLabels:k8s-app: archerytemplate:metadata:labels:k8s-app: archeryspec:imagePullSecrets:- name: registry-comcontainers:- name: archeryimage: registry.drifter.com/archery:1.8.0imagePullPolicy: Alwaysports:- containerPort: 9123name: archeryenv:- name: MYSQL_DBvalue: "archery"- name: MYSQL_HOSTvalue: "mysql_host"- name: MYSQL_PWDvalue: "{drifter@archery&qwed)"- name: MYSQL_USERvalue: "archery"- name: REDIS_LOCATION1value: "redis_url"- name: REDIS_LOCATION2value: "redis_url"- name: REDIS_PWDvalue: "Psxmsxis@isnxs71Ws"- name: LDAP_FLAGvalue: "False"

archery-svc.yaml

more  archery-svc.yaml
kind: Service
apiVersion: v1
metadata:labels:k8s-app: archeryname: archerynamespace: mysql-config
spec:sessionAffinity: ClientIPsessionAffinityConfig:clientIP:timeoutSeconds: 10800selector:k8s-app: archeryports:- protocol: TCPport: 9123name: archerytype: ClusterIP

archery-ing.yaml

more   archery-ing.yaml
apiVersion: extensions/v1beta1
kind: Ingress
metadata:name: archerynamespace: mysql-configannotations:nginx.ingress.kubernetes.io/whitelist-source-range: 'IP' ## (公司内部出口IP)spec:rules:- host: sql.drifter.comhttp:paths:- backend:serviceName: archeryservicePort: 9123path: /tls:- hosts:- sql.drifter.comsecretName: drifter-com
status:loadBalancer:ingress:

部署goinception 审核规则:

审核规则详细查看 https://hanchuanchuan.github.io/goInception/options.html

goinception Dockerfile

FROM hanchuanchuan/goinception:latest
WORKDIR /opt/archery/src/docker-compose/inception/
COPY config.toml /etc/

goinception config.toml

host = "0.0.0.0"advertise_address = ""port = 4000##  审核规则详情查看
## https://hanchuanchuan.github.io/goInception/options.html
[inc]
check_autoincrement_datatype            =true
check_autoincrement_init_value          =true
check_autoincrement_name                =true
check_column_comment                    =false
check_column_default_value              =false
check_column_position_change            =false
check_column_type_change                =true
check_dml_limit                         =true
check_dml_orderby                       =true
check_dml_where                         =false
check_identifier                        =true
check_index_prefix                      =true
check_insert_field                      =true
check_primary_key                       =true
check_table_comment                     =true
check_timestamp_default                 =true
check_timestamp_count                   =false
enable_autoincrement_unsigned           =false
enable_blob_type                        =true
enable_column_charset                   =true
enable_drop_database                    =true
enable_drop_table                       =true
enable_enum_set_bit                     =false
enable_fingerprint                      =true
enable_foreign_key                      =false
enable_json_type                        =true
enable_identifer_keyword                =false
enable_nullable                         =true
enable_null_index_name                  =false
enable_orderby_rand                     =true
enable_partition_table                  =true
enable_pk_columns_only_int              =true
enable_select_star                      =false
enable_set_charset                      =true
enable_set_collation                    =true
enable_set_engine                       =true
max_char_length                         =0
max_insert_rows                         =0
max_keys                                =5
max_key_parts                           =5
max_update_rows                         =5000
max_primary_key_parts                   =1
max_allowed_packet                      =33554432
merge_alter_table                       =false
check_float_double                      =false
support_charset                         ="utf8,utf8mb4"
support_collation                       ="utf8mb4_unicode_ci"backup_host = "mysql_url"
backup_port = 3306
backup_user = "archery"
backup_password = "****"lock_wait_timeout = 10sql_safe_updates = -1lang = "zh-CN"general_log = falseenable_sql_statistic = true[inc_level]
er_cant_set_engine = 2
er_cant_set_collation = 2
er_table_must_have_comment = 2
er_column_have_no_comment = 2
er_table_must_have_pk = 2
er_index_name_idx_prefix = 1
er_index_name_uniq_prefix =  1
er_autoinc_unsigned = 2
er_alter_table_once = 2
er_pk_too_many_parts = 2[osc]
osc_on = false
osc_min_table_size = 16
osc_print_none = false
osc_bin_dir = "/usr/local/bin"[ghost]ghost_on = false
ghost_allow_on_master = true
ghost_assume_rbr = true
ghost_chunk_size = 1000
ghost_concurrent_rowcount = true
ghost_cut_over = "atomic"
ghost_cut_over_lock_timeout_seconds = 3
ghost_default_retries = 60
ghost_heartbeat_interval_millis = 500
ghost_max_lag_millis = 1500
ghost_approve_renamed_columns = true
ghost_exponential_backoff_max_interval = 64
ghost_dml_batch_size = 10[log]
level = "error"
format = "console"

goinception-deployment.yaml

apiVersion: apps/v1
kind: Deployment
metadata:name: goinceptionnamespace: mysql-config
spec:replicas: 1selector:matchLabels:k8s-app: goinceptiontemplate:metadata:labels:k8s-app: goinceptionspec:imagePullSecrets:- name: registry-comcontainers:- name: goinceptionimage: registry.drifter.com/goinception:1.8.1imagePullPolicy: Alwaysports:- containerPort: 4000name: goinception

goinception-svc.yaml

kind: Service
apiVersion: v1
metadata:labels:k8s-app: goinceptionname: goinceptionnamespace: mysql-config
spec:sessionAffinity: ClientIPsessionAffinityConfig:clientIP:timeoutSeconds: 10800selector:k8s-app: goinceptionports:- protocol: TCPport: 4000name: goinceptiontype: ClusterIP

部署 inception 审核规则:

详细 inception相关查看 https://github.com/cookieY/inception-document/blob/master/docs/variables.md

inception Dockerfile

FROM hhyo/inception:latest
WORKDIR /opt/archery/src/docker-compose/inception/
COPY inc.cnf /etc/

inception inc.cnf 文件：

[inception]
general_log=1
general_log_file=inception.log
port=6669
socket=/tmp/inc.socket
character-set-client-handshake=0
character-set-server=utf8mb4
inception_language_code=zh-CN
inception_remote_system_password='****'
inception_remote_system_user=archery
inception_remote_backup_port=3306
inception_remote_backup_host=mysql-host
inception_support_charset=utf8,utf8mb4
inception_enable_nullable=0
inception_check_primary_key=1
inception_check_column_comment=1
inception_check_table_comment=1
inception_osc_on=OFF
inception_osc_bin_dir=/usr/bin
inception_osc_min_table_size=10
inception_osc_chunk_time=0.1
inception_enable_blob_type=1
inception_check_column_default_value=1inception_enable_select_star=ON
inception_enable_identifer_keyword=ON
inception_enable_autoincrement_unsigned=ON
inception_check_identifier=OFF## 审核规则说明
## https://github.com/cookieY/inception-document/blob/master/docs/variables.md

inception-deployment.yaml

apiVersion: apps/v1
kind: Deployment
metadata:name: inceptionnamespace: mysql-config
spec:replicas: 1selector:matchLabels:k8s-app: inceptiontemplate:metadata:labels:k8s-app: inceptionspec:imagePullSecrets:- name: registry-comcontainers:- name: inceptionimage: registry.drifter.com/inception:1.8.0imagePullPolicy: Alwaysports:- containerPort: 6669name: inception

inception-svc.yaml

kind: Service
apiVersion: v1
metadata:labels:k8s-app: inceptionname: inceptionnamespace: mysql-config
spec:sessionAffinity: ClientIPsessionAffinityConfig:clientIP:timeoutSeconds: 10800selector:k8s-app: inceptionports:- protocol: TCPport: 6669name: inceptiontype: ClusterIP

依次启动相关服务后，访问 http://sql.drifter.net 使用账号密码登录

其他的就不多说了，看官方帮助文档很清楚咯。