如何让novnc/websockify支持tls1.2 (by quqi99)

作者：张华发表于：2020-09-10
版权声明：可以任意转载，转载时请务必以超链接形式标明文章原始出处和作者信息及本版权声明

如果希望novnc使用tls1.2该怎么办？
各软件(openssl, websockify, nova)支持tls1.2的历史情况如下：

openssl在1.0.0h到1.0.1的时候才开始支持tls1.2, 见"Add TLS v1.2 server support for client authentication" - https://www.openssl.org/news/changelog.html，xenial使用的是1.0.2g版本
ubuntu 20.04开始默认使用tls1.2 [2]
websockify在0.9.0才开始支持配置tls版本, 见 - https://pypi.org/project/websockify/, 所以在nova xenial/mitaka版本中因为使用了websockify0.8.0所以列代码也不支持配置tls版本

# grep -r 'ssl.wrap_sock' /usr/lib/python2.7/dist-packages/nova/console/rfb/authvencrypt.py -A 7wrapped_sock = ssl.wrap_socket(compute_sock,keyfile=client_key,certfile=client_cert,server_side=False,cert_reqs=ssl.CERT_REQUIRED,ca_certs=CONF.vnc.vencrypt_ca_certs)

这样，后来在nova升级到websockify 0.9.0之后也开始有了一个patch支持配置tls版本,见
https://review.opendev.org/#/c/679502/
但是，目前我们使用的就是xenial，该如何办呢？
首先，因为使用的是websockify 0.8.0不支持配置ssl版本，那样想在ssl.wrap_socket中直接指定＂ssl_version=ssl.PROTOCOL_TLSv1_2＂来强制使用tls1_2是可行的．（如果不可行，是因为下列错误导致，见： https://github.com/freedesktop/spice-html5/commit/293d405e15a4499219fe81e830862cc2b1518e3e )

SecurityError: Failed to construct 'WebSocket': An insecure WebSocket connection may not be initiated from a page loaded over HTTPS.

那样，是否应该改变底层openssl的默认配置成tls1.2, 参考文档[1], 修改/etc/ssl/openssl.cnf在oid_section之后添加下列内容

root@juju-055b8b-ssl-7:~# grep -r 'oid_section' /etc/ssl/openssl.cnf -A 2
oid_section             = new_oidsopenssl_conf = default_confroot@juju-055b8b-ssl-7:~# cat /etc/ssl/openssl.cnf |tail -n9
[default_conf]
ssl_conf = ssl_sect[ssl_sect]
system_default = system_default_sect[system_default_sect]
MinProtocol = TLSv1.2
CipherString = DEFAULT@SECLEVEL=2

但是使用下列方法测试时报错．

systemctl restart nova-novncproxy nova-consoleauth
# verify if tls 1.2 is supported - https://devanswers.co/test-server-tls-1-2-ubuntu/
# https://www.poftut.com/use-openssl-s_client-check-verify-ssltls-https-webserver/
OPENSSL_CONF=/etc/ssl/ openssl s_client -connect 10.5.100.4:6080 -tls1
OPENSSL_CONF=/etc/ssl/ openssl s_client -connect 10.5.100.4:6080 -tls1_2#list all supported ciphers
nmap --script ssl-enum-ciphers -p 6080 10.5.100.4
OPENSSL_CONF=/etc/ssl/ openssl s_client -connect 10.5.2.196:6082 -tls1_2

root@juju-055b8b-ssl-7:~# openssl s_client -connect 10.5.100.4:6080 -tls1_2
Error configuring OpenSSL
139929571108504:error:25066067:DSO support routines:DLFCN_LOAD:could not load the shared library:dso_dlfcn.c:187:filename(libssl_conf.so): libssl_conf.so: cannot open shared object file: No such file or directory
139929571108504:error:25070067:DSO support routines:DSO_load:could not load the shared library:dso_lib.c:233:
139929571108504:error:0E07506E:configuration file routines:MODULE_LOAD_DSO:error loading dso:conf_mod.c:271:module=ssl_conf, path=ssl_conf
139929571108504:error:0E076071:configuration file routines:MODULE_RUN:unknown module name:conf_mod.c:212:module=ssl_conf

错误是找不着libssl_conf.so，作下列更改后，问题依旧．

sudo apt-get install libssl1.0.0 libssl-dev
cd /lib/x86_64-linux-gnu/
sudo ln -s libssl.so.1.0.0 libssl.so.10
sudo ln -s libcrypto.so.1.0.0 libcrypto.so.10

20201120更新-是需要添加OPENSSL_CONF=/etc/ssl/

# OPENSSL_CONF=/etc/ssl/ openssl ciphers -v TLSv1.2 | head -4
ECDHE-RSA-AES256-GCM-SHA384 TLSv1.2 Kx=ECDH     Au=RSA  Enc=AESGCM(256) Mac=AEAD
ECDHE-ECDSA-AES256-GCM-SHA384 TLSv1.2 Kx=ECDH     Au=ECDSA Enc=AESGCM(256) Mac=AEAD
ECDHE-RSA-AES256-SHA384 TLSv1.2 Kx=ECDH     Au=RSA  Enc=AES(256)  Mac=SHA384
ECDHE-ECDSA-AES256-SHA384 TLSv1.2 Kx=ECDH     Au=ECDSA Enc=AES(256)  Mac=SHA384

将xenial上的openssl 1.1.0g升级到1.1.1a后问题依旧:

#upgrade openssl from 1.1.0g to 1.1.1a in ubuntu 18.04
sudo apt install build-essential checkinstall zlib1g-dev gcc make -y
wget https://www.openssl.org/source/openssl-1.1.1a.tar.gz
tar zxvf openssl-1.1.1a.tar.gz && cd openssl-1.1.1a/
./config --prefix=/usr/local/ssl --openssldir=/usr/local/ssl
make
sudo make install
sudo bash -c 'cat > /etc/ld.so.conf.d/openss1-1.1.1b.conf' << EOF
/usr/local/ssl/lib
EOF
sudo ldconfig -v
sudo mv /usr/bin/c_rehash /usr/bin/c_rehash.BAK
sudo mv /usr/bin/openssl /usr/bin/openssl.BAK
sudo cp /etc/environment /etc/environment.BAK
sudo bash -c 'cat > /etc/environment' << EOF
PATH="/usr/local/ssl/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games"
EOF
echo $PATH
openssl version -a
sudo apt-get install libssl-dev -y

然后修改/usr/lib/python2.7/dist-packages/websockify/websocket.py（注意，不是/usr/lib/python2.7/dist-packages/nova/console/rfb/authvencrypt.py）里的ssl.wrap_socket添加’ssl_version=ssl.PROTOCOL_TLSv1_2’就可以了，

# nmap --script ssl-enum-ciphers -p 6080 10.5.100.4 |grep TLSv
|   TLSv1.2:
root@juju-055b8b-ssl-7:~/openssl-1.1.1a# openssl s_client -connect 10.5.100.4:6080 -tls1_2 |grep 'CN'
depth=0 C = GB, ST = England, L = London, O = Ubuntu Cloud, OU = Cloud, CN = 10.5.100.0
...

此时，去掉/usr/lib/python2.7/dist-packages/nova/console/rfb/authvencrypt.py里之前做的tls1.2修改也是可以的．
结论，似乎修改底层openssl的默认版本到tls1.2不好使，同时websockify 0.8.0默认的只是tls1.0, 此时只是修改nova端是不work的，直接修改websockify 0.8.0改到tls1.2是可以的．在websockify 0.9.0之后支持配置tls1.2，所以此时nova端修改tls的patch也就能派上用场了．
相关代码分析：
python ssl模块在wrap_socket中是将参数硬编码成ssl_version=PROTOCOL_SSLv23，所以在websockify 0.8.0不传ssl_version参数的话是即使改底层openssl的默认编码也是无济于事的．

def wrap_socket(sock, keyfile=None, certfile=None,server_side=False, cert_reqs=CERT_NONE,ssl_version=PROTOCOL_SSLv23, ca_certs=None,do_handshake_on_connect=True,suppress_ragged_eofs=True, ciphers=None):

看来唯一的办法是修改python-websockify 0.8.0包，加上那一行，做个临时hotfix了．

20201106更新

创建了一个xenial的spice ssl测试环境．

./generate-bundle.sh -s xenial -r queens --create-model --name ssl2:stsstack --num-compute 1 --openstack-dashboard --ssl --nova-console --run
juju config openstack-dashboard enforce-ssl=true
ssl_results=/home/ubuntu/ed/stsstack-bundles/openstack/ssl/openstack-ssl2/results
juju config openstack-dashboard ssl_ca=`base64 ${ssl_results}/cacert.pem| tr -d '\n'`
juju config openstack-dashboard ssl_cert=`base64 ${ssl_results}/servercert.pem| tr -d '\n'`
juju config openstack-dashboard ssl_key=`base64 ${ssl_results}/serverkey.pem| tr -d '\n'`
juju config nova-cloud-controller console-access-protocol=spice
juju config nova-cloud-controller console-ssl-cert=`base64 ${ssl_results}/servercert.pem| tr -d '\n'`
juju config nova-cloud-controller console-ssl-key=`base64 ${ssl_results}/serverkey.pem| tr -d '\n'`

但我们尽量不用–ssl，改用–vault

./generate-bundle.sh -s xenial -r queens --create-model --name ssl-queens:stsstack --num-compute 1 --nova-console --vault

看到下列错，实际上这是正常的．

$ nova get-vnc-console bionic-061058 spice-html5
+-------------+------------------------------------------------------------------------------------+
| Type | Url |
+-------------+------------------------------------------------------------------------------------+
| spice-html5 | https://10.5.100.4:6082/spice_auto.html?token=69b15db8-2575-4bc9-980b-3e4149881015 |
+-------------+------------------------------------------------------------------------------------+
$ curl -k -vvv https://10.5.100.4:6082/spice_auto.html?token=69b15db8-2575-4bc9-980b-3e4149881015
* Trying 10.5.100.4:6082...
* TCP_NODELAY set
* Connected to 10.5.100.4 (10.5.100.4) port 6082 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* successfully set certificate verify locations:
* CAfile: /etc/ssl/certs/ca-certificates.crt
CApath: /etc/ssl/certs
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
* TLSv1.3 (IN), TLS handshake, Server hello (2):
* TLSv1.3 (OUT), TLS alert, protocol version (582):
* error:1425F102:SSL routines:ssl_choose_client_version:unsupported protocol
* Closing connection 0
curl: (35) error:1425F102:SSL routines:ssl_choose_client_version:unsupported protocol-root@juju-1833ad-ssl2-7:~# tail -f /var/log/nova/nova-spiceproxy.log
2020-11-06 06:42:33.666 26162 DEBUG nova.console.websocketproxy [-] 10.5.0.8: new handler Process vmsg /usr/lib/python2.7/dist-packages/websockify/websocket.py:878
2020-11-06 06:42:34.166 19170 INFO nova.console.websocketproxy [-] handler exception: [SSL: SSLV3_ALERT_CERTIFICATE_UNKNOWN] sslv3 alert certificate unknown (_ssl.c:590)

为什么说它是正常的呢？因为我们是在focal上运行的curl命令或者浏览器，focal支持的最低ssl版本是tlsv1.2，而spiceproxy运行在xenial只支持tlsv1.0.　所以找一台xenial机器运行下列加了–tlsv1.0的curl命令是OK的。

curl -k -vvv https://10.5.100.4:6082/spice_auto.html?token=bd00bb9f-83e7-4b25-9b6a-57dde9941dce --tlsv1.0

还是上面类似的老问题，spice也用到了websockify0.8，而它写死了 ssl_version=PROTOCOL_TLSv1

> /usr/lib/python2.7/dist-packages/websockify/websocket.py(837)do_handshake()
-> retsock = ssl.wrap_socket(
(Pdb) l
832                     raise self.EClose("SSL connection but '%s' not found"
833                                       % self.cert)
834                 retsock = None
835                 try:
836                     import rpdb;rpdb.set_trace()
837  ->                 retsock = ssl.wrap_socket(
838                             sock,
839                             server_side=True,
840                             certfile=self.cert,
841                             keyfile=self.key)
842                 except ssl.SSLError:
(Pdb) p self.cert
'/etc/nova/ssl/nova_cert.pem'
(Pdb) p self.key
'/etc/nova/ssl/nova_key.pem'
(Pdb) l48  49         def __init__(self, sock, keyfile=None, certfile=None,50                      server_side=False, cert_reqs=CERT_NONE,51                      ssl_version=PROTOCOL_TLSv1, ca_certs=None,52                      do_handshake_on_connect=True, *args, **kw):53  ->         if not isinstance(sock, GreenSocket):54                 sock = GreenSocket(sock)55  56             self.act_non_blocking = sock.act_non_blocking57  58             if six.PY2:root@juju-1833ad-ssl2-7:~# pip list |grep web
websockify (0.8.0)

但直接在xenial上升级到websockify=0.9.0失败.

在queens中使用websockify 0.9.0

将websockify的patch backedport到0.8.0时依赖太多，所以只能将0.9.0整体backport到queens. 测试一下：

mv /usr/lib/python2.7/dist-packages/websockify ~/websockify_bak
git clone https://github.com/novnc/websockify.git
cd websockify && git checkout -b v0.9.0 v0.9.0
cd ../ && cp -r websockify/websockify /usr/lib/python2.7/dist-packages/cp allow-TLS-ciphers-protocols-to-be-configurable-for-c.patch /usr/lib/python2.7/dist-packages/
cd  /usr/lib/python2.7/dist-packages/ && patch -p1 <allow-TLS-ciphers-protocols-to-be-configurable-for-c.patchthen set the following content in /etc/nova/nova.conf
[console]
ssl_minimum_version=tlsv1_2vim /usr/lib/python2.7/dist-packages/nova/console/websocketproxy.pydef socket(self, *args, **kwargs):#return websockify.WebSocketServer.socket(*args, **kwargs)return websockify.websockifyserver.WebSockifyServer.socket(*args, **kwargs)find /usr/lib/python2.7/dist-packages/websockify -name "*.pyc" -exec rm -rf {} \;
find /usr/lib/python2.7/dist-packages/nova -name "*.pyc" -exec rm -rf {} \;systemctl restart nova-spiceproxy
nmap --script ssl-enum-ciphers -p 6082 10.5.2.196 |grep -i tlsv

代码流程

websocketproxy.py#websockify_init (opts.ssl_options = select_ssl_version(opts.ssl_version)) -> websockifyserver.py#start_server()when client is connecting:
WebSocketProxy -> ./websocketproxy.py#ProxyRequestHandler -> WebSockifyRequestHandler#handle -> /usr/lib/python3.6/http/server.py(377)handle_one_request() -> _websocket_do_GET -> handle_upgrade (self.headers.get('upgrade').lower() == 'websocket') ->  WebSocketRequestHandlerMixIn.handle_upgrade(self) -> handle_websocket(SSL/TLS)-> new_websocket_client　-> /usr/lib/python3/dist-packages/nova/console/websocketproxy.py(166)new_websocket_client

似乎是在queens中下面的self.headers.get(‘upgrade’)不为wesocket

 54     def _websocket_do_GET(self):55         # Checks if it is a websocket request and redirects56         self.do_GET = self._real_do_GET57 58         if (self.headers.get('upgrade') and59             self.headers.get('upgrade').lower() == 'websocket'):60             self.handle_upgrade()61         else:62             self.do_GET()

根据这个网页（https://www.slideshare.net/DvidHalsz/smuggling-tcp-traffic-through-http-71473570）

结论

主要是由两个问题导致：
1, one spice-html5 bug
https://github.com/freedesktop/spice-html5/commit/293d405e15a4499219fe81e830862cc2b1518e3e
2, ssl默认使用PROTOCOL_SSLv23 （ https://docs.python.org/2/library/ssl.html#socket-creation ）要使用tlsv1_2，或者在ssl.wrap_socket中直接指定＂ssl_version=ssl.PROTOCOL_TLSv1_2，或者使用nova与websockify的patch可以通过ssl_option来配置使用的tls版本。
3, 或者/usr/lib/python2.7/dist-packages/eventlet/green/ssl.py中不应该用PROTOCOL_TLSv1作为默认，而应该用PROTOCOL_SSLv23，这样PROTOCOL_SSLv23会依赖底层ssl版本自动选择tlsv1_0, tlsv1_1, tlsv1_2, 这样即使防火墙disable tlsv1_0也不影响使用tlsv1_2. 最终的原因是因为python-eventlet 0.18.4-1引入了Rebased set-defaults-to-be-tlsv1-not-sslv23.patch，是它使用了PROTOCOL_TLSv1，应该去掉。

附录 - Websocket based spice client

1, a test vm with spice enabled<graphics type='spice' port='5900' autoport='yes' listen='127.0.0.1'><listen type='address' address='127.0.0.1'/></graphics>2, spice proxy side
sudo apt install python3-websockify -y
websockify 192.168.2.139:6082 127.0.0.1:59003, spice client side
sudo apt install spice-html5 apache2 -y
ls /usr/share/spice-html5/spice_auto.html #it's similar to https://10.5.1.11:6082/spice_auto.html?token=xxx
cat << EOF | sudo tee /etc/apache2/conf-available/ws.conf
Alias /spice /usr/share/spice-html5
<Directory /usr/share/spice-html5># This page is broadly available, tune here to make it more restricted.Allow from allSatisfy AnyDirectoryIndex spice.html
</Directory>
EOF
sudo ln -s /etc/apache2/conf-available/ws.conf /etc/apache2/conf-enabled/ws.conf
sudo systemctl restart apache24, access it via http://192.168.2.139/spice/spice_auto.html?host=192.168.2.139&port=6082

附录 - Websocket ssl based spice client

1, create key pairsmkdir ~/ca && cd ~/ca
openssl req -newkey rsa:4096 -x509 -sha256 -days 3650 -nodes -out ca.crt -keyout ca.key -subj "/C=US/ST=UK/L=London/O=Ubuntu/OU=IT/CN=CA"
for DOMAIN in server client
doopenssl genrsa -out $DOMAIN.keyopenssl req -new -key $DOMAIN.key -out $DOMAIN.csr -subj "/C=GB/ST=UK/L=London/O=Ubuntu/OU=Cloud/CN=$DOMAIN"openssl x509 -req -in $DOMAIN.csr -out $DOMAIN.crt -sha256 -CA ca.crt -CAkey ca.key -CAcreateserial -days 3650
done2, spice server sidesudo mkdir /etc/apache2/ssl && sudo chown -R $USER /etc/apache2/ssl
cp /home/hua/ca/server.crt /etc/apache2/ssl/
cp /home/hua/ca/server.key /etc/apache2/ssl/
cat << EOF | sudo tee /etc/apache2/conf-available/ws.conf
<VirtualHost 192.168.2.139:443>SSLEngine onSSLCertificateFile      /etc/apache2/ssl/server.crtSSLCertificateKeyFile   /etc/apache2/ssl/server.keyCustomLog       "/var/log/apache2/ws.log" combinedErrorLog        "/var/log/apache2/ws.log"Alias /spice /usr/share/spice-html5<Directory /usr/share/spice-html5># This page is broadly available, tune here to make it more restricted.Allow from allSatisfy AnyDirectoryIndex spice.html</Directory>
</VirtualHost>
EOF
sudo a2enmod ssl
sudo systemctl restart apache23, spice proxy side
websockify --cert=/home/hua/ca/server.crt --key=/home/hua/ca/server.key --cafile=~/home/hua/ca/ca.crt --ssl-only --ssl-version=tlsv1_2 192.168.2.139:6082 127.0.0.1:59004, access it via: https://192.168.2.139/spice/spice_auto.html?host=192.168.2.139&port=6082nmap --script ssl-enum-ciphers -p 6082 192.168.2.139openssl s_client -connect 192.168.2.139:6082 -tls1_25, debug, print http headers by adding the following scripts in /usr/share/spice-html5/spice_auto.htmlvar req = new XMLHttpRequest();
req.open('GET', document.location, false);  #but this is not the headers for websocket upgrade
req.send(null);
var headers = req.getAllResponseHeaders();
alert(headers);6, why we see the following error when visitting the url 'https://10.5.2.196:6082/spice_auto.html?token=xxxx'SecurityError: Failed to construct 'WebSocket': An insecure WebSocket connection may not be initiated from a page loaded over HTTPS.That's because the following content is missing in /usr/share/spice-html5/spice_auto.html, see this patch - https://github.com/freedesktop/spice-html5/commit/293d405e15a4499219fe81e830862cc2b1518e3eif (window.location.protocol == 'https:') {scheme = "wss://";}

Debug novnc

Debug novnc
先debug http, "10.5.1.160:6080"是已存在的相当于上面的apache2一步它里面带了novnc中的vnc_auto.html
root@juju-9c8c65-ssl-8:~# virsh dumpxml 1 |grep vnc -A2<graphics type='vnc' port='5900' autoport='yes' listen='10.5.2.13' keymap='en-us'><listen type='address' address='10.5.2.13'/></graphics>
websockify 10.5.1.161:6081 10.5.2.13:5900 -v
http://10.5.1.161:6080/vnc_auto.html?host=10.5.1.161&port=6081#some log
zhhuabj-bastion.cloud.sts - - [27/Jan/2021 06:09:51] "GET /websockify HTTP/1.1" 101 -
zhhuabj-bastion.cloud.sts - - [27/Jan/2021 06:09:51] 10.5.0.8: Plain non-SSL (ws://) WebSocket connection
zhhuabj-bastion.cloud.sts - - [27/Jan/2021 06:09:51] 10.5.0.8: Version hybi-13, base64: 'False'
zhhuabj-bastion.cloud.sts - - [27/Jan/2021 06:09:51] 10.5.0.8: Path: '/websockify'
zhhuabj-bastion.cloud.sts - - [27/Jan/2021 06:09:51] connecting to: 10.5.2.13:5900再debug https,　沿用nova.conf(nova-novncproxy)中的cert与key, 只是用于在浏览器与websockify建立ssl, websockify与qemu之间还是http
#--cafile=~/home/hua/ca/ca.crt  --ssl-version=tlsv1_2
openssl x509 -noout -text -in /etc/apache2/ssl/nova/cert_10.5.1.161
websockify --cert=/etc/apache2/ssl/nova/cert_10.5.1.161 --key=/etc/apache2/ssl/nova/key_10.5.1.161 --ssl-only 10.5.1.161:6081 10.5.2.13:5900 -v
https://10.5.1.161:6080/vnc_auto.html?host=10.5.1.161&port=6081#some log
root@juju-9c8c65-ssl-7:~# websockify --cert=/etc/apache2/ssl/nova/cert_10.5.1.161 --key=/etc/apache2/ssl/nova/key_10.5.1.161 --ssl-only 10.5.1.161:6081 10.5.2.13
:5900 -v
WebSocket server settings:- Listen on 10.5.1.161:6081- Flash security policy server- SSL/TLS support- Deny non-SSL/TLS connections- proxying from 10.5.1.161:6081 to 10.5.2.13:5900
10.5.0.8: new handler Process
handler exception: [SSL: SSLV3_ALERT_CERTIFICATE_UNKNOWN] sslv3 alert certificate unknown (_ssl.c:590)
exception
Traceback (most recent call last):File "/usr/lib/python2.7/dist-packages/websockify/websocket.py", line 933, in top_new_clientclient = self.do_handshake(startsock, address)File "/usr/lib/python2.7/dist-packages/websockify/websocket.py", line 840, in do_handshakekeyfile=self.key)File "/usr/lib/python2.7/ssl.py", line 933, in wrap_socketciphers=ciphers)File "/usr/lib/python2.7/ssl.py", line 601, in __init__self.do_handshake()File "/usr/lib/python2.7/ssl.py", line 830, in do_handshakeself._sslobj.do_handshake()
SSLError: [SSL: SSLV3_ALERT_CERTIFICATE_UNKNOWN] sslv3 alert certificate unknown (_ssl.c:590)
Ignoring interrupted syscall
10.5.0.8: new handler Process
zhhuabj-bastion.cloud.sts - - [27/Jan/2021 06:21:27] "GET /websockify HTTP/1.1" 101 -
zhhuabj-bastion.cloud.sts - - [27/Jan/2021 06:21:27] 10.5.0.8: SSL/TLS (wss://) WebSocket connection
zhhuabj-bastion.cloud.sts - - [27/Jan/2021 06:21:27] 10.5.0.8: Version hybi-13, base64: 'False'
zhhuabj-bastion.cloud.sts - - [27/Jan/2021 06:21:27] 10.5.0.8: Path: '/websockify'
zhhuabj-bastion.cloud.sts - - [27/Jan/2021 06:21:27] connecting to: 10.5.2.13:5900
zhhuabj-bastion.cloud.sts - - [27/Jan/2021 06:21:28] 10.5.2.13:5900: Client closed connection
zhhuabj-bastion.cloud.sts - - [27/Jan/2021 06:21:28] 10.5.2.13:5900: Closed target
Ignoring interrupted syscall#/usr/lib/python2.7/dist-packages/websockify/websocket.py
#ssl_version=ssl.PROTOCOL_TLSv1_2,这样改试spice client不work
sudo apt install spice-html5 -y
sed -i "s/ws:/wss:/g" /usr/share/spice-html5/spice_auto.html
cat << EOF | sudo tee /etc/apache2/conf-available/ws.conf
Alias /spice /usr/share/spice-html5
<Directory /usr/share/spice-html5># This page is broadly available, tune here to make it more restricted.Allow from allSatisfy AnyDirectoryIndex spice.html
</Directory>
EOF
sudo ln -s /etc/apache2/conf-available/ws.conf /etc/apache2/conf-enabled/ws.conf
sudo systemctl restart apache2
websockify --cert=/etc/apache2/ssl/nova/cert_10.5.1.161 --key=/etc/apache2/ssl/nova/key_10.5.1.161 --ssl-only 10.5.1.161:6081 10.5.2.13:5900 -v
#https://10.5.1.161:6080/vnc_auto.html?host=10.5.1.161&port=6081
https://10.5.1.161:443/spice/spice_auto.html?host=10.5.1.161&port=6081再这样改试spice却是work的
juju config nova-cloud-controller console-access-protocol=spice
#need to run 'nova reboot --hard bionic-032738' to fix 'Unavailable console type spice'
nova get-vnc-console bionic-032738 spice-html5
websockify --cert=/etc/apache2/ssl/nova/cert_10.5.1.161 --key=/etc/apache2/ssl/nova/key_10.5.1.161 --ssl-only 10.5.1.161:6081 10.5.2.13:5900 -v
#https://10.5.1.161:6080/vnc_auto.html?host=10.5.1.161&port=6081
https://10.5.1.161:6082/spice_auto.html?token=a44045f9-0d22-495e-831a-8f64ae445d2f&host=10.5.1.161&port=6081
https://10.5.1.161:6082/spice_auto.html?host=10.5.1.161&port=6081这样确定了novnc的问题不是是ssl造成的，而是timeout，所以在/usr/share/novnc/include/rfb.js修改下面配置之后
def_con_timeout  = Websock_native ? 200 : 500,
再restart(sudo service nova-novncproxy restart)就可以成功访问了。
https://10.5.1.161:6080/vnc_auto.html?host=10.5.1.161&port=6081
https://10.5.1.161:6080/vnc_auto.html?token=887a070e-1f45-4320-912f-577a50417b1d&host=10.5.1.161&port=6081

20210204更新

有时候如heat charm在里面会创建haproxy实例（外面用hacluster提供corosync做HA)，但并不需要在haproxy.cnf里设置如下设置：

ssl-default-bind-options ssl-min-ver TLSv1.2 no-tls-tickets
ssl-default-server-options ssl-min-ver TLSv1.2 no-tls-tickets

因为haproxy的后面是apache2，它由这个bug (https://bugs.launchpad.net/charm-helpers/+bug/1886630)已经提供了fix (https://github.com/juju/charm-helpers/commit/27d6ceb385e44a0610c1a6aba8e225368c4af384 )在apache2层面禁用了sslv3/tlsv1

root@juju-170792-stein-2:~# grep ':7994' /etc/apache2/sites-available/openstack_https_frontend.conf -A 7
<VirtualHost 10.5.3.29:7994>
ServerName 10.5.3.29
SSLEngine on# This section is based on Mozilla's recommendation
# as the "intermediate" profile as of July 7th, 2020.
# https://wiki.mozilla.org/Security/Server_Side_TLS
SSLProtocol all -SSLv3 -TLSv1 -TLSv1.1root@juju-170792-stein-2:~# grep -r 7994 /etc/haproxy/haproxy.cfg -B2
backend heat_api_10.5.3.29
balance leastconn
server heat-0 10.5.3.29:7994 check
root@juju-170792-stein-2:~# grep -r 'use_backend heat_api_10.5.3.29' /etc/haproxy/haproxy.cfg -B4
frontend tcp-in_heat_api
bind *:8004
bind :::8004
acl net_10.5.3.29 dst 10.5.3.29/255.255.0.0
use_backend heat_api_10.5.3.29 if net_10.5.3.29

landscape也会在postgresql前安装haproxy，但也没提供enable TLSv1.2的配置项，可暂时用下列workaround:

juju config landscape-postgresql extra_pg_conf="ssl_ciphers='TLSv1.2'"

keystone也会在apache2前安装haproxy，这个apache2不是来自charm-helper, 可单独修改charm template(/var/lib/juju/agents/unit-keystone-0/charm/templates/openstack_https_frontend.conf)作为workaround:

-SSLProtocol +TLSv1 +TLSv1.1 +TLSv1.2
+SSLProtocol -TLSv1 -TLSv1.1 +TLSv1.2

对于Landscape/Postfix, In /etc/postfix, master.cf or main.cf (whichever one you use):

smtpd_tls_mandatory_protocols = x,y,z,!TLSv1, !TLSv1.1

[1] https://blog.surgut.co.uk/2019/08/how-to-disable-tls-10-and-tls-11-on.html
[2] https://discourse.ubuntu.com/t/default-to-tls-v1-2-in-all-tls-libraries-in-20-04-lts/12464
[3] https://notes.bitfunnel.net/?q=node/54
[4] https://github.com/certik/python-2.7/blob/master/Lib/ssl.py#L373