通过OpenCV实现虚拟键盘

环境配置

在Anaconda中创建一个python=3.7的环境

可在pycharm内直接新建环境，也可以通过命令行的方式

conda create -n your_env_name python=x.x

激活或者切换虚拟环境 activate your_env_name

然后对此虚拟环境中安装额外的包

pip install cvzone

Collecting cvzoneUsing cached cvzone-1.4.1.tar.gz (11 kB)
Collecting opencv-pythonDownloading opencv_python-4.5.3.56-cp37-cp37m-win_amd64.whl (34.9 MB)|████████████████████████████████| 34.9 MB 1.1 MB/s
Collecting numpyDownloading numpy-1.21.2-cp37-cp37m-win_amd64.whl (14.0 MB)|████████████████████████████████| 14.0 MB 1.1 MB/s
Building wheels for collected packages: cvzoneBuilding wheel for cvzone (setup.py) ... doneCreated wheel for cvzone: filename=cvzone-1.4.1-py3-none-any.whl size=17681 sha256=b9f8a2d3a160b07195f96790b8a4a8aa637bbba78d9de84e60467749947f9001Stored in directory: c:\users\lzq\appdata\local\pip\cache\wheels\7d\c8\ae\0e2ab1bda96471f8c6607160b1c96f76a963d3ef846d1bb9bb
Successfully built cvzone
Installing collected packages: numpy, opencv-python, cvzone
Successfully installed cvzone-1.4.1 numpy-1.21.2 opencv-python-4.5.3.56

pip install mediapipe

Collecting absl-pyUsing cached absl_py-0.13.0-py3-none-any.whl (132 kB)
Collecting opencv-contrib-pythonDownloading opencv_contrib_python-4.5.3.56-cp37-cp37m-win_amd64.whl (41.8 MB)|████████████████████████████████| 41.8 MB 3.2 kB/s
Requirement already satisfied: numpy in f:\environment\anaconda\envs\cv37\lib\site-packages (from mediapipe) (1.21.2)
Collecting sixUsing cached six-1.16.0-py2.py3-none-any.whl (11 kB)
Requirement already satisfied: wheel in f:\environment\anaconda\envs\cv37\lib\site-packages (from mediapipe) (0.37.0)
Collecting protobuf>=3.11.4Downloading protobuf-3.17.3-cp37-cp37m-win_amd64.whl (909 kB)|████████████████████████████████| 909 kB 3.3 MB/s
Collecting attrs>=19.1.0Downloading attrs-21.2.0-py2.py3-none-any.whl (53 kB)|████████████████████████████████| 53 kB 943 kB/s
Collecting matplotlibDownloading matplotlib-3.4.3-cp37-cp37m-win_amd64.whl (7.2 MB)|████████████████████████████████| 7.2 MB 939 kB/s
Collecting kiwisolver>=1.0.1Downloading kiwisolver-1.3.1-cp37-cp37m-win_amd64.whl (51 kB)|████████████████████████████████| 51 kB 1.3 MB/s
Collecting python-dateutil>=2.7Using cached python_dateutil-2.8.2-py2.py3-none-any.whl (247 kB)
Collecting pyparsing>=2.2.1Downloading pyparsing-2.4.7-py2.py3-none-any.whl (67 kB)|████████████████████████████████| 67 kB 823 kB/s
Collecting pillow>=6.2.0Downloading Pillow-8.3.1-1-cp37-cp37m-win_amd64.whl (3.2 MB)|████████████████████████████████| 3.2 MB 1.6 MB/s
Collecting cycler>=0.10Downloading cycler-0.10.0-py2.py3-none-any.whl (6.5 kB)
Installing collected packages: six, python-dateutil, pyparsing, pillow, kiwisolver, cycler, protobuf, opencv-contrib-python, matplotlib, attrs, absl-py, mediapipe
Successfully installed absl-py-0.13.0 attrs-21.2.0 cycler-0.10.0 kiwisolver-1.3.1 matplotlib-3.4.3 mediapipe-0.8.7 opencv-contrib-python-4.5.3.56 pillow-8.3.1 prot
obuf-3.17.3 pyparsing-2.4.7 python-dateutil-2.8.2 six-1.16.0

项目实现

打开摄像

导入opencv库，并创造一个VideoCapture

import cv2cap = cv2.VideoCapture(0)# 0为webcam id
#通常为默认值，但为了有足够空间放置键盘键，所以进行设置
# HD分辨率
cap.set(3, 1280)  # 设置宽度
cap.set(4, 720)  # 设定高度while True:success, img = cap.read()cv2.imshow("Image", img)cv2.waitKey(1)

运行后应该可以顺利的打开一个窗口，展现电脑摄像头拍到的画面。

（我的电脑摄像头太渣了，所以set的分辨率完全配不上）

跟踪手部

首先导入并创建手部跟踪模块

from cvzone.HandTrackingModule import HandDetector# 创建检测器 置信度调为0.8（默认0.5）
detector=HandDetector(detectionCon=0.8)

在while循环内识别标注

此时再次运行

此时已经可以识别手部，并且标注出来了。

创建按键

屏幕中的按键有它的位置，大小，颜色，文字等属性，如果每一个都自己使用cv2.rectangle()与cv2.putText来填写太过麻烦，所以考虑创建一个类来容纳按键的这些属性，并由一个列表来管理这些按键

class Button():def __init__(self, pos, text, size=[42, 42]):self.pos = posself.text = textself.size = sizedef draw(self, img, color=(112, 161, 255)):x, y = self.posw, h = self.sizecv2.rectangle(img, self.pos, (x + w, y + h), color, cv2.FILLED)cv2.putText(img, self.text, (x + 13, y + 23), cv2.FONT_HERSHEY_PLAIN, 1.5, (255, 255, 255), 1)return img

键盘字符

keys = [["Q", "W", "E", "R", "T", "Y", "U", "I", "O", 'P', "[", ']'],["A", "S", "D", "F", "G", "H", "J", "K", "L", ";", "'"],["Z", "X", "C", "V", "B", "N", 'M', ",", ".", "/"]]

绘制

buttonList = []
for i in range(len(keys)):for j, key in enumerate(keys[i]):buttonList.append(Button([50 * j + 10+ i*12, 60+50 * i], key))def AllDraw(img):for btn in buttonList:img = btn.draw(img)return img

然后将img=AllDraw(img)加入循环之中

则可出现一个键盘

检测手指

通过mediapipe获得手势信息相关链接

通过检测8号的位置，来判断食指尖停留在哪个键上

while True:success, img = cap.read()img = cv2.flip(img, 1)img = detector.findHands(img)lmList, bboxInfo = detector.findPosition(img)img = AllDraw(img)if lmList:for button in buttonList:x, y = button.posw, h = button.sizedeviation=1if x - deviation < lmList[8][0] < x + w + deviation \and y - deviation < lmList[8][1] < y + h + deviation:img = button.draw(img, (12, 95, 255))cv2.imshow("Image", img)cv2.waitKey(1)

点击按键

手指的点击只是轻微完全，判断起来较为麻烦，干扰较多

因此可以选择其他方式来替换点击

我选择了大拇指 4号点轻碰 5号点位置视为一次点击

通过l,_,_=detector.findDistance(4,5,img,draw=False)判断l的长度，如l<20是视为一次点击(可以添加变色效果)

l,_,_=detector.findDistance(4,5,img,draw=False)if l<20 :img = button.draw(img, (34,126,230))

当然，也可以使用别的方式来代替点击，比如食指与中指并在一起视为点击，等（因为我的按键设置太小所以没这样判断）

其中有一个需要注意的点，就是点击过程中可能会一下触发多次

有两种方式

一种是加点判断逻辑，当l<20后变为l>20视为触发一次点击（一直按着不算点击，松手才算）
一种是当第一次l<20时sleep0.7秒，

添加透明效果

def AllDraw(img):imgNew = np.zeros_like(img, np.uint8)for btn in buttonList:imgNew = btn.draw(imgNew)out = img.copy()alpha = 0.2mask = imgNew.astype(bool)out[mask] = cv2.addWeighted(img,alpha,imgNew,1-alpha,0)[mask]return out