连接PC桌面和android手机的神器--scrcpy

无意中发现一个手机调试和自动化测试的神器， Scrcpy，甚是强大，探索了一番，配合minitouch可以实现更自动化的操作，但是也可以用python的pythonautogui这样的工具来完成自动化控制PC端的操作鼠标和键盘，从而实现在手机端自动点击或打开应用，当然，这需要对手机屏幕的显示内容进行识别和分析，或者是依据绝对坐标。

开源库链接

scrcpy

GitHub - Genymobile/scrcpy: Display and control your Android device

minitouch

GitHub - openstf/minitouch: Minimal multitouch event producer for Android.

1. 开源，支持跨平台。

这个厉害，windows， linux，mac都可以用

2. 低延时，适当降低分辨率，实测延时可以低于50ms，都可以处理手机视频了，这个妙用后面会提到

3. 桌面投屏，可以鼠标操作，这样就可以进行自动化测试了

安装很简单

linux用snap安装（测试环境在虚拟机下）

sudo apt install adb
先adb查看设备

hy@ubuntu:~$ adb devices
List of devices attached
6e5ece64 device
然后运行连接

hy@ubuntu:~$ scrcpy --window-x 100 --window-y 100 --window-width 400 --window-height 300 -m 640 -b 16M
INFO: scrcpy 1.17 <https://github.com/Genymobile/scrcpy>
/usr/local/share/scrcpy/scrcpy-server: 1 file pushed. 1.1 MB/s (34930 bytes in 0.031s)
[server] INFO: Device: Xiaomi M2002J9E (Android 10)
INFO: Renderer: opengl
INFO: OpenGL version: 3.3 (Compatibility Profile) Mesa 20.0.8
INFO: Trilinear filtering enabled
INFO: Initial texture: 288x640
上面的指令指定了窗口大小，位置，及分辨率，带宽以保证低延时

高级操作：

同时连接多个设备

通过wifi无线连接

1.首先把PC和手机连到同一个wifi，先用数据线来配置adb

2. 获取手机IP地址，当然，也可以在手机上查看

hy@ubuntu:~/dev/minitouch$ adb shell ip route | awk '{print $9}'
10.9.118.229
192.168.1.8
3. 打开手机端口

hy@ubuntu:~/dev/minitouch$ adb tcpip 5555
restarting in TCP mode port: 5555
4.拔掉数据线，改为无线连接

hy@ubuntu:~/dev/minitouch$ adb connect 192.168.1.8:5555
connected to 192.168.1.8:5555
5. 运行scrcpy

其实，此时也可以正常运行其他adb指令

实际应用方向：

1. 投屏演示APP应用页面

之前也有用过其他投屏软件，要么要收费，要么速度像蜗牛，太不友好了。用scrcpy做投屏，讲解手机蓝牙/小程序/APP应用开发等课程，那是一个爽。

还有个很方便的，就是直接在PC上对手机进行录屏，只需要一条指令。

scrcpy --record file.mp4

2. 自动识别执行/自动登录等自动化操作或者自动化执行

通过PC端运行脚本控制手机APP模拟人手操作，可以自动抢红包，自动点赞，自动投票，定时抢茅台，自动挂机打游戏，这样可以用来做AI游戏训练等高级应用，也可以控制多台手机集群操作，自动拨打电话，自动发送信息，定时自动执行登录，自动回复消息（对显示内容OCR）等等，有很多想象空间。

3. 监控手机执行画面，并进行实时分析或操作外设

比如自拍云台应用，就可以通过这种方式，调用手机摄像头画面进行AI识别后做目标跟踪，人脸跟踪，人脸识别，自动拍照，自动录像，自动回复点赞等，视频可以做到100ms以下延时的话，这些都可以达到比较好的效果。

如果应用在直播软件上，则可以监看直播软件的运行页面，进行效果分析等等。

minitouch的应用，可以更准确模拟人手操作。

仓库链接

GitHub - openstf/minitouch: Minimal multitouch event producer for Android.

首先要配置NDK环境编译

下载解压后，设置目录

sudo gedit ~/.bashrc
加入下面一句

export NDK=/home/hy/dev/android-ndk-r21

进入minitouch目录

编译

NDK/ndk-build

加载运行

获取CPU类型

ABI=$(adb shell getprop ro.product.cpu.abi | tr -d '\r')

将应用代码推到手机

adb push libs/$ABI/minitouch /data/local/tmp/

运行
adb shell /data/local/tmp/minitouch

但是提示没有权限

adb shell /data/local/tmp/minitouch -i
open: Permission denied
Unable to open device /dev/input/event7 for inspectionopen: Permission denied
Unable to open device /dev/input/event6 for inspectionopen: Permission denied
Unable to open device /dev/input/event5 for inspectionopen: Permission denied
Unable to open device /dev/input/event1 for inspectionopen: Permission denied
Unable to open device /dev/input/event3 for inspectionopen: Permission denied
Unable to open device /dev/input/event0 for inspectionopen: Permission denied
Unable to open device /dev/input/event2 for inspectionopen: Permission denied
Unable to open device /dev/input/event4 for inspectionUnable to find a suitable touch device
using Android InputManager

此时，可以用另外个工具进行连接服务建立GitHub - openstf/STFService.apk: Monitor and perform various actions on your Android device. Not meant for actual user devices.

需要把release版本的apk安装包在手机进行安装，注意这个应用是没有图标的。

Features

Protocol Buffers based socket interface
Monitoring
- Battery statistics
- Network connectivity
- Phone state
- Device rotation
- Airplane mode
- Installed browser packages
Actions
- List user accounts
- Remove user accounts
- Get list of installed browsers
- Get/set clipboard contents
- Get display information
- Get device IMEI number
- Get device phone number
- Get device ICCID number
- Get device network operator
- Get device network type
- Get/set device ringer mode
- Check whether an SD card is installed
- Enable/disable WIFI, Bluetooth and query its status
- Unlock/relock KeyGuard (essentially unlocking a device)
- Acquire/release WakeLock (i.e. prevent a device from or allow it to sleep)
- Invoke key events (with meta key support)
- Set device rotation (optionally locking it)
Misc tasks
- Show a bright red screen with identifying information that wakes up and keeps the device on. Amazingly useful when you've got a whole bunch of devices you're connected to remotely, and you want to find the real, physical device.

看起来好强大的样子，但怎么获取这些信息呢？ stf则是个更强大的web版的管理后台，有空了再研究。

GitHub - openstf/stf: Control and manage Android devices from your browser.

手机安装完apk后，可以启动服务

hy@ubuntu:~$ adb shell am startservice --user 0 \
> -a jp.co.cyberagent.stf.ACTION_START \
> -n jp.co.cyberagent.stf/.Service
Starting service: Intent { act=jp.co.cyberagent.stf.ACTION_START cmp=jp.co.cyberagent.stf/.Service }
Error: app is in background uid null
这个报错还不知道什么原因，但是好像就运行起来了

如果想用事件触发形式来操作，则要启动转发代理

adb forward tcp:1100 localabstract:stfservice

nc localhost 1100

连接以后可以建立基于proto的消息机制

While both sockets use the same format, neither is able to respond to all requests. You must know which socket to send your request to. The agent is currently only able to respond to the following requests:

DO_KEYEVENT
DO_TYPE
DO_WAKE
SET_ROTATION

这时如果在终端启动代理后，就可以正常运行minitouch指令了

hy@ubuntu:~/dev/minitouch$ adb shell export CLASSPATH="$APK"\; exec app_process /system/bin jp.co.cyberagent.stf.Agent
Starting minitouch agent
Listening on @stfagent
再运行minitouch

hy@ubuntu:~$ adb shell /data/local/tmp/minitouch -i
open: Permission denied
Unable to open device /dev/input/event7 for inspectionopen: Permission denied
Unable to open device /dev/input/event6 for inspectionopen: Permission denied
Unable to open device /dev/input/event5 for inspectionopen: Permission denied
Unable to open device /dev/input/event1 for inspectionopen: Permission denied
Unable to open device /dev/input/event3 for inspectionopen: Permission denied
Unable to open device /dev/input/event0 for inspectionopen: Permission denied
Unable to open device /dev/input/event2 for inspectionopen: Permission denied
Unable to open device /dev/input/event4 for inspectionUnable to find a suitable touch device
using Android InputManager
Reading from STDIN
v 1
^ 10 1080 2400 0
$ 17316
此时就可以正常输入指令，完成远端控制了

Examples

Tap on (10, 10) with 50 pressure using a single contact.

d 0 10 10 50
c
u 0
c

Long tap on (10, 10) with 50 pressure using a single contact.

d 0 10 10 50
c
<wait in your own code>
u 0
c

Tap on (10, 10) and (20, 20) simultaneously with 50 pressure using two contacts.

d 0 10 10 50
d 1 20 20 50
c
u 0
u 1
c

Tap on (10, 10), keep it pressed, then after a while also tap on (20, 20), keep it pressed, then release the first contact and finally release the second contact.

d 0 10 10 50
c
<wait in your own code>
d 1 20 20 50
c
<wait in your own code>
u 0
c
<wait in your own code>
u 1
c

Swipe from (0, 0) to (100, 0) using a single contact. You'll need to wait between commits in your own code to slow it down.

d 0 0 0 50
c
m 0 20 0 50
c
m 0 40 0 50
c
m 0 60 0 50
c
m 0 80 0 50
c
m 0 100 0 50
c
u 0
c

Pinch with two contacts going from (0, 100) to (50, 50) and (100, 0) to (50, 50). You'll need to wait between commits in your own code to slow it down.

d 0 0 100 50
d 1 100 0 50
c
m 0 10 90 50
m 1 90 10 50
c
m 0 20 80 50
m 1 80 20 50
c
m 0 20 80 50
m 1 80 20 50
c
m 0 30 70 50
m 1 70 30 50
c
m 0 40 60 50
m 1 60 40 50
c
m 0 50 50 50
m 1 50 50 50
c
u 0
u 1
c

The same pinch but with more chaotic (or natural) ordering.

d 1 100 0 50
c
d 0 0 100 50
c
m 1 90 10 50
m 0 10 90 50
c
m 0 20 80 50
c
m 1 80 20 50
c
m 0 20 80 50
m 1 80 20 50
c
m 0 30 70 50
c
m 1 70 30 50
c
m 1 60 40 50
c
m 0 40 60 50
c
m 0 50 50 50
m 1 50 50 50
c
u 0
c
u 1
c

不过，如果不是通过socket方式进行操控的话，这种方式并没有直接截取屏幕更方便，可能直接下发触屏指令的优势在于速度。

附注： ADB如何调用app

1. 查找应用包名

adb shell pm list packages -f | grep 包名 (可以输入tencent等关键字筛选)

也可以输出到文本文件慢慢看

adb shell pm list packages -f | grep 包名 ->1.txt

2）根据包名，运行调起应用

adb shell monkey -p 包名 -c android.intent.category.LAUNCHER 1

例子：

调用微信

hy@ubuntu:~/dev/minitouch$ adb shell monkey -p com.tencent.mm -c android.intent.category.LAUNCHER 1
bash arg: -p
bash arg: com.tencent.mm
bash arg: -c
bash arg: android.intent.category.LAUNCHER
bash arg: 1
args: [-p, com.tencent.mm, -c, android.intent.category.LAUNCHER, 1]
arg: "-p"
arg: "com.tencent.mm"
arg: "-c"
arg: "android.intent.category.LAUNCHER"
arg: "1"
data="com.tencent.mm"
data="android.intent.category.LAUNCHER"
Events injected: 1
## Network stats: elapsed time=104ms (0ms mobile, 0ms wifi, 104ms not connected)

也可以通过adb直接控制手机操作

ADB控制手机命令

监听事件存入文本进行分析，可以用

adb shell getevent >1.txt

add device 1: /dev/input/event7
name: "lito-mtp-snd-card USB_3_5 Jack"
add device 2: /dev/input/event6
name: "lito-mtp-snd-card Button Jack"
add device 3: /dev/input/event5
name: "lito-mtp-snd-card Headset Jack"
add device 4: /dev/input/event1
name: "uinput-goodix"
add device 5: /dev/input/event3
name: "aw8624_haptic"
add device 6: /dev/input/event0
name: "qpnp_pon"
add device 7: /dev/input/event2
name: "fts_ts"
add device 8: /dev/input/event4
name: "gpio-keys"
/dev/input/event2: 0003 0039 000020e8
/dev/input/event2: 0001 0152 00000001
/dev/input/event2: 0001 014a 00000001
/dev/input/event2: 0003 0035 0000023e
/dev/input/event2: 0003 0036 00000844
/dev/input/event2: 0003 0032 00000064
/dev/input/event2: 0003 0033 00000064
/dev/input/event2: 0000 0000 00000000
/dev/input/event2: 0003 0035 00000243
/dev/input/event2: 0003 0036 00000860
/dev/input/event2: 0000 0000 00000000
/dev/input/event2: 0003 0036 00000862
/dev/input/event2: 0000 0000 00000000
/dev/input/event2: 0003 0036 00000863
/dev/input/event2: 0000 0000 00000000

然后模拟操作就可以控制

adb shell sendevent xxxxx xxxx xxxx

这个指令就可以在python里直接调用shell命令来进行操作了

例如

sendevent /dev/input/event3 3 48 37 可以删除
sendevent /dev/input/event3 3 50 5 保留一个
sendevent /dev/input/event3 3 53 352 点击事件位置
sendevent /dev/input/event3 3 54 550 点击事件位置
sendevent /dev/input/event3 0 2 0 事件分割
sendevent /dev/input/event3 0 0 0 事件分割
sendevent /dev/input/event3 3 48 37
sendevent /dev/input/event3 3 50 5
sendevent /dev/input/event3 3 53 352
sendevent /dev/input/event3 3 54 550
sendevent /dev/input/event3 0 2 0
sendevent /dev/input/event3 0 0 0
sendevent /dev/input/event7 3 40 18
sendevent /dev/input/event7 0 0 0
sendevent /dev/input/event3 0 2 0
sendevent /dev/input/event3 0 0 0
sendevent /dev/input/event7 3 40 22
sendevent /dev/input/event7 0 0 0
sendevent /dev/input/event7 3 40 24
sendevent /dev/input/event7 0 0 0
sendevent /dev/input/event3 0 2 0
sendevent /dev/input/event3 0 0 0