【聆思CSK6 视觉AI开发套件试用】人脸识别和 tflite-micro 语音识别推理实验

本篇文章来自极术社区与聆思科技组织的CSK6 视觉AI开发套件活动，更多开发板试用活动请关注极术社区网站。作者：SunnyCase

1. 前言

非常幸运得到这次测评 CSK6 视觉AI开发套件的机会，在这里先感谢极术社区和聆思科技举办的活动。这次试用下来，从工具链、SDK、文档到示例，让我体会到了聆思科技在 AI 芯片深厚的技术积累，我只有一个评价：专业。

2. CSK6芯片简介

CSK6 是聆思科技新一代的 AI 芯片 SoC 产品系列，采用多核异构架构，集成了 ARM Star MCU，HiFi4 DSP，以及聆思全新设计的 AI 神经网络处理内核 NPU，算力达到 128 GOPS。多核异构的设计使芯片能以较低功耗满足音频及图像视频的 AI 应用需求。

本系列芯片集成了 SRAM 与 PSRAM，支持内置或外接Flash，可提供最高 4 入 2 出的 Audio Codec，VGA 像素的 DVP 摄像头接口，多达 6 路的触控检测以及 SPI、UART、USB、SDIO、I2C、I2S 等各类外设接口，丰富接口支持各类应用方案的开发。

3. 开发环境搭建

CSK6 的开发环境搭建非常简单，只需要参考官方文档
我个人使用家里工作站上的 ubuntu 20.04 进行编译，PC 的 windows 11 用来烧录和查看，代码和编译结果都在 NAS 上，所以这里遇到了一点问题：
官方提供的离线包是 snap 包，而snap 不支持访问某些自行 mount 的目录，也就没法访问我的 NAS 。
于是我还是采用了在线安装的方式：

curl -o- https://cdn.iflyos.cn/public/cskTools/lisa-zephyr-install.sh | bash

SDK 会安装在~/.listenai目录。

4. 人脸识别

clone sample
lisa zep create --from-githttps://cloud.listenai.com/zephyr/applications/app_algo_fd_sample_for_csk6.git
编译固件
lisa zep build -b csk6011a_nano
看到如下显示就表示成功了
[248/248] Linking C executable zephyr/zephyr.elf Memory region Used Size Region Size %age Used FLASH: 191040 B 16 MB 1.14% SRAM: 127504 B 320 KB 38.91% ITCM: 6268 B 16 KB 38.26% DTCM: 64 B 16 KB 0.39% PSRAMAP: 1591536 B 2164 KB 71.82% PSRAMAP_NOCACHE: 0 GB 1804 KB 0.00% IDT_LIST: 0 GB 2 KB 0.00% ✔ 构建成功
烧录固件和资源
我需要在 windows 上查看结果，可以使用 cskburn，可以在GitHub release 下载
串口查看工具可以使用免费的 MobaXterm。
烧录 DSP 固件和模型资源
(base) PS D:\Software\Development\listenai> .\cskburn -s \.\COM5 -b 748800 -C 6 0x400000 “X:\work\repo\csk6-learn\app_algo_fd_sample_for_csk6\resource\cp.bin” 0x500000 “X:\work\repo\csk6-learn\app_algo_fd_sample_for_csk6\resource\res.bin” Partition 1: 0x00400000 (783.35 KB) - X:\work\repo\csk6-learn\app_algo_fd_sample_for_csk6\resource\cp.bin Partition 2: 0x00500000 (7003.59 KB) - X:\work\repo\csk6-learn\app_algo_fd_sample_for_csk6\resource\res.bin Waiting for device… Entering update mode… Detected flash size: 16 MB Burning partition 1/2… (0x00400000, 783.35 KB) 783.35 KB / 783.35 KB (100.00%) Burning partition 2/2… (0x00500000, 7003.59 KB) 7003.59 KB / 7003.59 KB (100.00%) Finished
烧录固件
(base) PS D:\Software\Development\listenai> .\cskburn -s \.\COM5 -b 748800 -C 6 0x0 “X:\work\repo\csk6-learn\app_algo_fd_sample_for_csk6\build\zephyr\zephyr.bin” Partition 1: 0x00000000 (186.56 KB) - X:\work\repo\csk6-learn\app_algo_fd_sample_for_csk6\build\zephyr\zephyr.bin Waiting for device… Entering update mode… Detected flash size: 16 MB Burning partition 1/1… (0x00000000, 186.56 KB) 186.56 KB / 186.56 KB (100.00%) Finished
PC 查看结果
拉取查看工具
git clonehttps://cloud.listenai.com/zephyr/applications/csk_view_finder_spd.git
按照提示安装驱动选择设备并连接

就可以实时查看识别结果了

5. tflite-micro 语音识别推理测试

新建项目
lisa zep create tflite_micro --from hello_world
选择 hello-world，输入项目名 tflite_micro
启用 tflite-micro 和 C++ 支持
启动 menuconfig
lisa zep build -t menuconfig -b csk6011a_nano

Modules 里打开 tflite micro support
C Library中选择Newlib C Library
Sub Systems and OS Services中选中C++ support for the application

编写测试
主要功能是加载micro_speech.tflite模型并识别一段语音是yesorno
main.cpp 代码如下
注：GenerateMicroFeatures无法链接成功，于是使用了预先生成好的 features。

/** Copyright (c) 2012-2014 Wind River Systems, Inc.** SPDX-License-Identifier: Apache-2.0*/#include <zephyr/zephyr.h>
#include <tensorflow/lite/micro/all_ops_resolver.h>
#include <tensorflow/lite/micro/micro_error_reporter.h>
#include <tensorflow/lite/micro/micro_interpreter.h>
#include <tensorflow/lite/schema/schema_generated.h>
#include <tensorflow/lite/micro/testing/micro_test.h>
#include "micro_features_generator.h"
#include "micro_model_settings.h"
#include "yes_micro_features_data.h"#define INCBIN_STYLE INCBIN_STYLE_SNAKE
#include "incbin.h"INCBIN(micro_speech_tflite, "micro_speech.tflite");const int tensor_arena_size = 64 * 1024;
uint8_t tensor_arena[tensor_arena_size];TF_LITE_MICRO_TESTS_BEGINTF_LITE_MICRO_TEST(CSK6_TFLITE)
{printk("Hello CSK6 TFLITE! %s\n", CONFIG_BOARD);micro_test::tests_passed = 0;micro_test::tests_failed = 0;tflite::InitializeTest();tflite::MicroErrorReporter micro_error_reporter;tflite::ErrorReporter *error_reporter = &micro_error_reporter;const tflite::Model *model = ::tflite::GetModel(gmicro_speech_tflite_data);if (model->version() != TFLITE_SCHEMA_VERSION){TF_LITE_REPORT_ERROR(error_reporter,"Model provided is schema version %d not equal ""to supported version %d.\n",model->version(), TFLITE_SCHEMA_VERSION);}tflite::MicroMutableOpResolver<4> micro_op_resolver;micro_op_resolver.AddDepthwiseConv2D();micro_op_resolver.AddFullyConnected();micro_op_resolver.AddReshape();micro_op_resolver.AddSoftmax();tflite::MicroInterpreter interpreter(model, micro_op_resolver, tensor_arena, tensor_arena_size, error_reporter);interpreter.AllocateTensors();TfLiteTensor *input = interpreter.input(0);TF_LITE_MICRO_EXPECT(input != nullptr);TF_LITE_MICRO_EXPECT_EQ(2, input->dims->size);TF_LITE_MICRO_EXPECT_EQ(1, input->dims->data[0]);TF_LITE_MICRO_EXPECT_EQ(1960, input->dims->data[1]);TF_LITE_MICRO_EXPECT_EQ(kTfLiteInt8, input->type);const int8_t *yes_features_data = g_yes_micro_f2e59fea_nohash_1_data;for (size_t i = 0; i < input->bytes; ++i){input->data.int8[i] = yes_features_data[i];}TfLiteStatus invoke_status = interpreter.Invoke();if (invoke_status != kTfLiteOk){MicroPrintf("Invoke failed\n");}TF_LITE_MICRO_EXPECT_EQ(kTfLiteOk, invoke_status);const int kSilenceIndex = 0;const int kUnknownIndex = 1;const int kYesIndex = 2;const int kNoIndex = 3;TfLiteTensor *output = interpreter.output(0);TF_LITE_MICRO_EXPECT_EQ(2, output->dims->size);TF_LITE_MICRO_EXPECT_EQ(1, output->dims->data[0]);TF_LITE_MICRO_EXPECT_EQ(4, output->dims->data[1]);TF_LITE_MICRO_EXPECT_EQ(kTfLiteInt8, output->type);// Make sure that the expected "Yes" score is higher than the other classes.uint8_t silence_score = output->data.int8[kSilenceIndex] + 128;uint8_t unknown_score = output->data.int8[kUnknownIndex] + 128;uint8_t yes_score = output->data.int8[kYesIndex] + 128;uint8_t no_score = output->data.int8[kNoIndex] + 128;TF_LITE_MICRO_EXPECT_GT(yes_score, silence_score);TF_LITE_MICRO_EXPECT_GT(yes_score, unknown_score);TF_LITE_MICRO_EXPECT_GT(yes_score, no_score);
}TF_LITE_MICRO_TESTS_END

4、烧录运行

(base) PS D:\Software\Development\listenai> .\cskburn -s \\.\COM5 -b 748800 -C 6 0x0 "X:\work\repo\csk6-learn\tflite_micro\build\zephyr\zephyr.bin"
Partition 1: 0x00000000 (156.93 KB) - X:\work\repo\csk6-learn\tflite_micro\build\zephyr\zephyr.bin
Waiting for device...
Entering update mode...
Detected flash size: 16 MB
Burning partition 1/1... (0x00000000, 156.93 KB)
156.93 KB / 156.93 KB (100.00%)
Finished

串口打印

**** Booting Zephyr OS build v1.1.1-alpha.2  ***
Testing CSK6_TFLITE
Hello CSK6 TFLITE! csk6011a_nano
1/1 tests passed
~~~ALL TESTS PASSED~~~

正确识别出了yes

6. 碎碎念

希望官方快点放出 DSP 和 NPU 的工具链呀~