linux V4L2子系统——v4l2架构（5）之v4l2_device与v4l2_subdev异步机制

备注：
1. Kernel版本：5.4
2. 使用工具：Source Insight 4.0
3. 参考博客：
（1）Linux V4L2子系统分析（一）
（2）linux v4l2 学习之-v4l2设备注册过程及各个设备之间的联系

文章目录

linux V4L2子系统——v4l2架构（5）之v4l2_device与v4l2_subdev异步机制
- 概述
- - 背景
  - 用途
- 主要结构体介绍
- V4L2主从设备匹配过程分析
- - V4L2主设备匹配过程
  - V4L2从设备匹配过程
  - V4L2异步注册过程

概述

背景

在soc中的视频处理一般由多个ip组成，比如cis_dphy、mipi_cis、isp、sensor等，甚至更多的ip, 这样就导致了v4l2的复杂性。在v4l2中的视频数据流是有方向和顺序的，因此在linux中引入了异步注册机制。异步注册的核心在于设备树引入port接口，在子设备中有一个或多个port接口，port接口就是子设备的纽带。

用途

在异步模式下，子设备 probing 可以被独立地被调用以检查桥驱动是否可用，子设备驱动必须确认所有的 probing 请求是否成功，如果有任意一个请求条件没有满足，驱动就会返回 -EPROBE_DEFER 来继续下一次尝试，一旦所有的请求条件都被满足，子设备就需要调用 v4l2_async_register_subdev 函数来进行注册（用 v4l2_async_unregister_subdev 卸载）。桥驱动反过来得注册一个 notifier 对象（v4l2_async_notifier_register），该函数的第二个参数类型是 v4l2_async_notifier 类型的结构体，里面包含有一个指向指针数组的指针成员，指针数组每一个成员都指向 v4l2_async_subdev 类型结构体。

v4l2 核心层会利用上述的异步子设备结构体描述符来进行子设备的匹配，过程如下：

1. 如果成功匹配，.bound() 回调函数将会被调用；
1. 当所有的子设备全部被加载完毕之后，.complete() 回调函数就会被调用；
1. 子设备被移除的时候 .unbind() 函数就会被调用。

主要结构体介绍

1）匹配类型：

// 源码： include/media/v4l2-async.h
/*** enum v4l2_async_match_type - type of asynchronous subdevice logic to be used*  in order to identify a match** @V4L2_ASYNC_MATCH_CUSTOM: Match will use the logic provided by &struct*  v4l2_async_subdev.match ops* @V4L2_ASYNC_MATCH_DEVNAME: Match will use the device name* @V4L2_ASYNC_MATCH_I2C: Match will check for I2C adapter ID and address* @V4L2_ASYNC_MATCH_FWNODE: Match will use firmware node** This enum is used by the asyncrhronous sub-device logic to define the* algorithm that will be used to match an asynchronous device.*/
enum v4l2_async_match_type {// 传统的匹配方式，使用v4l2_async_subdev的match方法进行匹配V4L2_ASYNC_MATCH_CUSTOM,// 使用设备名称进行匹配V4L2_ASYNC_MATCH_DEVNAME,// 使用I2C adapter ID and address进行匹配V4L2_ASYNC_MATCH_I2C,// 使用firmware node 进行匹配V4L2_ASYNC_MATCH_FWNODE,
};

2）async桥——v4l2_async_subdev：

// 源码： include/media/v4l2-async.h/*** struct v4l2_async_subdev - sub-device descriptor, as known to a bridge** @match_type:  type of match that will be used* @match:    union of per-bus type matching data sets* @match.fwnode:*       pointer to &struct fwnode_handle to be matched.*        Used if @match_type is %V4L2_ASYNC_MATCH_FWNODE.* @match.device_name:*      string containing the device name to be matched.*       Used if @match_type is %V4L2_ASYNC_MATCH_DEVNAME.* @match.i2c:  embedded struct with I2C parameters to be matched.*     Both @match.i2c.adapter_id and @match.i2c.address*      should be matched.*     Used if @match_type is %V4L2_ASYNC_MATCH_I2C.* @match.i2c.adapter_id:*      I2C adapter ID to be matched.*      Used if @match_type is %V4L2_ASYNC_MATCH_I2C.* @match.i2c.address:*     I2C address to be matched.*     Used if @match_type is %V4L2_ASYNC_MATCH_I2C.* @match.custom:*      Driver-specific match criteria.*        Used if @match_type is %V4L2_ASYNC_MATCH_CUSTOM.* @match.custom.match:*     Driver-specific match function to be used if*       %V4L2_ASYNC_MATCH_CUSTOM.* @match.custom.priv:*     Driver-specific private struct with match parameters*       to be used if %V4L2_ASYNC_MATCH_CUSTOM.* @asd_list: used to add struct v4l2_async_subdev objects to the*        master notifier @asd_list* @list:   used to link struct v4l2_async_subdev objects, waiting to be*       probed, to a notifier->waiting list** When this struct is used as a member in a driver specific struct,* the driver specific struct shall contain the &struct* v4l2_async_subdev as its first member.*/
struct v4l2_async_subdev {// 匹配方式enum v4l2_async_match_type match_type;union {struct fwnode_handle *fwnode;// 设备名称匹配方式const char *device_name;struct {// 使用I2C adapter ID and address进行匹配int adapter_id;unsigned short address;} i2c;struct {// 传统的匹配方式bool (*match)(struct device *dev,struct v4l2_async_subdev *sd);void *priv;} custom;} match;/* v4l2-async core private: not to be used by drivers */// v4l2-async核心层使用，将此结构体挂入到notifier的waiting链表，驱动不可使用struct list_head list;struct list_head asd_list;
};

3）notifier 句柄：

// 源码： include/media/v4l2-async.h/*** struct v4l2_async_notifier_operations - Asynchronous V4L2 notifier operations* @bound: a subdevice driver has successfully probed one of the subdevices* @complete:    All subdevices have been probed successfully. The complete*     callback is only executed for the root notifier.* @unbind:  a subdevice is leaving*/
struct v4l2_async_notifier_operations {// 驱动匹配到从设备后调用此函数int (*bound)(struct v4l2_async_notifier *notifier,struct v4l2_subdev *subdev,struct v4l2_async_subdev *asd);// 所有从设备被probed成功，调用此函数int (*complete)(struct v4l2_async_notifier *notifier);// 从设备注销时调用此函数void (*unbind)(struct v4l2_async_notifier *notifier,struct v4l2_subdev *subdev,struct v4l2_async_subdev *asd);
};/*** struct v4l2_async_notifier - v4l2_device notifier data** @ops:   notifier operations* @v4l2_dev: v4l2_device of the root notifier, NULL otherwise* @sd:      sub-device that registered the notifier, NULL otherwise* @parent:   parent notifier* @asd_list: master list of struct v4l2_async_subdev* @waiting:  list of struct v4l2_async_subdev, waiting for their drivers* @done: list of struct v4l2_subdev, already probed* @list:  member in a global list of notifiers*/
struct v4l2_async_notifier {const struct v4l2_async_notifier_operations *ops;// 指向struct v4l2_devicestruct v4l2_device *v4l2_dev;struct v4l2_subdev *sd;struct v4l2_async_notifier *parent;struct list_head asd_list;// v4l2_async_subdev的链表，等待匹配driversstruct list_head waiting;// 已经probed的v4l2_subdev链表struct list_head done;// 挂在全局的notifiers链表上struct list_head list;
};

V4L2主从设备匹配过程分析

V4L2主设备和从设备采用异步的匹配方法。首先介绍一下异步匹配用到的方法。主设备使用 v4l2_async_notifier_register 函数进行异步匹配，匹配到从设备，则调用 v4l2_device_register_subdev 函数注册从设备，使用 v4l2_async_notifier_unregister 函数异步取消匹配。

从设备使用 v4l2_async_register_subdev 函数异步匹配主设备，若匹配到主设备，则调用 v4l2_device_register_subdev 函数注册从设备，使用 v4l2_async_unregister_subdev 函数异步取消匹配。

匹配的方法由 v4l2_async_subdev 结构体决定，主设备可以有多个 v4l2_async_subdev 结构体，也说明主设备有多种匹配从设备的方法。match_type表示匹配方式，由枚举 v4l2_async_match_type 定义，具体有使用设备名称匹配-V4L2_ASYNC_MATCH_DEVNAME、使用I2C adapter ID and address进行匹配-V4L2_ASYNC_MATCH_I2C等。联合体match中包含了具体的匹配信息，根据匹配方式进行设置。v4l2_async_notifier 管理整个匹配过程，未匹配的 v4l2_async_subdev 结构体被挂到waiting链表，匹配完成的挂到 done链表 同时调用 bound函数 进行绑定。

V4L2主设备匹配过程

以sun6i_csi为例分析主设备和从设备的匹配过程。

1）首先初始化需要匹配的 v4l2_async_notifier 结构体，主要设备匹配方式、bound函数、complete函数。示例如下：

a）初始化 v4l2_async_notifier 结构体
b）实现 v4l2_async_notifier ops
c）注册 v4l2_async_notifier

// 源码：  drivers/media/platform/sunxi/sun6i-csi/sun6i_csi.c
// 源码：  drivers/media/platform/sunxi/sun6i-csi/sun6i_csi.h
struct sun6i_csi {......struct v4l2_async_notifier  notifier;/* video port settings */struct v4l2_fwnode_endpoint   v4l2_ep;......
};static const struct v4l2_async_notifier_operations sun6i_csi_async_ops = {.complete = sun6i_subdev_notify_complete,
};static int sun6i_csi_v4l2_init(struct sun6i_csi *csi)
{.......// 初始化 notifier，实为初始化 notifier 的 asd_list 链表v4l2_async_notifier_init(&csi->notifier);......// 解析 endpoint，以及调用 v4l2_async_notifier_fwnode_parse_endpoint// 申请 asd， match_type = V4L2_ASYNC_MATCH_FWNODE ，// 将其添加到 notifier asd_list 链表中ret = v4l2_async_notifier_parse_fwnode_endpoints(csi->dev,&csi->notifier,sizeof(struct v4l2_async_subdev),sun6i_csi_fwnode_parse);if (ret)goto clean_video;// 初始化 notifier 的 ops，在此只实现了 complete 函数csi->notifier.ops = &sun6i_csi_async_ops;// 注册当前 notifier 到 全局 notifier 链表中，并将 当前 notifier 与 V4L2 主设备绑定ret = v4l2_async_notifier_register(&csi->v4l2_dev, &csi->notifier);if (ret) {dev_err(csi->dev, "notifier registration failed\n");goto clean_video;}return 0;......
}

（2）设置v4l2_async_notifier的v4l2_dev指针指向主设备的v4l2_device结构体。

// 源码： drivers/media/v4l2-core/v4l2-async.c
int v4l2_async_notifier_register(struct v4l2_device *v4l2_dev,struct v4l2_async_notifier *notifier)
{int ret;if (WARN_ON(!v4l2_dev || notifier->sd))return -EINVAL;// notifier 的 v4l2_dev 指向当前 主设备v4l2_devnotifier->v4l2_dev = v4l2_dev;// 注册当前 notifier 到 全局 notifier 链表中，// 并执行异步机制ret = __v4l2_async_notifier_register(notifier);if (ret)notifier->v4l2_dev = NULL;return ret;
}
EXPORT_SYMBOL(v4l2_async_notifier_register);

（3）调用 __v4l2_async_notifier_register，初始化当前 notifier 中的 waitting、done链表
（4）将当前 notifier asd_list链表中的v4l2_async_subdev，添加到其 waitting链表
（5）调用 v4l2_async_notifier_try_all_subdevs，匹配 subdev及注册subdev，并调用当前notifier bound函数
（6）匹配完成，调用 v4l2_async_notifier_try_complete进行回调当前notifier complete函数
（7）将当前 notifier 添加到全局的 notifier_list 链表中

a）__v4l2_async_notifier_register 函数解析：

// 源码： drivers/media/v4l2-core/v4l2-async.cstatic int __v4l2_async_notifier_register(struct v4l2_async_notifier *notifier)
{struct v4l2_async_subdev *asd;int ret, i = 0;INIT_LIST_HEAD(&notifier->waiting);    // 初始化 wait 链表INIT_LIST_HEAD(&notifier->done);   // 初始化 done 链表mutex_lock(&list_lock);// 遍历 asd 链表，查找 v4l2_async_subdev， 并将其添加到 waitting 链表list_for_each_entry(asd, &notifier->asd_list, asd_list) {ret = v4l2_async_notifier_asd_valid(notifier, asd, i++);if (ret)goto err_unlock;list_add_tail(&asd->list, &notifier->waiting);}// 匹配 subdev 设备， 及调用 notifier bound函数ret = v4l2_async_notifier_try_all_subdevs(notifier);if (ret < 0)goto err_unbind;// 匹配完成，调用 notifier complete函数ret = v4l2_async_notifier_try_complete(notifier);if (ret < 0)goto err_unbind;/* Keep also completed notifiers on the list */// 将当前 notifier 添加到 全局notifier_listlist_add(&notifier->list, &notifier_list);mutex_unlock(&list_lock);return 0;err_unbind:/** On failure, unbind all sub-devices registered through this notifier.*/v4l2_async_notifier_unbind_all_subdevs(notifier);err_unlock:mutex_unlock(&list_lock);return ret;
}

b）v4l2_async_notifier_try_all_subdevs 函数解析：

// 源码： drivers/media/v4l2-core/v4l2-async.c/* Test all async sub-devices in a notifier for a match. */
static int
v4l2_async_notifier_try_all_subdevs(struct v4l2_async_notifier *notifier)
{struct v4l2_device *v4l2_dev =v4l2_async_notifier_find_v4l2_dev(notifier);struct v4l2_subdev *sd;if (!v4l2_dev)return 0;again:// 遍历subdev_list链表，所有从设备的v4l2_subdev结构体都挂到subdev_list链表list_for_each_entry(sd, &subdev_list, async_list) {struct v4l2_async_subdev *asd;int ret;// 判断子设备的v4l2_subdev是否和主设备的notifier匹配，// 匹配则返回v4l2_async_subdev结构体asd = v4l2_async_find_match(notifier, sd);if (!asd)continue;// 注册 subdev，并调用 notifier 的 bound函数ret = v4l2_async_match_notify(notifier, v4l2_dev, sd, asd);if (ret < 0)return ret;/** v4l2_async_match_notify() may lead to registering a* new notifier and thus changing the async subdevs* list. In order to proceed safely from here, restart* parsing the list from the beginning.*/goto again;}return 0;
}

V4L2从设备匹配过程

以ov7251为例，进行分析主设备和从设备的匹配过程。

（1）subdev的异步注册——调用 v4l2_async_register_subdev 异步注册 subdev

//源码： drivers/media/i2c/ov7251.cstatic int ov7251_probe(struct i2c_client *client)
{struct device *dev = &client->dev;struct fwnode_handle *endpoint;struct ov7251 *ov7251;u8 chip_id_high, chip_id_low, chip_rev;int ret;......ret = v4l2_async_register_subdev(&ov7251->sd);if (ret < 0) {dev_err(dev, "could not register v4l2 device\n");goto free_entity;}......return ret;
}

（2）遍历全局 notifier_list 获取已注册的 notifier
（3）根据当前获取到的 notifier，调用 v4l2_async_find_match根据 match_type 进行匹配
（4）匹配成功，则调用 v4l2_async_match_notify 注册 subdev，以及回调 notifier 的 bound 函数
（5）bound成功，调用 v4l2_async_notifier_try_complete 回调 notifier 的 complete 函数

v4l2_async_register_subdev 函数分析：

// 源码： drivers/media/v4l2-core/v4l2-async.cint v4l2_async_register_subdev(struct v4l2_subdev *sd)
{struct v4l2_async_notifier *subdev_notifier;struct v4l2_async_notifier *notifier;int ret;/** No reference taken. The reference is held by the device* (struct v4l2_subdev.dev), and async sub-device does not* exist independently of the device at any point of time.*/if (!sd->fwnode && sd->dev)sd->fwnode = dev_fwnode(sd->dev);mutex_lock(&list_lock);// 初始化当前 subdev 的 async_list 链表INIT_LIST_HEAD(&sd->async_list);// 遍历 全局notifier_list，查找已注册的 notifierlist_for_each_entry(notifier, &notifier_list, list) {// 获取当前 notifier 的主设备 v4l2_devicestruct v4l2_device *v4l2_dev =v4l2_async_notifier_find_v4l2_dev(notifier);struct v4l2_async_subdev *asd;if (!v4l2_dev)continue;// 调用match函数，进行设备匹配// 匹配方式有以下几种：// V4L2_ASYNC_MATCH_CUSTOM// V4L2_ASYNC_MATCH_DEVNAME// V4L2_ASYNC_MATCH_I2C// V4L2_ASYNC_MATCH_FWNODEasd = v4l2_async_find_match(notifier, sd);if (!asd)continue;// 注册subdev设备，并调用notify中的bound函数ret = v4l2_async_match_notify(notifier, v4l2_dev, sd, asd);if (ret)goto err_unbind;// 调用notify中的complete函数ret = v4l2_async_notifier_try_complete(notifier);if (ret)goto err_unbind;goto out_unlock;}/* None matched, wait for hot-plugging */list_add(&sd->async_list, &subdev_list);out_unlock:mutex_unlock(&list_lock);return 0;err_unbind:/** Complete failed. Unbind the sub-devices bound through registering* this async sub-device.*/subdev_notifier = v4l2_async_find_subdev_notifier(sd);if (subdev_notifier)v4l2_async_notifier_unbind_all_subdevs(subdev_notifier);if (sd->asd)v4l2_async_notifier_call_unbind(notifier, sd, sd->asd);v4l2_async_cleanup(sd);mutex_unlock(&list_lock);return ret;
}

V4L2异步注册过程

（1）主从设备 match

// 源码： drivers/media/v4l2-core/v4l2-async.c
static bool match_i2c(struct v4l2_subdev *sd, struct v4l2_async_subdev *asd)
{
#if IS_ENABLED(CONFIG_I2C)struct i2c_client *client = i2c_verify_client(sd->dev);return client &&asd->match.i2c.adapter_id == client->adapter->nr &&asd->match.i2c.address == client->addr;
#elsereturn false;
#endif
}static bool match_devname(struct v4l2_subdev *sd,struct v4l2_async_subdev *asd)
{return !strcmp(asd->match.device_name, dev_name(sd->dev));
}static bool match_fwnode(struct v4l2_subdev *sd, struct v4l2_async_subdev *asd)
{return sd->fwnode == asd->match.fwnode;
}static bool match_custom(struct v4l2_subdev *sd, struct v4l2_async_subdev *asd)
{if (!asd->match.custom.match)/* Match always */return true;return asd->match.custom.match(sd->dev, asd);
}static struct v4l2_async_subdev *
v4l2_async_find_match(struct v4l2_async_notifier *notifier,struct v4l2_subdev *sd)
{// 定义匹配的函数指针bool (*match)(struct v4l2_subdev *sd, struct v4l2_async_subdev *asd);struct v4l2_async_subdev *asd;list_for_each_entry(asd, &notifier->waiting, list) {/* bus_type has been verified valid before */// 确认匹配方式switch (asd->match_type) {case V4L2_ASYNC_MATCH_CUSTOM: // 传统的匹配方式match = match_custom;break;case V4L2_ASYNC_MATCH_DEVNAME: // 设备名称匹配方法match = match_devname;break;case V4L2_ASYNC_MATCH_I2C:       // i2c 匹配方法match = match_i2c;break;case V4L2_ASYNC_MATCH_FWNODE:    // firmware node 匹配方法match = match_fwnode;break;default:/* Cannot happen, unless someone breaks us */WARN_ON(true);return NULL;}/* match cannot be NULL here */// 根据匹配方式，调用相应的匹配方法if (match(sd, asd))return asd;}return NULL;
}

（2）主从设备 bound

// 源码： drivers/media/v4l2-core/v4l2-async.cstatic int v4l2_async_match_notify(struct v4l2_async_notifier *notifier,struct v4l2_device *v4l2_dev,struct v4l2_subdev *sd,struct v4l2_async_subdev *asd)
{struct v4l2_async_notifier *subdev_notifier;int ret;// 注册 subdevret = v4l2_device_register_subdev(v4l2_dev, sd);if (ret < 0)return ret;// bound非空，则调用bound函数，// bound函数 的主要作用是设置主设备的v4l2_subdev指针，// 使其指向匹配的从设备的v4l2_subdev结构体，从而完成主设备到从设备的绑定ret = v4l2_async_notifier_call_bound(notifier, sd, asd);if (ret < 0) {v4l2_device_unregister_subdev(sd);return ret;}/* Remove from the waiting list */// 将 asd 从 waitting链表 移除list_del(&asd->list);// subdev 绑定 asd 及 notifiersd->asd = asd;sd->notifier = notifier;/* Move from the global subdevice list to notifier's done */// 将subdevice从async_list链表中移除后挂到done链表中list_move(&sd->async_list, &notifier->done);/** See if the sub-device has a notifier. If not, return here.*/// 检查 subdev 是否有 notifier，有则调用，无则到此，就地返回subdev_notifier = v4l2_async_find_subdev_notifier(sd);if (!subdev_notifier || subdev_notifier->parent)return 0;/** Proceed with checking for the sub-device notifier's async* sub-devices, and return the result. The error will be handled by the* caller.*/subdev_notifier->parent = notifier;return v4l2_async_notifier_try_all_subdevs(subdev_notifier);
}

（3）主从设备complete

// 源码： drivers/media/v4l2-core/v4l2-async.c/** Complete the master notifier if possible. This is done when all async* sub-devices have been bound; v4l2_device is also available then.*/
static int
v4l2_async_notifier_try_complete(struct v4l2_async_notifier *notifier)
{/* Quick check whether there are still more sub-devices here. */if (!list_empty(&notifier->waiting))return 0;/* Check the entire notifier tree; find the root notifier first. */while (notifier->parent)notifier = notifier->parent;/* This is root if it has v4l2_dev. */if (!notifier->v4l2_dev)return 0;/* Is everything ready? */if (!v4l2_async_notifier_can_complete(notifier))return 0;return v4l2_async_notifier_call_complete(notifier);
}