Linux内核之XArray

https://www.kernel.org/doc/html/v5.0/core-api/xarray.html

https://elixir.bootlin.com/linux/latest/source/include/linux/xarray.h

Table of Contents

总览

普通API

分配XArray

内存分配

锁定

进阶API

内部条目

附加功能

多索引条目

总览

XArray是一种抽象的数据类型，其行为类似于一个非常大的指针数组。它满足了与散列或常规可调整大小的数组相同的许多需求。与散列不同，它允许您以高效缓存的方式明智地转到下一个或上一个条目。与可调整大小的阵列相比，无需为了扩展阵列而复制数据或更改MMU映射。与双向链接列表相比，它具有更高的内存效率，可并行性和缓存友好性。它利用RCU来执行查找而无需锁定。

当使用的索引密集聚集时，XArray实现非常有效。对对象进行哈希处理并将哈希值用作索引将不能很好地执行。XArray针对小索引进行了优化，但是对于大索引仍然具有良好的性能。如果索引可以大于， ULONG_MAX则XArray不是您的数据类型。XArray最重要的用户是页面缓存。

NULL数组中的每个非条目都有与其关联的三个位，称为标记。每个标记可以独立设置或清除。您可以遍历已标记的条目。

普通指针可以直接存储在XArray中。它们必须对齐4个字节，这对于从kmalloc()和返回的任何指针都是正确的alloc_page()。对于任意用户空间指针或函数指针而言，它不是正确的。您可以存储指向静态分配的对象的指针，只要这些对象的对齐至少为4。

您还可以LONG_MAX在XArray中存储0到之间的整数。您必须先使用将其转换为条目xa_mk_value()。从XArray检索条目时，可以通过调用检查它是否为值条目xa_is_value()，然后通过调用将其转换回整数xa_to_value()。

一些用户希望存储带标签的指针，而不是使用上述标记。他们可以调用xa_tag_pointer()以创建带有标签的条目，xa_untag_pointer()将带标签的条目变回无标签的指针，并xa_pointer_tag()检索条目的标签。标记指针使用相同的位来区分值条目和普通指针，因此每个用户必须决定是否要在任何特定XArray中存储值条目或标记指针。

XArray不支持存储IS_ERR()指针，因为它与值条目或内部条目有些冲突。

XArray的一个不寻常的功能是能够创建占用一系列索引的条目。一旦存储到其中，查找范围内的任何索引将返回与查找范围内的任何其他索引相同的条目。在一个索引上设置标记将在所有索引上设置它。存储到任何索引将存储到所有索引。可以将多索引条目显式拆分为较小的条目，或者将其存储NULL到任何条目中都会使XArray忘记范围。

普通API

从初始化XArray开始，DEFINE_XARRAY() 对于静态分配的XArray或xa_init()动态分配的XArray都要初始化。刚初始化的XArray NULL 在每个索引处都包含一个指针。

然后，您可以使用设置条目，xa_store()并使用获取条目xa_load()。xa_store将用新条目覆盖任何条目，并返回存储在该索引处的先前条目。您可以使用xa_erase()而不是xa_store()使用 NULL条目来调用。从未存储到的条目，已删除的条目和最近NULL存储到的条目之间没有区别。

您可以使用来有条件地替换索引中的条目 xa_cmpxchg()。像一样cmpxchg()，只有在该索引的条目具有“旧”值时，它才会成功。它还返回该索引处的条目。如果它返回与“旧”相同的条目，则xa_cmpxchg()成功。

如果您只想在索引的当前条目为的情况下仅将新条目存储到索引，则NULL可以使用如果条目不为空xa_insert()返回-EEXIST的值。

您可以使用来查询条目上是否设置了标记 xa_get_mark()。如果不是该条目，则NULL可以通过使用在其上设置标记，xa_set_mark()并通过调用从条目中删除该标记xa_clear_mark()。您可以通过调用询问XArray中的任何条目是否设置了特定标记xa_marked()。

您可以通过调用将条目从XArray复制到纯数组中 xa_extract()。或者，您可以通过调用遍历XArray中的当前条目xa_for_each()。您可能更喜欢使用 xa_find()或xa_find_after()移至XArray中的下一个当前条目。

调用xa_store_range()将同一条目存储在一系列索引中。如果执行此操作，则其他一些操作的行为会有些奇怪。例如，在一个索引上标记条目可能会导致在其他但不是所有其他索引上标记条目。存储到一个索引中可能会导致某些（但不是所有）其他索引检索到的条目发生变化。

有时您需要确保后续调用xa_store() 不需要分配内存。该xa_reserve()函数将在指示的索引处存储保留的条目。普通API的用户将看到此条目包含NULL。如果不需要使用保留的条目，则可以调用xa_release() 以删除未使用的条目。如果与此同时有另一个用户存储到该条目，xa_release()则不执行任何操作；相反，如果您希望该条目成为NULL，则应使用xa_erase()。使用xa_insert()上的保留条目将失败。

如果数组中的所有条目都是NULL，xa_empty()函数将返回true。

最后，您可以通过调用从XArray删除所有条目 xa_destroy()。如果XArray条目是指针，则您可能希望先释放这些条目。您可以通过使用xa_for_each()迭代器遍历XArray中的所有当前条目来实现此目的。

分配XArray

如果您用于DEFINE_XARRAY_ALLOC()定义XArray或通过将其传递XA_FLAGS_ALLOC给XArray进行初始化xa_init_flags()，则XArray会更改以跟踪条目是否在使用中。

您可以调用xa_alloc()将条目存储在XArray中任何未使用的索引处。如果需要从中断上下文中修改数组，则可以在分配ID时使用xa_alloc_bh()或xa_alloc_irq()禁用中断。

使用xa_store()，xa_cmpxchg()或xa_insert() 将迎来条目被分配。与普通的XArray不同，存储 NULL会将该条目标记为正在使用中，例如xa_reserve()。要释放条目，请使用xa_erase()（或者xa_release()如果您只想释放条目，则使用NULL）。

您不能XA_MARK_0与分配XArray一起使用，因为此标记用于跟踪条目是否空闲。其他标记可供您使用。

内存分配

的xa_store()，xa_cmpxchg()，xa_alloc()， xa_reserve()和xa_insert()功能采取以防XArray需要分配存储器来存储这个条目的gfp_t参数。如果要删除该条目，则无需执行任何内存分配，并且指定的GFP标志将被忽略。

不可能分配任何内存，特别是如果您传递一组限制性的GFP标志时。在这种情况下，函数将返回一个特殊值，可以使用将其转换为errno xa_err()。如果您不需要确切地知道发生了哪个错误，使用起来 xa_is_err()会更有效率。

锁定

使用Normal API时，您不必担心锁定。XArray使用RCU和内部自旋锁来同步访问：

无需锁定：

xa_empty()
xa_marked()

采取RCU读取锁定：

xa_load()
xa_for_each()
xa_find()
xa_find_after()
xa_extract()
xa_get_mark()

内部采用xa_lock：

xa_store()
xa_store_bh()
xa_store_irq()
xa_insert()
xa_insert_bh()
xa_insert_irq()
xa_erase()
xa_erase_bh()
xa_erase_irq()
xa_cmpxchg()
xa_cmpxchg_bh()
xa_cmpxchg_irq()
xa_store_range()
xa_alloc()
xa_alloc_bh()
xa_alloc_irq()
xa_reserve()
xa_reserve_bh()
xa_reserve_irq()
xa_destroy()
xa_set_mark()
xa_clear_mark()

假设xa_lock保留在条目上：

__xa_store()
__xa_insert()
__xa_erase()
__xa_cmpxchg()
__xa_alloc()
__xa_reserve()
__xa_set_mark()
__xa_clear_mark()

如果您想利用锁来保护要存储在XArray中的数据结构，可以在调用xa_lock() 之前调用xa_load()，然后在调用之前对找到的对象进行引用计数xa_unlock()。这将防止商店在查找对象和增加引用计数之间从数组中删除对象。您也可以使用RCU避免取消引用释放的内存，但是对此的解释超出了本文的范围。

XArray在修改阵列时不会禁用中断或softirq。从RCU锁定提供足够的保护，从中断或softirq上下文中读取XArray是安全的。

例如，如果要在进程上下文中将条目存储在XArray中，然后在softirq上下文中删除它们，则可以通过以下方式进行操作：

void foo_init(struct foo *foo)
{xa_init_flags(&foo->array, XA_FLAGS_LOCK_BH);
}int foo_store(struct foo *foo, unsigned long index, void *entry)
{int err;xa_lock_bh(&foo->array);err = xa_err(__xa_store(&foo->array, index, entry, GFP_KERNEL));if (!err)foo->count++;xa_unlock_bh(&foo->array);return err;
}/* foo_erase() is only called from softirq context */
void foo_erase(struct foo *foo, unsigned long index)
{xa_lock(&foo->array);__xa_erase(&foo->array, index);foo->count--;xa_unlock(&foo->array);
}

如果要从中断或softirq上下文修改XArray，则需要使用xa_init_flags()， XA_FLAGS_LOCK_IRQ或初始化数组XA_FLAGS_LOCK_BH。

上面的示例还显示了一种常见的模式，该模式希望扩展商店侧xa_lock的覆盖范围，以保护与该数组关联的某些统计信息。

与中断上下文共享XArray也是可能的，或者使用xa_lock_irqsave()两个中断处理程序和进程上下文，或xa_lock_irq()在进程上下文和xa_lock() 在中断处理程序。一些较常见的模式有帮助的功能，例如xa_store_bh()，xa_store_irq()， xa_erase_bh()，xa_erase_irq()，xa_cmpxchg_bh() 和xa_cmpxchg_irq()。

有时您需要使用互斥锁来保护对XArray的访问，因为该锁位于锁定层次结构中的另一个互斥锁之上。这并不意味着您有权使用诸如__xa_erase()不带xa_lock之类的功能；xa_lock用于lockdep验证，将来将用于其他目的。

该__xa_set_mark()和__xa_clear_mark()功能也可用于在那里你仰望的条目并想原子设置或清除标记的情况。在这种情况下，使用高级API可能会更有效率，因为它将使您免于两次行走。

进阶API

先进的API以接口为代价提供了更高的灵活性和更好的性能，而接口却可能更难使用且保护措施更少。高级API不会为您完成锁定，并且要求您在修改数组时使用xa_lock。您可以选择在阵列上执行只读操作时使用xa_lock还是RCU锁。您可以在同一阵列上混合使用高级操作和普通操作。实际上，普通API是根据高级API实现的。高级API仅适用于具有GPL兼容许可证的模块。

高级API基于xa_state。这是一个不透明的数据结构，您可以使用XA_STATE() 宏在堆栈上声明。该宏初始化了准备开始在XArray周围移动的xa_state。它用作游标以维护XArray中的位置，并让您一起进行各种操作，而不必每次都从顶部重新启动。

xa_state也用于存储错误。您可以致电 xas_error()以检索错误。所有操作都将在继续操作之前检查xa_state是否处于错误状态，因此您无需在每次调用后检查错误；您可以连续拨打多个电话，并且只能在方便的时候进行检查。XArray代码本身当前生成的唯一错误是ENOMEM和 EINVAL，但是它支持任意错误，以备您自称 xas_set_err()。

如果xa_state ENOMEM出错，则调用xas_nomem() 将尝试使用指定的gfp标志分配更多的内存，并将其缓存在xa_state中，以备下次尝试。这个想法是您采用xa_lock，尝试操作并放弃锁。该操作尝试在持有锁定的同时分配内存，但是更有可能失败。放开锁后，xas_nomem() 可以尝试分配更多的内存。true如果值得重试该操作（即存在内存错误并且分配了更多的内存），它将返回。如果它先前已经分配了内存，并且没有使用该内存，并且没有错误（或某些不是not的错误 ENOMEM），那么它将释放先前分配的内存。

内部条目

XArray为自己的目的保留了一些条目。这些通常不会通过常规API公开，但是在使用高级API时，可能会看到它们。通常，处理它们的最佳方法是将它们传递给xas_retry()，如果返回则重试该操作true。

名称	测试	用法
节点	`xa_is_node()`	XArray节点。使用多索引xa_state时可能可见。
兄弟	`xa_is_sibling()`	多索引条目的非规范条目。该值指示此节点中的哪个插槽具有规范条目。
重试	`xa_is_retry()`	该条目当前正在由具有xa_lock的线程修改。包含该条目的节点可以在该RCU周期结束时释放。您应该从数组的开头重新开始查找。
零	`xa_is_zero()`	零条目的显示`NULL`方式与普通API相同，但在XArray中占据了一个条目，可用于保留索引以备将来使用。这是通过为分配的条目分配XArray来使用的`NULL`。

将来可能会添加其他内部条目。尽可能由处理xas_retry()。

附加功能

该xas_create_range()函数分配所有必需的内存以存储范围内的每个条目。如果无法分配内存，它将在xa_state中设置ENOMEM。

您可以使用xas_init_marks()将条目上的标记重置为其默认状态。通常，所有标记都是清除的，除非XArray被标记为XA_FLAGS_TRACK_FREE，在这种情况下，标记0被设置，所有其他标记都被清除。将一个条目替换为另一个条目 xas_store()不会重置该条目上的标记；如果要重设标记，则应明确地进行。

该xas_load()会走xa_state尽可能靠近入口，因为它可以。如果您知道xa_state已经移至该条目并且需要检查该条目没有更改，则可以使用 xas_reload()保存函数调用。

如果您需要移至XArray中的其他索引，请调用 xas_set()。这会将光标重置到树的顶部，这通常会使下一个操作将光标移到树中的所需位置。如果要移至下一个或上一个索引，请调用xas_next()或xas_prev()。设置索引不会使游标在数组中移动，因此在移至下一个或上一个索引时不需要保持锁定。

您可以使用搜索下一个当前条目xas_find()。这等同于xa_find()和xa_find_after(); 如果光标已移至某个条目，则它将在当前引用的条目之后找到下一个条目。如果不是，它将在xa_state的索引处返回条目。在大多数情况下，使用xas_next_entry()而不是移动到下一个当前条目xas_find()将节省函数调用，但以发出更多内联代码为代价。

该xas_find_marked()功能是相似的。如果尚未遍历xa_state，它将被标记xa_state的索引处返回条目。否则，它将返回xa_state引用的条目之后的第一个标记条目。该xas_next_marked() 功能等效于xas_next_entry()。

当使用xas_for_each() 或迭代XArray的范围时xas_for_each_marked()，可能需要暂时停止迭代。该xas_pause()函数存在用于此目的。完成必要的工作并希望恢复后，xa_state处于适当的状态，可以在上次处理条目之后继续迭代。如果您在迭代时禁用了中断，那么暂停迭代并重新启用每个XA_CHECK_SCHED条目的中断是一种很好的方式。

的xas_get_mark()，xas_set_mark()和 xas_clear_mark()功能需要xa_state光标已移动到xarray适当的位置; 如果您已致电xas_pause()或xas_set() 之前致电，他们将无能为力。

您可以xas_set_update()在每次XArray更新节点时调用来调用一个回调函数。页面缓存工作集代码使用它来维护其仅包含影子条目的节点列表。

多索引条目

XArray能够将多个索引绑定在一起，因此对一个索引的操作会影响所有索引。例如，存储到任何索引中都会更改从任何索引中检索到的条目的值。在任何索引上设置或清除标记将在每个捆绑在一起的索引上设置或清除标记。当前的实现方式仅允许将两个均方幂对齐的范围绑定在一起。例如，索引64-127可以捆绑在一起，但2-6可以不捆绑在一起。这样可以节省大量内存。例如，将512个条目捆绑在一起将节省超过4kB。

您可以通过使用XA_STATE_ORDER() 或xas_set_order()调用来创建多索引条目xas_store()。xas_load()使用多索引xa_state进行调用会将xa_state移到树中的正确位置，但是返回值没有意义，可能是内部条目，NULL甚至当范围内存储了条目时也是如此。调用xas_find_conflict() 将返回该范围内的第一个条目，或者该范围内NULL没有条目。该xas_for_each_conflict()迭代器将每一个重叠指定范围项遍历。

如果xas_load()遇到多索引条目，则不会更改xa_state中的xa_index。在XArray上进行迭代或调用时xas_find()，如果初始索引位于多索引条目的中间，则它不会被更改。随后的调用或迭代会将索引移至该范围内的第一个索引。每个条目仅返回一次，无论它占用多少索引。

使用xas_next()或xas_prev()用多指标xa_state不支持。在多索引条目上使用这两个功能之一将显示同级条目。这些应由调用者跳过。

存储NULL到多索引条目的任何索引中将把每个索引处的条目设置为NULL并消除联系。尚不支持将多索引条目拆分为较小范围的条目。

功能和结构

略。

xarray.h

/* SPDX-License-Identifier: GPL-2.0+ */
#ifndef _LINUX_XARRAY_H
#define _LINUX_XARRAY_H
/** eXtensible Arrays* Copyright (c) 2017 Microsoft Corporation* Author: Matthew Wilcox <willy@infradead.org>** See Documentation/core-api/xarray.rst for how to use the XArray.*/#include <linux/bug.h>
#include <linux/compiler.h>
#include <linux/gfp.h>
#include <linux/kconfig.h>
#include <linux/kernel.h>
#include <linux/rcupdate.h>
#include <linux/spinlock.h>
#include <linux/types.h>/** The bottom two bits of the entry determine how the XArray interprets* the contents:** 00: Pointer entry* 10: Internal entry* x1: Value entry or tagged pointer** Attempting to store internal entries in the XArray is a bug.** Most internal entries are pointers to the next node in the tree.* The following internal entries have a special meaning:** 0-62: Sibling entries* 256: Zero entry* 257: Retry entry** Errors are also represented as internal entries, but use the negative* space (-4094 to -2).  They're never stored in the slots array; only* returned by the normal API.*/#define BITS_PER_XA_VALUE  (BITS_PER_LONG - 1)/*** xa_mk_value() - Create an XArray entry from an integer.* @v: Value to store in XArray.** Context: Any context.* Return: An entry suitable for storing in the XArray.*/
static inline void *xa_mk_value(unsigned long v)
{WARN_ON((long)v < 0);return (void *)((v << 1) | 1);
}/*** xa_to_value() - Get value stored in an XArray entry.* @entry: XArray entry.** Context: Any context.* Return: The value stored in the XArray entry.*/
static inline unsigned long xa_to_value(const void *entry)
{return (unsigned long)entry >> 1;
}/*** xa_is_value() - Determine if an entry is a value.* @entry: XArray entry.** Context: Any context.* Return: True if the entry is a value, false if it is a pointer.*/
static inline bool xa_is_value(const void *entry)
{return (unsigned long)entry & 1;
}/*** xa_tag_pointer() - Create an XArray entry for a tagged pointer.* @p: Plain pointer.* @tag: Tag value (0, 1 or 3).** If the user of the XArray prefers, they can tag their pointers instead* of storing value entries.  Three tags are available (0, 1 and 3).* These are distinct from the xa_mark_t as they are not replicated up* through the array and cannot be searched for.** Context: Any context.* Return: An XArray entry.*/
static inline void *xa_tag_pointer(void *p, unsigned long tag)
{return (void *)((unsigned long)p | tag);
}/*** xa_untag_pointer() - Turn an XArray entry into a plain pointer.* @entry: XArray entry.** If you have stored a tagged pointer in the XArray, call this function* to get the untagged version of the pointer.** Context: Any context.* Return: A pointer.*/
static inline void *xa_untag_pointer(void *entry)
{return (void *)((unsigned long)entry & ~3UL);
}/*** xa_pointer_tag() - Get the tag stored in an XArray entry.* @entry: XArray entry.** If you have stored a tagged pointer in the XArray, call this function* to get the tag of that pointer.** Context: Any context.* Return: A tag.*/
static inline unsigned int xa_pointer_tag(void *entry)
{return (unsigned long)entry & 3UL;
}/** xa_mk_internal() - Create an internal entry.* @v: Value to turn into an internal entry.** Internal entries are used for a number of purposes.  Entries 0-255 are* used for sibling entries (only 0-62 are used by the current code).  256* is used for the retry entry.  257 is used for the reserved / zero entry.* Negative internal entries are used to represent errnos.  Node pointers* are also tagged as internal entries in some situations.** Context: Any context.* Return: An XArray internal entry corresponding to this value.*/
static inline void *xa_mk_internal(unsigned long v)
{return (void *)((v << 2) | 2);
}/** xa_to_internal() - Extract the value from an internal entry.* @entry: XArray entry.** Context: Any context.* Return: The value which was stored in the internal entry.*/
static inline unsigned long xa_to_internal(const void *entry)
{return (unsigned long)entry >> 2;
}/** xa_is_internal() - Is the entry an internal entry?* @entry: XArray entry.** Context: Any context.* Return: %true if the entry is an internal entry.*/
static inline bool xa_is_internal(const void *entry)
{return ((unsigned long)entry & 3) == 2;
}#define XA_ZERO_ENTRY      xa_mk_internal(257)/*** xa_is_zero() - Is the entry a zero entry?* @entry: Entry retrieved from the XArray** The normal API will return NULL as the contents of a slot containing* a zero entry.  You can only see zero entries by using the advanced API.** Return: %true if the entry is a zero entry.*/
static inline bool xa_is_zero(const void *entry)
{return unlikely(entry == XA_ZERO_ENTRY);
}/*** xa_is_err() - Report whether an XArray operation returned an error* @entry: Result from calling an XArray function** If an XArray operation cannot complete an operation, it will return* a special value indicating an error.  This function tells you* whether an error occurred; xa_err() tells you which error occurred.** Context: Any context.* Return: %true if the entry indicates an error.*/
static inline bool xa_is_err(const void *entry)
{return unlikely(xa_is_internal(entry) &&entry >= xa_mk_internal(-MAX_ERRNO));
}/*** xa_err() - Turn an XArray result into an errno.* @entry: Result from calling an XArray function.** If an XArray operation cannot complete an operation, it will return* a special pointer value which encodes an errno.  This function extracts* the errno from the pointer value, or returns 0 if the pointer does not* represent an errno.** Context: Any context.* Return: A negative errno or 0.*/
static inline int xa_err(void *entry)
{/* xa_to_internal() would not do sign extension. */if (xa_is_err(entry))return (long)entry >> 2;return 0;
}/*** struct xa_limit - Represents a range of IDs.* @min: The lowest ID to allocate (inclusive).* @max: The maximum ID to allocate (inclusive).** This structure is used either directly or via the XA_LIMIT() macro* to communicate the range of IDs that are valid for allocation.* Two common ranges are predefined for you:* * xa_limit_32b - [0 - UINT_MAX]* * xa_limit_31b    - [0 - INT_MAX]*/
struct xa_limit {u32 max;u32 min;
};#define XA_LIMIT(_min, _max) (struct xa_limit) { .min = _min, .max = _max }#define xa_limit_32b   XA_LIMIT(0, UINT_MAX)
#define xa_limit_31b    XA_LIMIT(0, INT_MAX)typedef unsigned __bitwise xa_mark_t;
#define XA_MARK_0       ((__force xa_mark_t)0U)
#define XA_MARK_1       ((__force xa_mark_t)1U)
#define XA_MARK_2       ((__force xa_mark_t)2U)
#define XA_PRESENT      ((__force xa_mark_t)8U)
#define XA_MARK_MAX     XA_MARK_2
#define XA_FREE_MARK        XA_MARK_0enum xa_lock_type {XA_LOCK_IRQ = 1,XA_LOCK_BH = 2,
};/** Values for xa_flags.  The radix tree stores its GFP flags in the xa_flags,* and we remain compatible with that.*/
#define XA_FLAGS_LOCK_IRQ   ((__force gfp_t)XA_LOCK_IRQ)
#define XA_FLAGS_LOCK_BH    ((__force gfp_t)XA_LOCK_BH)
#define XA_FLAGS_TRACK_FREE ((__force gfp_t)4U)
#define XA_FLAGS_ZERO_BUSY  ((__force gfp_t)8U)
#define XA_FLAGS_ALLOC_WRAPPED  ((__force gfp_t)16U)
#define XA_FLAGS_ACCOUNT    ((__force gfp_t)32U)
#define XA_FLAGS_MARK(mark) ((__force gfp_t)((1U << __GFP_BITS_SHIFT) << \(__force unsigned)(mark)))/* ALLOC is for a normal 0-based alloc.  ALLOC1 is for an 1-based alloc */
#define XA_FLAGS_ALLOC  (XA_FLAGS_TRACK_FREE | XA_FLAGS_MARK(XA_FREE_MARK))
#define XA_FLAGS_ALLOC1 (XA_FLAGS_TRACK_FREE | XA_FLAGS_ZERO_BUSY)/*** struct xarray - The anchor of the XArray.* @xa_lock: Lock that protects the contents of the XArray.** To use the xarray, define it statically or embed it in your data structure.* It is a very small data structure, so it does not usually make sense to* allocate it separately and keep a pointer to it in your data structure.** You may use the xa_lock to protect your own data structures as well.*/
/** If all of the entries in the array are NULL, @xa_head is a NULL pointer.* If the only non-NULL entry in the array is at index 0, @xa_head is that* entry.  If any other entry in the array is non-NULL, @xa_head points* to an @xa_node.*/
struct xarray {spinlock_t   xa_lock;
/* private: The rest of the data structure is not to be used directly. */gfp_t      xa_flags;void __rcu *   xa_head;
};#define XARRAY_INIT(name, flags) {                \.xa_lock = __SPIN_LOCK_UNLOCKED(name.xa_lock),     \.xa_flags = flags,                 \.xa_head = NULL,                   \
}/*** DEFINE_XARRAY_FLAGS() - Define an XArray with custom flags.* @name: A string that names your XArray.* @flags: XA_FLAG values.** This is intended for file scope definitions of XArrays.  It declares* and initialises an empty XArray with the chosen name and flags.  It is* equivalent to calling xa_init_flags() on the array, but it does the* initialisation at compiletime instead of runtime.*/
#define DEFINE_XARRAY_FLAGS(name, flags)                \struct xarray name = XARRAY_INIT(name, flags)/*** DEFINE_XARRAY() - Define an XArray.* @name: A string that names your XArray.** This is intended for file scope definitions of XArrays.  It declares* and initialises an empty XArray with the chosen name.  It is equivalent* to calling xa_init() on the array, but it does the initialisation at* compiletime instead of runtime.*/
#define DEFINE_XARRAY(name) DEFINE_XARRAY_FLAGS(name, 0)/*** DEFINE_XARRAY_ALLOC() - Define an XArray which allocates IDs starting at 0.* @name: A string that names your XArray.** This is intended for file scope definitions of allocating XArrays.* See also DEFINE_XARRAY().*/
#define DEFINE_XARRAY_ALLOC(name) DEFINE_XARRAY_FLAGS(name, XA_FLAGS_ALLOC)/*** DEFINE_XARRAY_ALLOC1() - Define an XArray which allocates IDs starting at 1.* @name: A string that names your XArray.** This is intended for file scope definitions of allocating XArrays.* See also DEFINE_XARRAY().*/
#define DEFINE_XARRAY_ALLOC1(name) DEFINE_XARRAY_FLAGS(name, XA_FLAGS_ALLOC1)void *xa_load(struct xarray *, unsigned long index);
void *xa_store(struct xarray *, unsigned long index, void *entry, gfp_t);
void *xa_erase(struct xarray *, unsigned long index);
void *xa_store_range(struct xarray *, unsigned long first, unsigned long last,void *entry, gfp_t);
bool xa_get_mark(struct xarray *, unsigned long index, xa_mark_t);
void xa_set_mark(struct xarray *, unsigned long index, xa_mark_t);
void xa_clear_mark(struct xarray *, unsigned long index, xa_mark_t);
void *xa_find(struct xarray *xa, unsigned long *index,unsigned long max, xa_mark_t) __attribute__((nonnull(2)));
void *xa_find_after(struct xarray *xa, unsigned long *index,unsigned long max, xa_mark_t) __attribute__((nonnull(2)));
unsigned int xa_extract(struct xarray *, void **dst, unsigned long start,unsigned long max, unsigned int n, xa_mark_t);
void xa_destroy(struct xarray *);/*** xa_init_flags() - Initialise an empty XArray with flags.* @xa: XArray.* @flags: XA_FLAG values.** If you need to initialise an XArray with special flags (eg you need* to take the lock from interrupt context), use this function instead* of xa_init().** Context: Any context.*/
static inline void xa_init_flags(struct xarray *xa, gfp_t flags)
{spin_lock_init(&xa->xa_lock);xa->xa_flags = flags;xa->xa_head = NULL;
}/*** xa_init() - Initialise an empty XArray.* @xa: XArray.** An empty XArray is full of NULL entries.** Context: Any context.*/
static inline void xa_init(struct xarray *xa)
{xa_init_flags(xa, 0);
}/*** xa_empty() - Determine if an array has any present entries.* @xa: XArray.** Context: Any context.* Return: %true if the array contains only NULL pointers.*/
static inline bool xa_empty(const struct xarray *xa)
{return xa->xa_head == NULL;
}/*** xa_marked() - Inquire whether any entry in this array has a mark set* @xa: Array* @mark: Mark value** Context: Any context.* Return: %true if any entry has this mark set.*/
static inline bool xa_marked(const struct xarray *xa, xa_mark_t mark)
{return xa->xa_flags & XA_FLAGS_MARK(mark);
}/*** xa_for_each_range() - Iterate over a portion of an XArray.* @xa: XArray.* @index: Index of @entry.* @entry: Entry retrieved from array.* @start: First index to retrieve from array.* @last: Last index to retrieve from array.** During the iteration, @entry will have the value of the entry stored* in @xa at @index.  You may modify @index during the iteration if you* want to skip or reprocess indices.  It is safe to modify the array* during the iteration.  At the end of the iteration, @entry will be set* to NULL and @index will have a value less than or equal to max.** xa_for_each_range() is O(n.log(n)) while xas_for_each() is O(n).  You have* to handle your own locking with xas_for_each(), and if you have to unlock* after each iteration, it will also end up being O(n.log(n)).* xa_for_each_range() will spin if it hits a retry entry; if you intend to* see retry entries, you should use the xas_for_each() iterator instead.* The xas_for_each() iterator will expand into more inline code than* xa_for_each_range().** Context: Any context.  Takes and releases the RCU lock.*/
#define xa_for_each_range(xa, index, entry, start, last)        \for (index = start,                        \entry = xa_find(xa, &index, last, XA_PRESENT);     \entry;                         \entry = xa_find_after(xa, &index, last, XA_PRESENT))/*** xa_for_each_start() - Iterate over a portion of an XArray.* @xa: XArray.* @index: Index of @entry.* @entry: Entry retrieved from array.* @start: First index to retrieve from array.** During the iteration, @entry will have the value of the entry stored* in @xa at @index.  You may modify @index during the iteration if you* want to skip or reprocess indices.  It is safe to modify the array* during the iteration.  At the end of the iteration, @entry will be set* to NULL and @index will have a value less than or equal to max.** xa_for_each_start() is O(n.log(n)) while xas_for_each() is O(n).  You have* to handle your own locking with xas_for_each(), and if you have to unlock* after each iteration, it will also end up being O(n.log(n)).* xa_for_each_start() will spin if it hits a retry entry; if you intend to* see retry entries, you should use the xas_for_each() iterator instead.* The xas_for_each() iterator will expand into more inline code than* xa_for_each_start().** Context: Any context.  Takes and releases the RCU lock.*/
#define xa_for_each_start(xa, index, entry, start) \xa_for_each_range(xa, index, entry, start, ULONG_MAX)/*** xa_for_each() - Iterate over present entries in an XArray.* @xa: XArray.* @index: Index of @entry.* @entry: Entry retrieved from array.** During the iteration, @entry will have the value of the entry stored* in @xa at @index.  You may modify @index during the iteration if you want* to skip or reprocess indices.  It is safe to modify the array during the* iteration.  At the end of the iteration, @entry will be set to NULL and* @index will have a value less than or equal to max.** xa_for_each() is O(n.log(n)) while xas_for_each() is O(n).  You have* to handle your own locking with xas_for_each(), and if you have to unlock* after each iteration, it will also end up being O(n.log(n)).  xa_for_each()* will spin if it hits a retry entry; if you intend to see retry entries,* you should use the xas_for_each() iterator instead.  The xas_for_each()* iterator will expand into more inline code than xa_for_each().** Context: Any context.  Takes and releases the RCU lock.*/
#define xa_for_each(xa, index, entry) \xa_for_each_start(xa, index, entry, 0)/*** xa_for_each_marked() - Iterate over marked entries in an XArray.* @xa: XArray.* @index: Index of @entry.* @entry: Entry retrieved from array.* @filter: Selection criterion.** During the iteration, @entry will have the value of the entry stored* in @xa at @index.  The iteration will skip all entries in the array* which do not match @filter.  You may modify @index during the iteration* if you want to skip or reprocess indices.  It is safe to modify the array* during the iteration.  At the end of the iteration, @entry will be set to* NULL and @index will have a value less than or equal to max.** xa_for_each_marked() is O(n.log(n)) while xas_for_each_marked() is O(n).* You have to handle your own locking with xas_for_each(), and if you have* to unlock after each iteration, it will also end up being O(n.log(n)).* xa_for_each_marked() will spin if it hits a retry entry; if you intend to* see retry entries, you should use the xas_for_each_marked() iterator* instead.  The xas_for_each_marked() iterator will expand into more inline* code than xa_for_each_marked().** Context: Any context.  Takes and releases the RCU lock.*/
#define xa_for_each_marked(xa, index, entry, filter) \for (index = 0, entry = xa_find(xa, &index, ULONG_MAX, filter); \entry; entry = xa_find_after(xa, &index, ULONG_MAX, filter))#define xa_trylock(xa)       spin_trylock(&(xa)->xa_lock)
#define xa_lock(xa)     spin_lock(&(xa)->xa_lock)
#define xa_unlock(xa)       spin_unlock(&(xa)->xa_lock)
#define xa_lock_bh(xa)      spin_lock_bh(&(xa)->xa_lock)
#define xa_unlock_bh(xa)    spin_unlock_bh(&(xa)->xa_lock)
#define xa_lock_irq(xa)     spin_lock_irq(&(xa)->xa_lock)
#define xa_unlock_irq(xa)   spin_unlock_irq(&(xa)->xa_lock)
#define xa_lock_irqsave(xa, flags) \spin_lock_irqsave(&(xa)->xa_lock, flags)
#define xa_unlock_irqrestore(xa, flags) \spin_unlock_irqrestore(&(xa)->xa_lock, flags)
#define xa_lock_nested(xa, subclass) \spin_lock_nested(&(xa)->xa_lock, subclass)
#define xa_lock_bh_nested(xa, subclass) \spin_lock_bh_nested(&(xa)->xa_lock, subclass)
#define xa_lock_irq_nested(xa, subclass) \spin_lock_irq_nested(&(xa)->xa_lock, subclass)
#define xa_lock_irqsave_nested(xa, flags, subclass) \spin_lock_irqsave_nested(&(xa)->xa_lock, flags, subclass)/** Versions of the normal API which require the caller to hold the* xa_lock.  If the GFP flags allow it, they will drop the lock to* allocate memory, then reacquire it afterwards.  These functions* may also re-enable interrupts if the XArray flags indicate the* locking should be interrupt safe.*/
void *__xa_erase(struct xarray *, unsigned long index);
void *__xa_store(struct xarray *, unsigned long index, void *entry, gfp_t);
void *__xa_cmpxchg(struct xarray *, unsigned long index, void *old,void *entry, gfp_t);
int __must_check __xa_insert(struct xarray *, unsigned long index,void *entry, gfp_t);
int __must_check __xa_alloc(struct xarray *, u32 *id, void *entry,struct xa_limit, gfp_t);
int __must_check __xa_alloc_cyclic(struct xarray *, u32 *id, void *entry,struct xa_limit, u32 *next, gfp_t);
void __xa_set_mark(struct xarray *, unsigned long index, xa_mark_t);
void __xa_clear_mark(struct xarray *, unsigned long index, xa_mark_t);/*** xa_store_bh() - Store this entry in the XArray.* @xa: XArray.* @index: Index into array.* @entry: New entry.* @gfp: Memory allocation flags.** This function is like calling xa_store() except it disables softirqs* while holding the array lock.** Context: Any context.  Takes and releases the xa_lock while* disabling softirqs.* Return: The entry which used to be at this index.*/
static inline void *xa_store_bh(struct xarray *xa, unsigned long index,void *entry, gfp_t gfp)
{void *curr;xa_lock_bh(xa);curr = __xa_store(xa, index, entry, gfp);xa_unlock_bh(xa);return curr;
}/*** xa_store_irq() - Store this entry in the XArray.* @xa: XArray.* @index: Index into array.* @entry: New entry.* @gfp: Memory allocation flags.** This function is like calling xa_store() except it disables interrupts* while holding the array lock.** Context: Process context.  Takes and releases the xa_lock while* disabling interrupts.* Return: The entry which used to be at this index.*/
static inline void *xa_store_irq(struct xarray *xa, unsigned long index,void *entry, gfp_t gfp)
{void *curr;xa_lock_irq(xa);curr = __xa_store(xa, index, entry, gfp);xa_unlock_irq(xa);return curr;
}/*** xa_erase_bh() - Erase this entry from the XArray.* @xa: XArray.* @index: Index of entry.** After this function returns, loading from @index will return %NULL.* If the index is part of a multi-index entry, all indices will be erased* and none of the entries will be part of a multi-index entry.** Context: Any context.  Takes and releases the xa_lock while* disabling softirqs.* Return: The entry which used to be at this index.*/
static inline void *xa_erase_bh(struct xarray *xa, unsigned long index)
{void *entry;xa_lock_bh(xa);entry = __xa_erase(xa, index);xa_unlock_bh(xa);return entry;
}/*** xa_erase_irq() - Erase this entry from the XArray.* @xa: XArray.* @index: Index of entry.** After this function returns, loading from @index will return %NULL.* If the index is part of a multi-index entry, all indices will be erased* and none of the entries will be part of a multi-index entry.** Context: Process context.  Takes and releases the xa_lock while* disabling interrupts.* Return: The entry which used to be at this index.*/
static inline void *xa_erase_irq(struct xarray *xa, unsigned long index)
{void *entry;xa_lock_irq(xa);entry = __xa_erase(xa, index);xa_unlock_irq(xa);return entry;
}/*** xa_cmpxchg() - Conditionally replace an entry in the XArray.* @xa: XArray.* @index: Index into array.* @old: Old value to test against.* @entry: New value to place in array.* @gfp: Memory allocation flags.** If the entry at @index is the same as @old, replace it with @entry.* If the return value is equal to @old, then the exchange was successful.** Context: Any context.  Takes and releases the xa_lock.  May sleep* if the @gfp flags permit.* Return: The old value at this index or xa_err() if an error happened.*/
static inline void *xa_cmpxchg(struct xarray *xa, unsigned long index,void *old, void *entry, gfp_t gfp)
{void *curr;xa_lock(xa);curr = __xa_cmpxchg(xa, index, old, entry, gfp);xa_unlock(xa);return curr;
}/*** xa_cmpxchg_bh() - Conditionally replace an entry in the XArray.* @xa: XArray.* @index: Index into array.* @old: Old value to test against.* @entry: New value to place in array.* @gfp: Memory allocation flags.** This function is like calling xa_cmpxchg() except it disables softirqs* while holding the array lock.** Context: Any context.  Takes and releases the xa_lock while* disabling softirqs.  May sleep if the @gfp flags permit.* Return: The old value at this index or xa_err() if an error happened.*/
static inline void *xa_cmpxchg_bh(struct xarray *xa, unsigned long index,void *old, void *entry, gfp_t gfp)
{void *curr;xa_lock_bh(xa);curr = __xa_cmpxchg(xa, index, old, entry, gfp);xa_unlock_bh(xa);return curr;
}/*** xa_cmpxchg_irq() - Conditionally replace an entry in the XArray.* @xa: XArray.* @index: Index into array.* @old: Old value to test against.* @entry: New value to place in array.* @gfp: Memory allocation flags.** This function is like calling xa_cmpxchg() except it disables interrupts* while holding the array lock.** Context: Process context.  Takes and releases the xa_lock while* disabling interrupts.  May sleep if the @gfp flags permit.* Return: The old value at this index or xa_err() if an error happened.*/
static inline void *xa_cmpxchg_irq(struct xarray *xa, unsigned long index,void *old, void *entry, gfp_t gfp)
{void *curr;xa_lock_irq(xa);curr = __xa_cmpxchg(xa, index, old, entry, gfp);xa_unlock_irq(xa);return curr;
}/*** xa_insert() - Store this entry in the XArray unless another entry is*         already present.* @xa: XArray.* @index: Index into array.* @entry: New entry.* @gfp: Memory allocation flags.** Inserting a NULL entry will store a reserved entry (like xa_reserve())* if no entry is present.  Inserting will fail if a reserved entry is* present, even though loading from this index will return NULL.** Context: Any context.  Takes and releases the xa_lock.  May sleep if* the @gfp flags permit.* Return: 0 if the store succeeded.  -EBUSY if another entry was present.* -ENOMEM if memory could not be allocated.*/
static inline int __must_check xa_insert(struct xarray *xa,unsigned long index, void *entry, gfp_t gfp)
{int err;xa_lock(xa);err = __xa_insert(xa, index, entry, gfp);xa_unlock(xa);return err;
}/*** xa_insert_bh() - Store this entry in the XArray unless another entry is*          already present.* @xa: XArray.* @index: Index into array.* @entry: New entry.* @gfp: Memory allocation flags.** Inserting a NULL entry will store a reserved entry (like xa_reserve())* if no entry is present.  Inserting will fail if a reserved entry is* present, even though loading from this index will return NULL.** Context: Any context.  Takes and releases the xa_lock while* disabling softirqs.  May sleep if the @gfp flags permit.* Return: 0 if the store succeeded.  -EBUSY if another entry was present.* -ENOMEM if memory could not be allocated.*/
static inline int __must_check xa_insert_bh(struct xarray *xa,unsigned long index, void *entry, gfp_t gfp)
{int err;xa_lock_bh(xa);err = __xa_insert(xa, index, entry, gfp);xa_unlock_bh(xa);return err;
}/*** xa_insert_irq() - Store this entry in the XArray unless another entry is*         already present.* @xa: XArray.* @index: Index into array.* @entry: New entry.* @gfp: Memory allocation flags.** Inserting a NULL entry will store a reserved entry (like xa_reserve())* if no entry is present.  Inserting will fail if a reserved entry is* present, even though loading from this index will return NULL.** Context: Process context.  Takes and releases the xa_lock while* disabling interrupts.  May sleep if the @gfp flags permit.* Return: 0 if the store succeeded.  -EBUSY if another entry was present.* -ENOMEM if memory could not be allocated.*/
static inline int __must_check xa_insert_irq(struct xarray *xa,unsigned long index, void *entry, gfp_t gfp)
{int err;xa_lock_irq(xa);err = __xa_insert(xa, index, entry, gfp);xa_unlock_irq(xa);return err;
}/*** xa_alloc() - Find somewhere to store this entry in the XArray.* @xa: XArray.* @id: Pointer to ID.* @entry: New entry.* @limit: Range of ID to allocate.* @gfp: Memory allocation flags.** Finds an empty entry in @xa between @limit.min and @limit.max,* stores the index into the @id pointer, then stores the entry at* that index.  A concurrent lookup will not see an uninitialised @id.** Context: Any context.  Takes and releases the xa_lock.  May sleep if* the @gfp flags permit.* Return: 0 on success, -ENOMEM if memory could not be allocated or* -EBUSY if there are no free entries in @limit.*/
static inline __must_check int xa_alloc(struct xarray *xa, u32 *id,void *entry, struct xa_limit limit, gfp_t gfp)
{int err;xa_lock(xa);err = __xa_alloc(xa, id, entry, limit, gfp);xa_unlock(xa);return err;
}/*** xa_alloc_bh() - Find somewhere to store this entry in the XArray.* @xa: XArray.* @id: Pointer to ID.* @entry: New entry.* @limit: Range of ID to allocate.* @gfp: Memory allocation flags.** Finds an empty entry in @xa between @limit.min and @limit.max,* stores the index into the @id pointer, then stores the entry at* that index.  A concurrent lookup will not see an uninitialised @id.** Context: Any context.  Takes and releases the xa_lock while* disabling softirqs.  May sleep if the @gfp flags permit.* Return: 0 on success, -ENOMEM if memory could not be allocated or* -EBUSY if there are no free entries in @limit.*/
static inline int __must_check xa_alloc_bh(struct xarray *xa, u32 *id,void *entry, struct xa_limit limit, gfp_t gfp)
{int err;xa_lock_bh(xa);err = __xa_alloc(xa, id, entry, limit, gfp);xa_unlock_bh(xa);return err;
}/*** xa_alloc_irq() - Find somewhere to store this entry in the XArray.* @xa: XArray.* @id: Pointer to ID.* @entry: New entry.* @limit: Range of ID to allocate.* @gfp: Memory allocation flags.** Finds an empty entry in @xa between @limit.min and @limit.max,* stores the index into the @id pointer, then stores the entry at* that index.  A concurrent lookup will not see an uninitialised @id.** Context: Process context.  Takes and releases the xa_lock while* disabling interrupts.  May sleep if the @gfp flags permit.* Return: 0 on success, -ENOMEM if memory could not be allocated or* -EBUSY if there are no free entries in @limit.*/
static inline int __must_check xa_alloc_irq(struct xarray *xa, u32 *id,void *entry, struct xa_limit limit, gfp_t gfp)
{int err;xa_lock_irq(xa);err = __xa_alloc(xa, id, entry, limit, gfp);xa_unlock_irq(xa);return err;
}/*** xa_alloc_cyclic() - Find somewhere to store this entry in the XArray.* @xa: XArray.* @id: Pointer to ID.* @entry: New entry.* @limit: Range of allocated ID.* @next: Pointer to next ID to allocate.* @gfp: Memory allocation flags.** Finds an empty entry in @xa between @limit.min and @limit.max,* stores the index into the @id pointer, then stores the entry at* that index.  A concurrent lookup will not see an uninitialised @id.* The search for an empty entry will start at @next and will wrap* around if necessary.** Context: Any context.  Takes and releases the xa_lock.  May sleep if* the @gfp flags permit.* Return: 0 if the allocation succeeded without wrapping.  1 if the* allocation succeeded after wrapping, -ENOMEM if memory could not be* allocated or -EBUSY if there are no free entries in @limit.*/
static inline int xa_alloc_cyclic(struct xarray *xa, u32 *id, void *entry,struct xa_limit limit, u32 *next, gfp_t gfp)
{int err;xa_lock(xa);err = __xa_alloc_cyclic(xa, id, entry, limit, next, gfp);xa_unlock(xa);return err;
}/*** xa_alloc_cyclic_bh() - Find somewhere to store this entry in the XArray.* @xa: XArray.* @id: Pointer to ID.* @entry: New entry.* @limit: Range of allocated ID.* @next: Pointer to next ID to allocate.* @gfp: Memory allocation flags.** Finds an empty entry in @xa between @limit.min and @limit.max,* stores the index into the @id pointer, then stores the entry at* that index.  A concurrent lookup will not see an uninitialised @id.* The search for an empty entry will start at @next and will wrap* around if necessary.** Context: Any context.  Takes and releases the xa_lock while* disabling softirqs.  May sleep if the @gfp flags permit.* Return: 0 if the allocation succeeded without wrapping.  1 if the* allocation succeeded after wrapping, -ENOMEM if memory could not be* allocated or -EBUSY if there are no free entries in @limit.*/
static inline int xa_alloc_cyclic_bh(struct xarray *xa, u32 *id, void *entry,struct xa_limit limit, u32 *next, gfp_t gfp)
{int err;xa_lock_bh(xa);err = __xa_alloc_cyclic(xa, id, entry, limit, next, gfp);xa_unlock_bh(xa);return err;
}/*** xa_alloc_cyclic_irq() - Find somewhere to store this entry in the XArray.* @xa: XArray.* @id: Pointer to ID.* @entry: New entry.* @limit: Range of allocated ID.* @next: Pointer to next ID to allocate.* @gfp: Memory allocation flags.** Finds an empty entry in @xa between @limit.min and @limit.max,* stores the index into the @id pointer, then stores the entry at* that index.  A concurrent lookup will not see an uninitialised @id.* The search for an empty entry will start at @next and will wrap* around if necessary.** Context: Process context.  Takes and releases the xa_lock while* disabling interrupts.  May sleep if the @gfp flags permit.* Return: 0 if the allocation succeeded without wrapping.  1 if the* allocation succeeded after wrapping, -ENOMEM if memory could not be* allocated or -EBUSY if there are no free entries in @limit.*/
static inline int xa_alloc_cyclic_irq(struct xarray *xa, u32 *id, void *entry,struct xa_limit limit, u32 *next, gfp_t gfp)
{int err;xa_lock_irq(xa);err = __xa_alloc_cyclic(xa, id, entry, limit, next, gfp);xa_unlock_irq(xa);return err;
}/*** xa_reserve() - Reserve this index in the XArray.* @xa: XArray.* @index: Index into array.* @gfp: Memory allocation flags.** Ensures there is somewhere to store an entry at @index in the array.* If there is already something stored at @index, this function does* nothing.  If there was nothing there, the entry is marked as reserved.* Loading from a reserved entry returns a %NULL pointer.** If you do not use the entry that you have reserved, call xa_release()* or xa_erase() to free any unnecessary memory.** Context: Any context.  Takes and releases the xa_lock.* May sleep if the @gfp flags permit.* Return: 0 if the reservation succeeded or -ENOMEM if it failed.*/
static inline __must_check
int xa_reserve(struct xarray *xa, unsigned long index, gfp_t gfp)
{return xa_err(xa_cmpxchg(xa, index, NULL, XA_ZERO_ENTRY, gfp));
}/*** xa_reserve_bh() - Reserve this index in the XArray.* @xa: XArray.* @index: Index into array.* @gfp: Memory allocation flags.** A softirq-disabling version of xa_reserve().** Context: Any context.  Takes and releases the xa_lock while* disabling softirqs.* Return: 0 if the reservation succeeded or -ENOMEM if it failed.*/
static inline __must_check
int xa_reserve_bh(struct xarray *xa, unsigned long index, gfp_t gfp)
{return xa_err(xa_cmpxchg_bh(xa, index, NULL, XA_ZERO_ENTRY, gfp));
}/*** xa_reserve_irq() - Reserve this index in the XArray.* @xa: XArray.* @index: Index into array.* @gfp: Memory allocation flags.** An interrupt-disabling version of xa_reserve().** Context: Process context.  Takes and releases the xa_lock while* disabling interrupts.* Return: 0 if the reservation succeeded or -ENOMEM if it failed.*/
static inline __must_check
int xa_reserve_irq(struct xarray *xa, unsigned long index, gfp_t gfp)
{return xa_err(xa_cmpxchg_irq(xa, index, NULL, XA_ZERO_ENTRY, gfp));
}/*** xa_release() - Release a reserved entry.* @xa: XArray.* @index: Index of entry.** After calling xa_reserve(), you can call this function to release the* reservation.  If the entry at @index has been stored to, this function* will do nothing.*/
static inline void xa_release(struct xarray *xa, unsigned long index)
{xa_cmpxchg(xa, index, XA_ZERO_ENTRY, NULL, 0);
}/* Everything below here is the Advanced API.  Proceed with caution. *//** The xarray is constructed out of a set of 'chunks' of pointers.  Choosing* the best chunk size requires some tradeoffs.  A power of two recommends* itself so that we can walk the tree based purely on shifts and masks.* Generally, the larger the better; as the number of slots per level of the* tree increases, the less tall the tree needs to be.  But that needs to be* balanced against the memory consumption of each node.  On a 64-bit system,* xa_node is currently 576 bytes, and we get 7 of them per 4kB page.  If we* doubled the number of slots per node, we'd get only 3 nodes per 4kB page.*/
#ifndef XA_CHUNK_SHIFT
#define XA_CHUNK_SHIFT      (CONFIG_BASE_SMALL ? 4 : 6)
#endif
#define XA_CHUNK_SIZE       (1UL << XA_CHUNK_SHIFT)
#define XA_CHUNK_MASK       (XA_CHUNK_SIZE - 1)
#define XA_MAX_MARKS        3
#define XA_MARK_LONGS       DIV_ROUND_UP(XA_CHUNK_SIZE, BITS_PER_LONG)/** @count is the count of every non-NULL element in the ->slots array* whether that is a value entry, a retry entry, a user pointer,* a sibling entry or a pointer to the next level of the tree.* @nr_values is the count of every element in ->slots which is* either a value entry or a sibling of a value entry.*/
struct xa_node {unsigned char   shift;      /* Bits remaining in each slot */unsigned char  offset;     /* Slot offset in parent */unsigned char    count;      /* Total entry count */unsigned char    nr_values;  /* Value entry count */struct xa_node __rcu *parent;    /* NULL at top of tree */struct xarray  *array;     /* The array we belong to */union {struct list_head private_list;   /* For tree user */struct rcu_head  rcu_head;   /* Used when freeing node */};void __rcu    *slots[XA_CHUNK_SIZE];union {unsigned long  tags[XA_MAX_MARKS][XA_MARK_LONGS];unsigned long marks[XA_MAX_MARKS][XA_MARK_LONGS];};
};void xa_dump(const struct xarray *);
void xa_dump_node(const struct xa_node *);#ifdef XA_DEBUG
#define XA_BUG_ON(xa, x) do {                   \if (x) {                   \xa_dump(xa);               \BUG();                 \}                      \} while (0)
#define XA_NODE_BUG_ON(node, x) do {                \if (x) {                   \if (node) xa_dump_node(node);      \BUG();                 \}                      \} while (0)
#else
#define XA_BUG_ON(xa, x)    do { } while (0)
#define XA_NODE_BUG_ON(node, x) do { } while (0)
#endif/* Private */
static inline void *xa_head(const struct xarray *xa)
{return rcu_dereference_check(xa->xa_head,lockdep_is_held(&xa->xa_lock));
}/* Private */
static inline void *xa_head_locked(const struct xarray *xa)
{return rcu_dereference_protected(xa->xa_head,lockdep_is_held(&xa->xa_lock));
}/* Private */
static inline void *xa_entry(const struct xarray *xa,const struct xa_node *node, unsigned int offset)
{XA_NODE_BUG_ON(node, offset >= XA_CHUNK_SIZE);return rcu_dereference_check(node->slots[offset],lockdep_is_held(&xa->xa_lock));
}/* Private */
static inline void *xa_entry_locked(const struct xarray *xa,const struct xa_node *node, unsigned int offset)
{XA_NODE_BUG_ON(node, offset >= XA_CHUNK_SIZE);return rcu_dereference_protected(node->slots[offset],lockdep_is_held(&xa->xa_lock));
}/* Private */
static inline struct xa_node *xa_parent(const struct xarray *xa,const struct xa_node *node)
{return rcu_dereference_check(node->parent,lockdep_is_held(&xa->xa_lock));
}/* Private */
static inline struct xa_node *xa_parent_locked(const struct xarray *xa,const struct xa_node *node)
{return rcu_dereference_protected(node->parent,lockdep_is_held(&xa->xa_lock));
}/* Private */
static inline void *xa_mk_node(const struct xa_node *node)
{return (void *)((unsigned long)node | 2);
}/* Private */
static inline struct xa_node *xa_to_node(const void *entry)
{return (struct xa_node *)((unsigned long)entry - 2);
}/* Private */
static inline bool xa_is_node(const void *entry)
{return xa_is_internal(entry) && (unsigned long)entry > 4096;
}/* Private */
static inline void *xa_mk_sibling(unsigned int offset)
{return xa_mk_internal(offset);
}/* Private */
static inline unsigned long xa_to_sibling(const void *entry)
{return xa_to_internal(entry);
}/*** xa_is_sibling() - Is the entry a sibling entry?* @entry: Entry retrieved from the XArray** Return: %true if the entry is a sibling entry.*/
static inline bool xa_is_sibling(const void *entry)
{return IS_ENABLED(CONFIG_XARRAY_MULTI) && xa_is_internal(entry) &&(entry < xa_mk_sibling(XA_CHUNK_SIZE - 1));
}#define XA_RETRY_ENTRY     xa_mk_internal(256)/*** xa_is_retry() - Is the entry a retry entry?* @entry: Entry retrieved from the XArray** Return: %true if the entry is a retry entry.*/
static inline bool xa_is_retry(const void *entry)
{return unlikely(entry == XA_RETRY_ENTRY);
}/*** xa_is_advanced() - Is the entry only permitted for the advanced API?* @entry: Entry to be stored in the XArray.** Return: %true if the entry cannot be stored by the normal API.*/
static inline bool xa_is_advanced(const void *entry)
{return xa_is_internal(entry) && (entry <= XA_RETRY_ENTRY);
}/*** typedef xa_update_node_t - A callback function from the XArray.* @node: The node which is being processed** This function is called every time the XArray updates the count of* present and value entries in a node.  It allows advanced users to* maintain the private_list in the node.** Context: The xa_lock is held and interrupts may be disabled.*     Implementations should not drop the xa_lock, nor re-enable*     interrupts.*/
typedef void (*xa_update_node_t)(struct xa_node *node);/** The xa_state is opaque to its users.  It contains various different pieces* of state involved in the current operation on the XArray.  It should be* declared on the stack and passed between the various internal routines.* The various elements in it should not be accessed directly, but only* through the provided accessor functions.  The below documentation is for* the benefit of those working on the code, not for users of the XArray.** @xa_node usually points to the xa_node containing the slot we're operating* on (and @xa_offset is the offset in the slots array).  If there is a* single entry in the array at index 0, there are no allocated xa_nodes to* point to, and so we store %NULL in @xa_node.  @xa_node is set to* the value %XAS_RESTART if the xa_state is not walked to the correct* position in the tree of nodes for this operation.  If an error occurs* during an operation, it is set to an %XAS_ERROR value.  If we run off the* end of the allocated nodes, it is set to %XAS_BOUNDS.*/
struct xa_state {struct xarray *xa;unsigned long xa_index;unsigned char xa_shift;unsigned char xa_sibs;unsigned char xa_offset;unsigned char xa_pad;        /* Helps gcc generate better code */struct xa_node *xa_node;struct xa_node *xa_alloc;xa_update_node_t xa_update;
};/** We encode errnos in the xas->xa_node.  If an error has happened, we need to* drop the lock to fix it, and once we've done so the xa_state is invalid.*/
#define XA_ERROR(errno) ((struct xa_node *)(((unsigned long)errno << 2) | 2UL))
#define XAS_BOUNDS  ((struct xa_node *)1UL)
#define XAS_RESTART ((struct xa_node *)3UL)#define __XA_STATE(array, index, shift, sibs)  { \.xa = array,                   \.xa_index = index,             \.xa_shift = shift,             \.xa_sibs = sibs,               \.xa_offset = 0,                    \.xa_pad = 0,                   \.xa_node = XAS_RESTART,                \.xa_alloc = NULL,              \.xa_update = NULL              \
}/*** XA_STATE() - Declare an XArray operation state.* @name: Name of this operation state (usually xas).* @array: Array to operate on.* @index: Initial index of interest.** Declare and initialise an xa_state on the stack.*/
#define XA_STATE(name, array, index)                \struct xa_state name = __XA_STATE(array, index, 0, 0)/*** XA_STATE_ORDER() - Declare an XArray operation state.* @name: Name of this operation state (usually xas).* @array: Array to operate on.* @index: Initial index of interest.* @order: Order of entry.** Declare and initialise an xa_state on the stack.  This variant of* XA_STATE() allows you to specify the 'order' of the element you* want to operate on.`*/
#define XA_STATE_ORDER(name, array, index, order)       \struct xa_state name = __XA_STATE(array,       \(index >> order) << order,     \order - (order % XA_CHUNK_SHIFT),  \(1U << (order % XA_CHUNK_SHIFT)) - 1)#define xas_marked(xas, mark)   xa_marked((xas)->xa, (mark))
#define xas_trylock(xas)    xa_trylock((xas)->xa)
#define xas_lock(xas)       xa_lock((xas)->xa)
#define xas_unlock(xas)     xa_unlock((xas)->xa)
#define xas_lock_bh(xas)    xa_lock_bh((xas)->xa)
#define xas_unlock_bh(xas)  xa_unlock_bh((xas)->xa)
#define xas_lock_irq(xas)   xa_lock_irq((xas)->xa)
#define xas_unlock_irq(xas) xa_unlock_irq((xas)->xa)
#define xas_lock_irqsave(xas, flags) \xa_lock_irqsave((xas)->xa, flags)
#define xas_unlock_irqrestore(xas, flags) \xa_unlock_irqrestore((xas)->xa, flags)/*** xas_error() - Return an errno stored in the xa_state.* @xas: XArray operation state.** Return: 0 if no error has been noted.  A negative errno if one has.*/
static inline int xas_error(const struct xa_state *xas)
{return xa_err(xas->xa_node);
}/*** xas_set_err() - Note an error in the xa_state.* @xas: XArray operation state.* @err: Negative error number.** Only call this function with a negative @err; zero or positive errors* will probably not behave the way you think they should.  If you want* to clear the error from an xa_state, use xas_reset().*/
static inline void xas_set_err(struct xa_state *xas, long err)
{xas->xa_node = XA_ERROR(err);
}/*** xas_invalid() - Is the xas in a retry or error state?* @xas: XArray operation state.** Return: %true if the xas cannot be used for operations.*/
static inline bool xas_invalid(const struct xa_state *xas)
{return (unsigned long)xas->xa_node & 3;
}/*** xas_valid() - Is the xas a valid cursor into the array?* @xas: XArray operation state.** Return: %true if the xas can be used for operations.*/
static inline bool xas_valid(const struct xa_state *xas)
{return !xas_invalid(xas);
}/*** xas_is_node() - Does the xas point to a node?* @xas: XArray operation state.** Return: %true if the xas currently references a node.*/
static inline bool xas_is_node(const struct xa_state *xas)
{return xas_valid(xas) && xas->xa_node;
}/* True if the pointer is something other than a node */
static inline bool xas_not_node(struct xa_node *node)
{return ((unsigned long)node & 3) || !node;
}/* True if the node represents RESTART or an error */
static inline bool xas_frozen(struct xa_node *node)
{return (unsigned long)node & 2;
}/* True if the node represents head-of-tree, RESTART or BOUNDS */
static inline bool xas_top(struct xa_node *node)
{return node <= XAS_RESTART;
}/*** xas_reset() - Reset an XArray operation state.* @xas: XArray operation state.** Resets the error or walk state of the @xas so future walks of the* array will start from the root.  Use this if you have dropped the* xarray lock and want to reuse the xa_state.** Context: Any context.*/
static inline void xas_reset(struct xa_state *xas)
{xas->xa_node = XAS_RESTART;
}/*** xas_retry() - Retry the operation if appropriate.* @xas: XArray operation state.* @entry: Entry from xarray.** The advanced functions may sometimes return an internal entry, such as* a retry entry or a zero entry.  This function sets up the @xas to restart* the walk from the head of the array if needed.** Context: Any context.* Return: true if the operation needs to be retried.*/
static inline bool xas_retry(struct xa_state *xas, const void *entry)
{if (xa_is_zero(entry))return true;if (!xa_is_retry(entry))return false;xas_reset(xas);return true;
}void *xas_load(struct xa_state *);
void *xas_store(struct xa_state *, void *entry);
void *xas_find(struct xa_state *, unsigned long max);
void *xas_find_conflict(struct xa_state *);bool xas_get_mark(const struct xa_state *, xa_mark_t);
void xas_set_mark(const struct xa_state *, xa_mark_t);
void xas_clear_mark(const struct xa_state *, xa_mark_t);
void *xas_find_marked(struct xa_state *, unsigned long max, xa_mark_t);
void xas_init_marks(const struct xa_state *);bool xas_nomem(struct xa_state *, gfp_t);
void xas_pause(struct xa_state *);void xas_create_range(struct xa_state *);/*** xas_reload() - Refetch an entry from the xarray.* @xas: XArray operation state.** Use this function to check that a previously loaded entry still has* the same value.  This is useful for the lockless pagecache lookup where* we walk the array with only the RCU lock to protect us, lock the page,* then check that the page hasn't moved since we looked it up.** The caller guarantees that @xas is still valid.  If it may be in an* error or restart state, call xas_load() instead.** Return: The entry at this location in the xarray.*/
static inline void *xas_reload(struct xa_state *xas)
{struct xa_node *node = xas->xa_node;if (node)return xa_entry(xas->xa, node, xas->xa_offset);return xa_head(xas->xa);
}/*** xas_set() - Set up XArray operation state for a different index.* @xas: XArray operation state.* @index: New index into the XArray.** Move the operation state to refer to a different index.  This will* have the effect of starting a walk from the top; see xas_next()* to move to an adjacent index.*/
static inline void xas_set(struct xa_state *xas, unsigned long index)
{xas->xa_index = index;xas->xa_node = XAS_RESTART;
}/*** xas_set_order() - Set up XArray operation state for a multislot entry.* @xas: XArray operation state.* @index: Target of the operation.* @order: Entry occupies 2^@order indices.*/
static inline void xas_set_order(struct xa_state *xas, unsigned long index,unsigned int order)
{
#ifdef CONFIG_XARRAY_MULTIxas->xa_index = order < BITS_PER_LONG ? (index >> order) << order : 0;xas->xa_shift = order - (order % XA_CHUNK_SHIFT);xas->xa_sibs = (1 << (order % XA_CHUNK_SHIFT)) - 1;xas->xa_node = XAS_RESTART;
#elseBUG_ON(order > 0);xas_set(xas, index);
#endif
}/*** xas_set_update() - Set up XArray operation state for a callback.* @xas: XArray operation state.* @update: Function to call when updating a node.** The XArray can notify a caller after it has updated an xa_node.* This is advanced functionality and is only needed by the page cache.*/
static inline void xas_set_update(struct xa_state *xas, xa_update_node_t update)
{xas->xa_update = update;
}/*** xas_next_entry() - Advance iterator to next present entry.* @xas: XArray operation state.* @max: Highest index to return.** xas_next_entry() is an inline function to optimise xarray traversal for* speed.  It is equivalent to calling xas_find(), and will call xas_find()* for all the hard cases.** Return: The next present entry after the one currently referred to by @xas.*/
static inline void *xas_next_entry(struct xa_state *xas, unsigned long max)
{struct xa_node *node = xas->xa_node;void *entry;if (unlikely(xas_not_node(node) || node->shift ||xas->xa_offset != (xas->xa_index & XA_CHUNK_MASK)))return xas_find(xas, max);do {if (unlikely(xas->xa_index >= max))return xas_find(xas, max);if (unlikely(xas->xa_offset == XA_CHUNK_MASK))return xas_find(xas, max);entry = xa_entry(xas->xa, node, xas->xa_offset + 1);if (unlikely(xa_is_internal(entry)))return xas_find(xas, max);xas->xa_offset++;xas->xa_index++;} while (!entry);return entry;
}/* Private */
static inline unsigned int xas_find_chunk(struct xa_state *xas, bool advance,xa_mark_t mark)
{unsigned long *addr = xas->xa_node->marks[(__force unsigned)mark];unsigned int offset = xas->xa_offset;if (advance)offset++;if (XA_CHUNK_SIZE == BITS_PER_LONG) {if (offset < XA_CHUNK_SIZE) {unsigned long data = *addr & (~0UL << offset);if (data)return __ffs(data);}return XA_CHUNK_SIZE;}return find_next_bit(addr, XA_CHUNK_SIZE, offset);
}/*** xas_next_marked() - Advance iterator to next marked entry.* @xas: XArray operation state.* @max: Highest index to return.* @mark: Mark to search for.** xas_next_marked() is an inline function to optimise xarray traversal for* speed.  It is equivalent to calling xas_find_marked(), and will call* xas_find_marked() for all the hard cases.** Return: The next marked entry after the one currently referred to by @xas.*/
static inline void *xas_next_marked(struct xa_state *xas, unsigned long max,xa_mark_t mark)
{struct xa_node *node = xas->xa_node;void *entry;unsigned int offset;if (unlikely(xas_not_node(node) || node->shift))return xas_find_marked(xas, max, mark);offset = xas_find_chunk(xas, true, mark);xas->xa_offset = offset;xas->xa_index = (xas->xa_index & ~XA_CHUNK_MASK) + offset;if (xas->xa_index > max)return NULL;if (offset == XA_CHUNK_SIZE)return xas_find_marked(xas, max, mark);entry = xa_entry(xas->xa, node, offset);if (!entry)return xas_find_marked(xas, max, mark);return entry;
}/** If iterating while holding a lock, drop the lock and reschedule* every %XA_CHECK_SCHED loops.*/
enum {XA_CHECK_SCHED = 4096,
};/*** xas_for_each() - Iterate over a range of an XArray.* @xas: XArray operation state.* @entry: Entry retrieved from the array.* @max: Maximum index to retrieve from array.** The loop body will be executed for each entry present in the xarray* between the current xas position and @max.  @entry will be set to* the entry retrieved from the xarray.  It is safe to delete entries* from the array in the loop body.  You should hold either the RCU lock* or the xa_lock while iterating.  If you need to drop the lock, call* xas_pause() first.*/
#define xas_for_each(xas, entry, max) \for (entry = xas_find(xas, max); entry; \entry = xas_next_entry(xas, max))/*** xas_for_each_marked() - Iterate over a range of an XArray.* @xas: XArray operation state.* @entry: Entry retrieved from the array.* @max: Maximum index to retrieve from array.* @mark: Mark to search for.** The loop body will be executed for each marked entry in the xarray* between the current xas position and @max.  @entry will be set to* the entry retrieved from the xarray.  It is safe to delete entries* from the array in the loop body.  You should hold either the RCU lock* or the xa_lock while iterating.  If you need to drop the lock, call* xas_pause() first.*/
#define xas_for_each_marked(xas, entry, max, mark) \for (entry = xas_find_marked(xas, max, mark); entry; \entry = xas_next_marked(xas, max, mark))/*** xas_for_each_conflict() - Iterate over a range of an XArray.* @xas: XArray operation state.* @entry: Entry retrieved from the array.** The loop body will be executed for each entry in the XArray that lies* within the range specified by @xas.  If the loop completes successfully,* any entries that lie in this range will be replaced by @entry.  The caller* may break out of the loop; if they do so, the contents of the XArray will* be unchanged.  The operation may fail due to an out of memory condition.* The caller may also call xa_set_err() to exit the loop while setting an* error to record the reason.*/
#define xas_for_each_conflict(xas, entry) \while ((entry = xas_find_conflict(xas)))void *__xas_next(struct xa_state *);
void *__xas_prev(struct xa_state *);/*** xas_prev() - Move iterator to previous index.* @xas: XArray operation state.** If the @xas was in an error state, it will remain in an error state* and this function will return %NULL.  If the @xas has never been walked,* it will have the effect of calling xas_load().  Otherwise one will be* subtracted from the index and the state will be walked to the correct* location in the array for the next operation.** If the iterator was referencing index 0, this function wraps* around to %ULONG_MAX.** Return: The entry at the new index.  This may be %NULL or an internal* entry.*/
static inline void *xas_prev(struct xa_state *xas)
{struct xa_node *node = xas->xa_node;if (unlikely(xas_not_node(node) || node->shift ||xas->xa_offset == 0))return __xas_prev(xas);xas->xa_index--;xas->xa_offset--;return xa_entry(xas->xa, node, xas->xa_offset);
}/*** xas_next() - Move state to next index.* @xas: XArray operation state.** If the @xas was in an error state, it will remain in an error state* and this function will return %NULL.  If the @xas has never been walked,* it will have the effect of calling xas_load().  Otherwise one will be* added to the index and the state will be walked to the correct* location in the array for the next operation.** If the iterator was referencing index %ULONG_MAX, this function wraps* around to 0.** Return: The entry at the new index.  This may be %NULL or an internal* entry.*/
static inline void *xas_next(struct xa_state *xas)
{struct xa_node *node = xas->xa_node;if (unlikely(xas_not_node(node) || node->shift ||xas->xa_offset == XA_CHUNK_MASK))return __xas_next(xas);xas->xa_index++;xas->xa_offset++;return xa_entry(xas->xa, node, xas->xa_offset);
}#endif /* _LINUX_XARRAY_H */