cgroups的实现

1. cgroups体系结构

每个内核的子系统如果想要挂载到cgroup系统中，必须要先拥有一个cgroup_subsys对象，通过这个对象将对子系统资源的操作函数以接口的形式进行约定，各个子系统再根据自己的需求去实现这些接口，除了这些函数指针，cgroup_subsys结构体还包括id, name, early_init等属性，分别表示该子系统在cgroup->subsys[]数组中的索引，子系统名称，在系统启动时是否需要尽早初始化等。

struct cgroup_subsys{struct cgroup_subsys_state* (*css_alloc)(struct cgroup_subsys_state *parent_css);void (*attach)(struct cgroup_taskset *tset);...bool early_init: 1;ind id;const char *name;struct cgroup_root *root;
}

除了这些，cgroup_subsys还有一个指向cgroup_root类型的root指针，cgroup_root表示一个cgroup hierarchy的根，注意与根结点相区别，所以说一个子系统只能通过cgroup_root结构与一个cgroup hierarchy绑定（但是反过来一个cgroup hierarchy可以绑定多个子系统 ）。关于cgroup_root下面具体讨论。

再看第一个函数指针css_alloc，其返回了一个cgroup_subsys_state结构体，每一个注册到系统中的cgroup子系统，除了代表它的cgroup_subsys，还有与之相关联的cgroup_subsys_state，简称css，用来表示某个子系统与某个cgroup相关联的状态，将一个子系统与一个cgroup相连接。每个subsys会对调用css_alloc的cgroup分配一个自己的cgroup_subsys_state结构。

struct cgroup_subsys_state {/* PI: the cgroup that this css is attached to */struct cgroup *cgroup;/* PI: the cgroup subsystem that this css is attached to */struct cgroup_subsys *ss;/* reference count - access via css_[try]get() and css_put() */struct percpu_ref refcnt;/* siblings list anchored at the parent's ->children */struct list_head sibling;struct list_head children;/** PI: Subsys-unique ID.  0 is unused and root is always 1.  The* matching css can be looked up using css_from_id().*/int id;unsigned int flags;...struct cgroup_subsys_state *parent;
};

cgroup指向此css所关联的cgroup
ss指向此css所关联的cgroup_subsys
sibling是一个双向链表，保存所关联的cgroup拥有的其他css
parent指向父css的指针

提到css，那就不得不提css_set，css_set是保存指向cgroup_subsys_state对象的一组引用计数指针的集合，用来保存与task相关的cgroup信息，同时也代表了一组子系统资源的组合，css_set的使用节省了cgroups在task_struct结构体中所占用的空间，而且加快了task fork()/exit()的速度。

struct css_set {/** Set of subsystem states, one for each subsystem. This array is* immutable after creation apart from the init_css_set during* subsystem registration (at boot time).*/struct cgroup_subsys_state *subsys[CGROUP_SUBSYS_COUNT];/* reference count */refcount_t refcount;.../** Lists running through all tasks using this cgroup group.*/struct list_head tasks;/** List running through all cgroup groups in the same hash* slot. Protected by css_set_lock*/struct hlist_node hlist;/** List of cgrp_cset_links pointing at cgroups referenced from this* css_set.  Protected by css_set_lock.*/struct list_head cgrp_links;/* dead and being drained, ignore for migration */bool dead;/* For RCU-protected deletion */struct rcu_head rcu_head;
};

subsys保存子系统状态的集合，初始化后不可修改
hlist用来链接同一个hashtable slots中的所有css_set
refcount用来引用计数
task用来链接所有使用此css_set的task_struct集合

从task到cgroup的关联

从task到其所属的cgroup之间是没有直接指针相连接的，但是task可以通过一个媒介来获取其所属的cgroup，这个媒介就是css_set和cgroup_subsys_state。通过task_struct -> cgroups -> subsys[ssid] ->cgroup即可访问到管理对应子系统的cgroup。之所以这么设计时因为获取子系统状态的操作预计会频繁发生，而且是在性能关键代码中，然而需要一个task实际的cgroup来执行的操作（尤其是task在cgroups之间迁移的操作）则并没有那么常见。task_struct中的cg_list则是用来连接使用同一个css_set的task的链表，css_set通过tasks来遍历访问此链表。

css_set与cgroup之间的关联

一个进程属于一个css_set, 一个css_set就存储了一组进程跟各个子系统相关的信息，但是这些信息有可能不是从一个cgroup那里获得的，因为一个进程可以同时属于几个cgroup，只要这些cgroup不在同一个层级。所以一个css_set存储的cgroup_subsys_state可以对应多个cgroup。

另一方面，cgroup也存储了一组cgroup_subsys_state，这一组cgroup_subsys_state则是cgroup从所在的层级附加的子系统获得的。一个cgroup中可以有多个进程，而这些进程的css_set不一定都相同，因为有些进程可能还加入了其他cgroup。但是同一个cgroup中的进程与该cgroup关联的cgroup_subsys_state都受到该cgroup的管理(cgroups中进程控制是以cgroup为单位的)的，所以一个cgroup也可以对应多个css_set。

因此它们之间的关系是 M ∗ N M*N M∗N , 这种关系的表达则是通过cgrp_cset_link来实现

struct cgrp_cset_link {/* the cgroup and css_set this link associates */struct cgroup        *cgrp;struct css_set        *cset;/* list of cgrp_cset_links anchored at cgrp->cset_links */struct list_head cset_link;/* list of cgrp_cset_links anchored at css_set->cgrp_links */struct list_head  cgrp_link;
};

在回到我们一开始所讲的cgroup_subsys结构，其拥有的一个属性名为root，指向一个cgroup_root结构体

struct cgroup_root {struct kernfs_root *kf_root;/* The bitmask of subsystems attached to this hierarchy */unsigned int subsys_mask;/* Unique id for this hierarchy. */int hierarchy_id;/* The root cgroup. Root is destroyed on its release. */struct cgroup cgrp;/* for cgrp->ancestor_ids[0] */u64 cgrp_ancestor_id_storage;/* Number of cgroups in the hierarchy, used only for /proc/cgroups */atomic_t nr_cgrps;/* A list running through the active hierarchies */struct list_head root_list;/* Hierarchy-specific flags */unsigned int flags;/* The path to use for release notifications. */char release_agent_path[PATH_MAX];/* The name for this hierarchy - may be empty */char name[MAX_CGROUP_ROOT_NAMELEN];
};

一个cgroup_root是一个层级的根，是cgroup的核心，且不由controller直接操作。

kf_root：
subsys_mask：与该层级相关联的子系统的点位图
hierarchy_id：层级的ID
cgrp：层级的根节点
root_list：包含所有层级的链表

接下来引入主角cgroup

struct cgroup {/* self css with NULL ->ss, points back to this cgroup */struct cgroup_subsys_state self;unsigned long flags;      /* "unsigned long" so bitops work *//** The depth this cgroup is at.  The root is at depth zero and each* step down the hierarchy increments the level.  This along with* ancestor_ids[] can determine whether a given cgroup is a* descendant of another without traversing the hierarchy.*/int level;/* Maximum allowed descent tree depth */int max_depth;...struct kernfs_node *kn;       /* cgroup kernfs entry */struct cgroup_file procs_file; /* handle for "cgroup.procs" */struct cgroup_file events_file;    /* handle for "cgroup.events" *//* Private pointers for each registered subsystem */struct cgroup_subsys_state __rcu *subsys[CGROUP_SUBSYS_COUNT];struct cgroup_root *root;/** List of cgrp_cset_links pointing at css_sets with tasks in this* cgroup.  Protected by css_set_lock.*/struct list_head cset_links;/* per-cpu recursive resource statistics */struct cgroup_rstat_cpu __percpu *rstat_cpu;struct list_head rstat_css_list;/* cgroup basic resource statistics */struct cgroup_base_stat last_bstat;struct cgroup_base_stat bstat;struct prev_cputime prev_cputime;  /* for printing out cputime *//** list of pidlists, up to two for each namespace (one for procs, one* for tasks); created on demand.*/struct list_head pidlists;struct mutex pidlist_mutex;/* used to wait for offlining of csses */wait_queue_head_t offline_waitq;/* used to schedule release agent */struct work_struct release_agent_work;/* If there is block congestion on this cgroup. */atomic_t congestion_count;/* Used to store internal freezer state */struct cgroup_freezer_state freezer;/* ids of the ancestors at each level including self */u64 ancestor_ids[];
};

sibling, children, parent三个list_head负责将同一层级的cgroup连接为一颗cgroup树
subsys存储一组指向cgroup_subsys_state的指针
root指向层级所对应的cgroup_root结构体
cset_links指向cgrp_cset_link连成的链表，负责与css_set相连接

2. cgroup文件系统

cgroup v1,v2的文件系统类型的数据结构

struct file_system_type cgroup_fs_type = {.name = "cgroup",.mount = cgroup_mount,.kill_sb = cgroup_kill_sb,.fs_flags = FS_USERNS_MOUNT,
};static struct file_system_type cgroup2_fs_type = {.name = "cgroup2",.mount = cgroup_mount,.kill_sb = cgroup_kill_sb,.fs_flags = FS_USERNS_MOUNT,
};

cgroup文件系统其实是通过kernfs来实现的，通过文件接口的形式将内核中cgroup信息以及子系统信息传递给用户态，用户照样通过调用vfs接口来进行一般的读写操作。

以cgroup创建为例，内核创建一个cgroup首先调用cgroup_mkdir，cgroup_mkdir调用cgroup_create和css_populate_dir，cgroup_create调用kernfs_create_dir在cgroup文件系统中创建cgroup对应的目录，css_populate_dir又调用cgroup_addrm_dir在对应cgroup目录中填充对应子系统控制文件。

那么是如果通过向控制文件中写入或删除字符（串）来实现资源控制呢？答案就在struct cftype中，cftype在源码中被定义为handler for definitions for cgroup control files。每个子系统通过实现自己的cftype数组（每个控制文件都对应一个cftype结构体，所以每个cgroup_subsys中都保存了一个cftype数组）来实现各自的资源分配功能。

其具体定义如下：

struct cftype {/** By convention, the name should begin with the name of the* subsystem, followed by a period.  Zero length string indicates* end of cftype array.*/char name[MAX_CFTYPE_NAME];unsigned long private;/** The maximum length of string, excluding trailing nul, that can* be passed to write.  If < PAGE_SIZE-1, PAGE_SIZE-1 is assumed.*/size_t max_write_len;/* CFTYPE_* flags */unsigned int flags;/** If non-zero, should contain the offset from the start of css to* a struct cgroup_file field.  cgroup will record the handle of* the created file into it.  The recorded handle can be used as* long as the containing css remains accessible.*/unsigned int file_offset;/** Fields used for internal bookkeeping.  Initialized automatically* during registration.*/struct cgroup_subsys *ss;    /* NULL for cgroup core files */struct list_head node;      /* anchored at ss->cfts */struct kernfs_ops *kf_ops;int (*open)(struct kernfs_open_file *of);void (*release)(struct kernfs_open_file *of);/** read_u64() is a shortcut for the common case of returning a* single integer. Use it in place of read()*/u64 (*read_u64)(struct cgroup_subsys_state *css, struct cftype *cft);/** read_s64() is a signed version of read_u64()*/s64 (*read_s64)(struct cgroup_subsys_state *css, struct cftype *cft);/* generic seq_file read interface */int (*seq_show)(struct seq_file *sf, void *v);/* optional ops, implement all or none */void *(*seq_start)(struct seq_file *sf, loff_t *ppos);void *(*seq_next)(struct seq_file *sf, void *v, loff_t *ppos);void (*seq_stop)(struct seq_file *sf, void *v);/** write_u64() is a shortcut for the common case of accepting* a single integer (as parsed by simple_strtoull) from* userspace. Use in place of write(); return 0 or error.*/int (*write_u64)(struct cgroup_subsys_state *css, struct cftype *cft,u64 val);/** write_s64() is a signed version of write_u64()*/int (*write_s64)(struct cgroup_subsys_state *css, struct cftype *cft,s64 val);/** write() is the generic write callback which maps directly to* kernfs write operation and overrides all other operations.* Maximum write size is determined by ->max_write_len.  Use* of_css/cft() to access the associated css and cft.*/ssize_t (*write)(struct kernfs_open_file *of,char *buf, size_t nbytes, loff_t off);__poll_t (*poll)(struct kernfs_open_file *of,struct poll_table_struct *pt);#ifdef CONFIG_DEBUG_LOCK_ALLOCstruct lock_class_key lockdep_key;
#endif
};

可以看到cfytpe定义了很多文件的函数指针，目的就是将用户对控制文件输入的字符（串）转换为对进程的资源管控，每一个cgroup 目录下的控制文件对应的cftype都包含cgroup_base_files + css -> ss -> cfts，分别对应来自cgroup和来自子系统的控制文件，cgroup的控制文件一般以cgroup.开头，来自子系统的控制文件一般以子系统的名称开头。

cgroup通过调用cgroup_addrm_file来添加删除控制文件，并将cftype保存到对应kernfs文件

cftype flags的类型

/* cftype->flags */
enum {CFTYPE_ONLY_ON_ROOT   = (1 << 0),  /* only create on root cgrp */CFTYPE_NOT_ON_ROOT    = (1 << 1),  /* don't create on root cgrp */CFTYPE_NS_DELEGATABLE   = (1 << 2),  /* writeable beyond delegation boundaries */CFTYPE_NO_PREFIX    = (1 << 3),  /* (DON'T USE FOR NEW FILES) no subsys prefix */CFTYPE_WORLD_WRITABLE  = (1 << 4),  /* (DON'T USE FOR NEW FILES) S_IWUGO */CFTYPE_DEBUG        = (1 << 5),  /* create when cgroup_debug *//* internal flags, do not use outside cgroup core proper */__CFTYPE_ONLY_ON_DFL   = (1 << 16), /* only on default hierarchy */__CFTYPE_NOT_ON_DFL  = (1 << 17), /* not on default hierarchy */
};