目录1.知识回顾2.fork()内对children、sibling的设置copy_process()dup_task_struct()设置task_struct-children和task_struct-sibling初始化尾插结论进程关系补充资料: Relationships Among Processes3.内核遍历所有进程的方法背景引入全局进程链表遍历方法: for_each_process宏父子链表遍历方法: prctl(PR_SET_CHILD_SUBREAPER,...)reaper进程的定义被subreaper进程收养和被init进程收养对比图prctl()me-signal-is_child_subreaper !!arg2★walk_process_tree()读前提醒函数分析准备工作for_each_threadlist_for_each_entry★详细分析简化后的遍历算法本文仍然分析的是v7.2.2的内核,和上篇文章一样1.知识回顾参见LKDS3.Linux内核的双向链表代码解析(3) 进程树遍历(上) 前置知识: children、sibling、real_parent、parent以及ptrace()调试进程文章2.fork()内对children、sibling的设置struct task_struct{ //...... struct list_head children; struct list_head sibling; struct list_head thread_node; /* Real parent process: */ struct task_struct __rcu *real_parent; /* Recipient of SIGCHLD, wait4() reports: */ struct task_struct __rcu *parent; //...... };上篇文章讲了Linux内核的task_struct里面的children、sibling、real_parent、parent用于描述进程之间的关系,那么负责创建子进程的fork()就需要在内部设置这些关系,这里只看设置的关键函数,以下是调用链:fork() └─ kernel_clone() └─ copy_process()copy_process()具体设置是在copy_process()中,定义在kernel/fork.c中:/* * This creates a new process as a copy of the old one, * but does not actually start it yet. * * It copies the registers, and all the appropriate * parts of the process environment (as per the clone * flags). The actual kick-off is left to the caller. */ __latent_entropy struct task_struct *copy_process( struct pid *pid, int trace, int node, struct kernel_clone_args *args) { struct task_struct *p; //...... p dup_task_struct(current, node); //...... INIT_LIST_HEAD(p-children); INIT_LIST_HEAD(p-sibling); //...... if (thread_group_leader(p)) { //...... list_add_tail(p-sibling, p-real_parent-children); //...... } return p; //...... }可以看到,父进程copy_process()创建子进程的过程中,先将父进程的task_struct部分内容复制到子进程的地址空间中,这个是dup_task_struct()做的://p是copy_process的局部变量 //current不是变量,而是宏,用于获取指向父进程的task_struct的指针 //node和NUMA有关,是NUMA_NO_NODE(是从kernel_clone()一路传递到这里的),涉及到内存管理,和本文目的无关,所以这里不讲 p dup_task_struct(current, node);dup_task_struct()dup_task_struct()定义在kernel/fork.c中,从名字中的dup(全称是dump,意思是转储、转存)看,肯定有复制功能我去掉了CONFIG_THREAD_INFO_IN_TASK、CONFIG_SECCOMP等内核配置的宏://orig指向父进程的task_struct static struct task_struct *dup_task_struct(struct task_struct *orig, int node) { struct task_struct *tsk; int err; if (node NUMA_NO_NODE) node tsk_fork_get_node(orig); tsk alloc_task_struct_node(node); if (!tsk) return NULL; err arch_dup_task_struct(tsk, orig); if (err) goto free_tsk; err alloc_thread_stack_node(tsk, node); if (err) goto free_tsk; account_kernel_stack(tsk, 1); err scs_prepare(tsk, node); if (err) goto free_stack; RCU_INIT_POINTER(tsk-exec_state, NULL); setup_thread_stack(tsk, orig); clear_user_return_notifier(tsk); clear_tsk_need_resched(tsk); set_task_stack_end_magic(tsk); clear_syscall_work_syscall_user_dispatch(tsk); if (orig-cpus_ptr orig-cpus_mask) tsk-cpus_ptr tsk-cpus_mask; dup_user_cpus_ptr(tsk, orig, node); /* * One for the user space visible state that goes away when reaped. * One for the scheduler. */ refcount_set(tsk-rcu_users, 2); /* One for the rcu users */ refcount_set(tsk-usage, 1); tsk-splice_pipe NULL; tsk-task_frag.page NULL; tsk-wake_q.next NULL; tsk-worker_private NULL; kcov_task_init(tsk); kmsan_task_create(tsk); kmap_local_fork(tsk); return tsk; free_stack: exit_task_stack_account(tsk); free_thread_stack(tsk); free_tsk: free_task_struct(tsk); return NULL; }这里只讲和本文相关的关键函数:1.为子进程分配task_struct的空间//tsk是函数内部变量,类型是struct task_struct * tsk alloc_task_struct_node(node);2.完全复制父进程的task_structerr arch_dup_task_struct(tsk, orig); if (err) goto free_tsk;arch_dup_task_struct()具体实现和架构有关,这里看x86的,定义在/arch/x86/kernel/process.c中:/* * this gets called so that we can store lazy state into memory and copy the * current task into the new thread. */ int arch_dup_task_struct(struct task_struct *dst, struct task_struct *src) { /* fpu_clone() will initialize the dst_fpu memory */ memcpy_and_pad(dst, arch_task_struct_size, src, sizeof(*dst), 0); #ifdef CONFIG_VM86 dst-thread.vm86 NULL; #endif return 0; }memcpy_and_pad()是内核改进的memcpy函数,定义在/lib/string_helpers.c中:/** * memcpy_and_pad - Copy one buffer to another with padding * dest: Where to copy to * dest_len: The destination buffer size * src: Where to copy from * count: The number of bytes to copy * pad: Character to use for padding if space is left in destination. */ void memcpy_and_pad(void *dest, size_t dest_len, const void *src, size_t count, int pad) { if (dest_len count) { memcpy(dest, src, count); memset(dest count, pad, dest_len - count); } else { memcpy(dest, src, dest_len); } } EXPORT_SYMBOL(memcpy_and_pad);结合起来看,就是:结论:arch_dup_task_struct()负责完整复制(底层用memcoy()从头复制到尾,不是部分复制!!!)父进程的task_struct给子进程3.设置子进程的线程栈(内核不区分进程和线程),也可以说是内核栈,注意不要和OS69.【Linux】使用线程求和、pthread_self、了解clone系统调用、创建多个线程、线程栈文章讲的用户态下的线程栈搞混了setup_thread_stack(tsk, orig);注: 内核栈是内核私有的,具体可看6. Kernel Stacks — The Linux Kernel documentation,不是本文重点4.更新子进程的task_struct的部分字段,dup_task_struct()的后半部分,这里就不摘了结论: dup_task_struct内部主要为子进程分配放task_struct的空间复制父进程的task_struct全部内容设置子进程的内核栈修改子进程的task_struct部分字段设置task_struct-children和task_struct-sibling子进程的task_struct对象已经准备完毕,那么就设置其task_struct-children和task_struct-sibling,具体来说是:初始化初始化task_struct-children和task_struct-sibling指向的节点的前后指针,INIT_LIST_HEAD的内部就是做这个的,具体对该宏的分析可以看LKDS1.Linux内核的双向链表代码解析(1) 初始化、插入、删除文章尾插调用list_add_tail()将p-sibling指向的节点插入到 p-real_parent-children链表尾部p指向的是父进程创建的子进程的task_struct由于父进程可能之前创建过其它子进程,这里展示一般情况,设置前:设置后:结论结论: sibling存的是临近兄弟进程的位置,而children串的是父进程创建的所有子进程; 内核通过双向链表的方法来维护进程树; list_add_tail(p-sibling, p-real_parent-children)把新子进程p的sibling节点,挂到父进程p-real_parent的children链表尾部,而不是挂新子进程p的task_struct,因为这样能方便地遍历附近的兄弟进程这里再给一个简图: shell父进程创建了n个子进程,它们分别为子进程p1、...、子进程pnmemraid代码,可以到Online Mermaid Viewer - Live Mermaid Diagram Editor渲染flowchart TB subgraph Parent[父进程 task_struct (p-real_parent)] PC[children: list_head] end subgraph Child1[子进程nbsp;nbsp;nbsp;p1nbsp;nbsp;nbsp;task_struct] C1[sibling: list_head] end DOTS[...] subgraph ChildN[子进程nbsp;nbsp;nbsp;nbsp;nbsp;nbsp;nbsp;nbsp;nbsp;nbsp;nbsp;nbsp;nbsp;nbsp;nbsp;nbsp;nbsp;pnnbsp;nbsp;nbsp;nbsp;nbsp;nbsp;nbsp;nbsp;nbsp;nbsp;nbsp;nbsp;nbsp;task_struct] CN[sibling: list_head] end PC -- C1 C1 -- DOTS DOTS -- CN CN -- PC进程关系补充资料:Relationships Among Processes如果读者还有点疑惑,可以看看Understanding The Linux Kernel 3rd edition书的第91~92页的描述,里面总结了内核中进程的几种链表:Relationships Among ProcessesProcesses created by a program have a parent/child relationship. When a process createsmultiple children,these children have sibling relationships(解释了struct list_head sibling的含义). Several fields must be introduced in aprocess descriptor(Linux下就是task_struct)to represent these relationships; they are listed in Table 3-3 with respect to a given process P. Processes 0 and 1 are created by the kernel; as we’ll see later in the chapter, process 1 (init) is the ancestor of all other processes.Table 3-3. Fields of a process descriptor used to express parenthood relationshipsField nameDescriptionreal_parentPoints to the process descriptor of the process that created P or to the descriptor of process 1(init) if the parent process no longer exists. (Therefore, when a user starts a background processand exits the shell, the background process becomes the child of init.)parentPoints to the current parent of P (this is the process that must be signaled when the child processterminates); its value usually coincides with that of real_parent. It may occasionally differ,such as when another process issues a ptrace( ) system call requesting that it be allowed tomonitor P (see the section “Execution Tracing” in Chapter 20).childrenThe head of the list containing all children created by P.siblingThe pointers to the next and previous elements in the list of the sibling processes, those thathave the same parent as P.Figure 3-4 illustrates the parent and sibling relationships of a group of processes. ProcessP0 successively created P1, P2, and P3. Process P3, in turn, created process P4.3.内核遍历所有进程的方法背景引入上篇文章讲过,在 Linux 内核中,进程的管理依赖两种核心数据结构: 一是全局的tasks双向链表,链上了所有的进程,是线性结构; 二是基于父子关系的children链表,用于维护进程树(其实还有哈希表,但这里只讲链表)内核对这两种链表给了2种遍历方法,如下全局进程链表遍历方法: for_each_process宏全局进程链表的节点是task_struct里面的tasks:​ struct task_struct { //...... struct list_head tasks; //...... }; ​for_each_process宏定义在/include/linux/sched/signal.h中:#define next_task(p) \ list_entry_rcu((p)-tasks.next, struct task_struct, tasks) //init_task就是PID为1的进程,p的类型是struct task_struct* #define for_each_process(p) \ for (p init_task ; (p next_task(p)) ! init_task ; )算法很简单,从init进程开始,按照next取下一个进程,最终遍历完所有的进程,这个适合直来直去的线性的全局进程链表具体取下一个进程的方法在next_task宏中,画个图演示:遍历方法: prctl(PR_SET_CHILD_SUBREAPER,...)prctl()是个系统调用函数,之前的文章没有讲过,看之前需要先了解reaper进程是什么reaper进程的定义之前在OS20.【Linux】进程状态(2) 僵尸进程、孤儿进程和进程优先级文章讲过孤儿进程的定义: 父进程创建子进程,如果父进程比子进程先终止,那么子进程就是孤儿进程,内核必须为孤儿进程找一个新的父进程,默认情况下,这个新的父进程是init进程,就是reaper,称为子进程的收割者,负责处理子进程变成僵尸状态后为其收尸那子进程的收割者能不能不是init进程呢?答: 其实可以,内核可以找subreaper,这个概念出现在解释PR_SET_CHILD_SUBREAPER宏的手册中,这个宏是Linux 3.4加入的新特性,要结合prctl()系统调用使用被subreaper进程收养和被init进程收养对比图这里直接摘宋宝华老师在《打通Linux脉络系列进程、线程和调度》课上的图《Linux进程、线程和调度》系列9.15日第二节课ppt分享,注意p2和p4终止后,p3和p5的新父进程不一样p2终止时,从p3向上遍历进程树发现没有subreaper,那么p3被init进程收养;p2终止时,从p3向上遍历树发现有subreaper,那么p3被subreaper进程收养;结论: 在Linux进程中,若父进程先于子进程死亡,那么Linux内核将把子进程“托孤”给进程树为subreaper的进程,并由这个subreaper来负责该孤儿进程的“收尸过程”,若未找到subreaper进程,则该孤儿进程直接托付给init进程prctl()先看看手册怎么讲的:man 2 prctlprctl的全称是processcontrol,用于控制调用进程或线程的行为,具体怎么控制由第一个参数op决定,参数op的取值有很多,这里只看PR_SET_CHILD_SUBREAPER:prctl(PR_SET_CHILD_SUBREAPER,一个非零参数)用于设置当前调用该函数的进程为子进程的子收割者,也就是说失去父进程的子进程,内核会将后者托孤给当前子进程最近的subreaper进程,如果没有subreaper进程,就会被init进程收养那内核是怎么找subreaper的呢?答: 这要看内核源码分析了,查看prctl()系统调用的实现,定义在kernel/sys.c中:SYSCALL_DEFINE5(prctl, int, option, unsigned long, arg2, unsigned long, arg3, unsigned long, arg4, unsigned long, arg5) { struct task_struct *me current; unsigned char comm[sizeof(me-comm)]; long error; error security_task_prctl(option, arg2, arg3, arg4, arg5); if (error ! -ENOSYS) return error; error 0; switch (option) { //...... case PR_SET_CHILD_SUBREAPER: me-signal-is_child_subreaper !!arg2; if (!arg2) break; walk_process_tree(me, propagate_has_child_subreaper, NULL); break; //...... default: trace_task_prctl_unknown(option, arg2, arg3, arg4, arg5); error -EINVAL; break; } return error; }me-signal-is_child_subreaper !!arg2根据手册的描述If arg2 is nonzero, set the child subreaper attribute of the calling process; if arg2 is zero, unset the attribute.当arg2!0,那么!arg2为0,!!arg2为1,此时内核设置了调用进程的task_struct里面的is_child_subreaper标志为1,即setthe child subreaper attribute of the calling process当arg20,那么!arg2为1,!!arg2为0,此时内核设置了调用进程的task_struct里面的is_child_subreaper标志为0,即unsetthe attribute接着,当arg2!0,执行walk_process_tree(),进程树遍历★walk_process_tree()读前提醒注: 读者必须明白task_struct.pid、task_struct.tgid、task_struct.group_leader、task_struct.signal.thread_head的真实含义才能继续往下看walk_process_tree()的分析,这里给三个测试题帮助读者判断是否真正掌握了它们:1.Linux系统中,无论是单进程线程还是多进程线程,getpid()系统调用返回的是进程的task_struct对象的pid成员( )struct task_struct { //...... pid_t pid; pid_t tgid; struct pid *thread_pid; //...... };A.正确B.错误2.Linux系统中,对于多线程的进程而言,每个线程都调用getpid(),所有的返回值( )A.一样B.不一样3.Linux系统中,对于多线程的进程而言,所有线程的task_struct的group_leader都指向主线程的task_struct对象( )struct task_struct { //...... struct task_struct *group_leader; //...... }A.正确B.错误4.Linux系统中,关于同一线程组的线程分为主线程和新线程,主线程创建新线程,下列说法正确的是( )struct signal_struct { //...... struct list_head thread_head; //...... } struct task_struct { //...... struct signal_struct *signal; //...... struct list_head thread_node; //...... }A. 主线程有独立的signal_struct对象,新线程没有B. 主线程和新线程各自拥有独立的signal_struct对象C. 主线程和新线程共享同一个 signal_struct对象,它们的task_struct-signal指针值相同D. 新线程创建时会复制一份主线程的signal_struct对象,内容同步但地址不同三道题的答案分别是A、B、A、C,如果读者不明白的话,看完这篇文章就明白了:LKDS4.Linux内核的双向链表代码解析(4) 进程树遍历(中) 前置知识: 解析PID、TID、TGID函数分析准备工作定义在/kernel/fork.c中://typedef int (*proc_visitor)(struct task_struct *p, void *data); //visitor参数是propagate_has_child_subreaper函数指针 //top是指向当前调用这个函数的进程的task_struct void walk_process_tree(struct task_struct *top, proc_visitor visitor, void *data) { struct task_struct *leader, *parent, *child; int res; read_lock(tasklist_lock); leader top top-group_leader; down: for_each_thread(leader, parent) { list_for_each_entry(child, parent-children, sibling) { res visitor(child, data); if (res) { if (res 0) goto out; leader child; goto down; } up: ; } } if (leader ! top) { child leader; parent child-real_parent; leader parent-group_leader; goto up; } out: read_unlock(tasklist_lock); }top指向当前调用walk_process_tree()任务(内核不区分进程和线程,都叫任务)的task_struct对象,为了方便讲解,给一个例子,假设是t8调用了walk_process_tree():局部指针:struct task_struct *leader, *parent, *child;为了防止多个CPU核心同时遍历进程树,遍历前需要加锁,防止链表结构被并发修改,保证链表结构稳定:read_lock(tasklist_lock);接着,内核修改传入walk_process_tree()的top为线程组组长top-group_leader,在这里线程组组长是t4,leader指针也一样:leader top top-group_leader;那么图为:for_each_thread接着执行遍历,先看for_each_thread,定义在/include/linux/sched/signal.h中:#define __for_each_thread(signal, t) \ list_for_each_entry_rcu(t, (signal)-thread_head, thread_node, \ lockdep_is_held(tasklist_lock)) #define for_each_thread(p, t) \ __for_each_thread((p)-signal, t) /** * list_for_each_entry_rcu - iterate over rcu list of given type * pos: the type * to use as a loop cursor. * head: the head for your list. * member: the name of the list_head within the struct. * cond: optional lockdep expression if called from non-RCU protection. * * This list-traversal primitive may safely run concurrently with * the _rcu list-mutation primitives such as list_add_rcu() * as long as the traversal is guarded by rcu_read_lock(). */ #define list_for_each_entry_rcu(pos, head, member, cond...) \ for (__list_check_rcu(dummy, ## cond, 0), \ pos list_entry_rcu((head)-next, typeof(*pos), member); \ pos-member ! (head); \ pos list_entry_rcu(pos-member.next, typeof(*pos), member))注: 代码中的##是可变参数宏里面的,见130.【C语言】可变参数宏文章展开过程:展开结果for (__list_check_rcu(dummy,lockdep_is_held(tasklist_lock), 0), \ parent list_entry_rcu((((leader)-signal)-thread_head)-next, typeof(*parent), thread_node); \ parent-thread_node ! (((leader)-signal)-thread_head); \ parent list_entry_rcu(parent-thread_node.next, typeof(*parent), thread_node))对于研究遍历算法而言,__list_check_rcu和lockdep_is_held不需要管,再把list_entry_rcu近似替换为container_of(注: list_entry_rcu在container_of的基础上添加了READ_ONCE宏,不扣那么细的话,其实没什么区别)那么for_each_thread可简化为:for (parent container_of((((leader)-signal)-thread_head)-next, typeof(*parent), thread_node); \ parent-thread_node ! (((leader)-signal)-thread_head); \ parent container_of(parent-thread_node.next, typeof(*parent), thread_node))list_for_each_entrylist_for_each_entry也是一个for循环.之前在Linux内核的双向链表代码解析(2) 遍历算法文章专门讲过,这里就不展开了结论: for_each_threadlist_for_each_entry是双重循环★详细分析简化后的遍历算法为了减小代码复杂程度,不展开for_each_thread和list_for_each_entry,去除锁//typedef int (*proc_visitor)(struct task_struct *p, void *data); //visitor参数是propagate_has_child_subreaper函数指针 //top是指向当前调用这个函数的进程的task_struct void walk_process_tree(struct task_struct *top, proc_visitor visitor, void *data) { struct task_struct *leader, *parent, *child; int res; leader top top-group_leader; down: for_each_thread(leader, parent) { list_for_each_entry(child, parent-children, sibling) { res visitor(child, data); if (res) { if (res 0) //propagate_has_child_subreaper只会返回0或1,下面会说到,这个if没有用 goto out; leader child; goto down; } up: ; //这里有个分号,是有意为之的!! } } if (leader ! top) { child leader; parent child-real_parent; leader parent-group_leader; goto up; } out: }for_each_thread从线程组长开始,顺着链表负责访问线程组中的每个线程,parent指向每次访问的线程的task_structlist_for_each_entry(child, parent-children, sibling)算法如下:propagate_has_child_subreaper()定义在/kernel/sys.c中:static int propagate_has_child_subreaper(struct task_struct *p, void *data) { /* * If task has has_child_subreaper - all its descendants * already have these flag too and new descendants will * inherit it on fork, skip them. * * If weve found child_reaper - skip descendants in * its subtree as they will never get out pidns. */ if (p-signal-has_child_subreaper || is_child_reaper(task_pid(p))) return 0; p-signal-has_child_subreaper 1; return 1; }这个函数非常直白,而且从字面意思上就能理解has_child_subreaper和is_child_reaper如果p-signal-has_child_subreaper为true,那么p指向的任务在成为孤儿时,会上层子收割者领养如果p-signal-is_child_reaper为true,那么p指向的任务就是子收割者if (p-signal-has_child_subreaper || is_child_reaper(task_pid(p))) //如果这个任务有子收割者或者本来就是子收割者,那么函数返回0 return 0; p-signal-has_child_subreaper 1; //否则将这个任务设置成有子收割者,函数返回1 return 1;注意到propagate_has_child_subreaper()返回值可0可1,那么内核在执行的时候一定会存储返回值的,显然walk_process_tree()确实做了,返回值存储在res中down: for_each_thread(leader, parent) { list_for_each_entry(child, parent-children, sibling) { res visitor(child, data); if (res) { //这个任务被propagate_has_child_subreaper()设置成有子收割者 leader child; goto down; } up: ; //如果这个任务有子收割者或者本来就是子收割者,那么执行到这里 } }下篇文章将演示遍历过程