Based on linux kernel4.14 arch/arm架构.

问题现象说明

死锁

crash现场

调用栈

core0:

PID: 0      TASK: c0e12a00  CPU: 0   COMMAND: "swapper/0"#0 [<c0a3df90>] (_raw_spin_lock) from [<c0198c04>]#1 [<c0198c04>] (handle_irq_event) from [<c019c8f0>]#2 [<c019c8f0>] (handle_fasteoi_irq) from [<c0197dc4>]#3 [<c0197dc4>] (__handle_domain_irq) from [<c01015e8>]#4 [<c01015e8>] (gic_handle_irq) from [<c010c9cc>]pc : [<c0747580>]    lr : [<c01c0efc>]    psr: 80000153sp : c0e01f50  ip : 00000000  fp : 1761ecfbr10: ede57600  r9 : 00000001  r8 : 00000001r7 : 83146d9f  r6 : 00000000  r5 : 83126e97  r4 : 8d4fdf3br3 : 00000000  r2 : 00039dc8  r1 : 00039dc8  r0 : 00000000Flags: Nzcv  IRQs on  FIQs off  Mode SVC_32  ISA ARM#5 [<c010c9cc>] (__irq_svc) from [<c01c0efc>]#6 [<c0747580>] (cpuidle_enter_state) from [<c017e6b8>]#7 [<c017e6b8>] (do_idle) from [<c017e788>]#8 [<c017e788>] (cpu_startup_entry) from [<c0d00a44>]
bt: WARNING: UNWIND: cannot find unwind table for c0d00a44

core1:

PID: 158    TASK: eedfdd00  CPU: 1   COMMAND: "irq/29-70800000"#0 [<c0a3e19c>] (_raw_spin_unlock_irq) from [<c019b420>]#1 [<c019b420>] (irq_finalize_oneshot) from [<c019b238>]#2 [<c019b238>] (irq_thread_fn) from [<c019b0c8>]#3 [<c019b0c8>] (irq_thread) from [<c014cd9c>]#4 [<c014cd9c>] (kthread) from [<c01077a8>]

寄存器信息

core0:

c0 pc  : c0a3df90, lr : c0198c04, cpsr : 20 0001d3, sp_usr : 00000000, sp_svc : c0e01e98
c0 pc :(_raw_spin_lock)
c0 sp  : c0e01e98, ip : d42ad42a, fp : c0e09ec8
c0 r10 : c0b896a4, r9 : 000003ff, r8 : 00000000
c0 r7  : c0e501d0, r6 : 00000002, r5 : eeaa6868
c0 r4  : eeaa6800, r3 : 00000001, r2 : 01ec01eb
c0 r1  : 01eb01eb, r0 : eeaa6868c0 sp : [c0e01e98 ---- c0e01f98]
c0 1e98: eeaa6868 eeaa6814 eeaa6800 c019c8f 0 00000000 ee802cc0 00000001 c0d7f0b4
c0 1eb8: c0e09ec8 c0197dc4 c0e01f00 c0e5014 4 0000002e 00000000 c0ec95ea c01015e8
c0 1ed8: 00000000 c0747580 80000153 fffffff f c0e01f34 00000001 c0e00000 ede57600
c0 1ef8: 1761ecfb c010c9cc 00000000 00039dc 8 00039dc8 00000000 8d4fdf3b 83126e97
c0 1f18: 00000000 83146d9f 00000001 0000000 1 ede57600 1761ecfb 00000000 c0e01f50
c0 1f38: c01c0efc c0747580 80000153 fffffff f 00000051 00000000 00000000 00000000
c0 1f58: 175e4f33 00000010 c0e091f0 ede5760 0 ede65000 00000001 c0e09254 c0d7ea48
c0 1f78: c0e091c8 c017e6b8 00000001 01e091c 5 2adb0b09 000000c0 c0d6c234 ffffffffc0 callstack:
c0 [<c0a3df90>] (_raw_spin_lock)
c0 [<c0198c04>] (handle_irq_event)
c0 [<c019c8f0>] (handle_fasteoi_irq)
c0 [<c0197dc4>] (__handle_domain_irq)
c0 [<c01015e8>] (gic_handle_irq)
c0 [<c010c9cc>] (__irq_svc)
c0 [<c0747580>] (cpuidle_enter_state)
c0 [<c017e6b8>] (do_idle)
c0 [<c017e788>] (cpu_startup_entry)
c0 [<c0d00a44>] (start_kernel)

core1:

c1 pc  : c0a3e19c, lr : c019b420, cpsr : 20 0001d3, sp_usr : 00000000, sp_svc : ede6df20
c1 pc :(_raw_spin_unlock_irq)
c1 sp  : ede6df20, ip : 00000001, fp : ede6 c000
c1 r10 : 00000001, r9 : eeaa6800, r8 : ede4 c800
c1 r7  : ede4c800, r6 : eeaa6868, r5 : eeaa 6814
c1 r4  : eeaa6800, r3 : 00000000, r2 : 0000 0c05
c1 r1  : ede6df20, r0 : 00000004c1 sp : [ede6df20 ---- ede6e020]
c1 df20: ede4c800 eeaa6800 00000001 ede4c82 4 ede6c000 c019b238 c0e091c8 ede6c00c
c1 df40: c019b1e8 c019b0c8 ffffffff eeaa686 8 c019b1e8 00000001 00000000 c019b240
c1 df60: 2adb0b09 00000000 eebb0380 ede4c84 0 ee897c28 0000000c c0ed0080 eebb039c
c1 df80: ede4c800 c014cd9c c019af5c ede4c84 0 c014cc40 00000000 00000000 00000000
c1 dfa0: 00000000 00000000 00000000 c01077a 8 00000000 00000000 00000000 00000000
c1 dfc0: 00000000 00000000 00000000 0000000 0 00000000 00000000 00000000 00000000
c1 dfe0: 00000000 00000000 00000000 0000000 0 00000013 00000000 00000000 00000000
c1 e000: 00000001 00000000 eea69000 ede4ce4 0 ede4a970 ede70c10 ede4e490 00000000c1 callstack:
c1 [<c0a3e19c>] (_raw_spin_unlock_irq)
c1 [<c019b420>] (irq_finalize_oneshot)
c1 [<c019b238>] (irq_thread_fn)
c1 [<c019b0c8>] (irq_thread)
c1 [<c014cd9c>] (kthread)
c1 [<c01077a8>] (ret_from_fork)

问题分析

core0在硬中断中，core1在中断线程中。
出现问题时，应该为core1先正在运行中断线程，但还未走到如下描述的irq_finalize_oneshot函数中。此时，core0收到了中断，并且快速的进入了handle_irq_event中。

core0:
__handle_domain_irqhandle_fasteoi_irqhandle_irq_eventcore1:
kthreadirq_threadirq_thread_fnirq_finalize_oneshot

core0在handle_irq_event中断处理函数中，将该irq配置为IRQD_IRQ_INPROGRESS，并放锁，处理一些事情，再拿锁。
core1在irq_finalize_oneshot中判断该irq是否有IRQD_IRQ_INPROGRESS标志，若有则进行循环，在循环中会不断的更新irq值。

handle_irq_event()
{...irqd_set(&desc->irq_data, IRQD_IRQ_INPROGRESS);raw_spin_unlock(&desc->lock);ret = handle_irq_event_percpu(desc);raw_spin_lock(&desc->lock); -- A 锁irqd_clear(&desc->irq_data, IRQD_IRQ_INPROGRESS);...
}irq_finalize_oneshot
{...
again:chip_bus_lock(desc);raw_spin_lock_irq(&desc->lock); -- B 锁if (unlikely(irqd_irq_inprogress(&desc->irq_data))) {raw_spin_unlock_irq(&desc->lock);chip_bus_sync_unlock(desc);cpu_relax()goto again;}...
}

在kernel中有机制可以保证，在不同的线程获取同一个锁，不会某一个线程持续获得，如下：

while (lockval.tickets.next != lockval.tickets.owner) {wfe();lockval.tickets.owner = ACCESS_ONCE(lock->tickets.owner);
}

那么问题是如何发生的？
通过返汇编，可以清晰的看到在arm与arm64，spin_lock的实现有区别，如下：

<_raw_spin_lock>:    mov     r0, lr
<_raw_spin_lock+4>:  movw    r1, #513        ; 0x201
<_raw_spin_lock+8>:  bl      0xc01309dc <__local_bh_enable_ip>
<_raw_spin_lock+12>: mov     r0, #0
<_raw_spin_lock+16>: pop     {r11, pc}
<_raw_spin_lock+20>: mov     r0, #1
<_raw_spin_lock+24>: dmb     ish
<_raw_spin_lock+28>: pop     {r11, pc}
<_raw_spin_lock+32>: mov     r1, #4
<_raw_spin_lock+36>: mov     r2, sp
<_raw_spin_lock+40>: bfi     r2, r1, #0, #13
<_raw_spin_lock+44>: ldr     r1, [r2]
<_raw_spin_lock+48>: add     r1, r1, #1
<_raw_spin_lock+52>: str     r1, [r2]
<_raw_spin_lock+56>: pldw    [r0]
<_raw_spin_lock+60>: ldrex   r1, [r0]
<_raw_spin_lock+64>: add     r2, r1, #65536  ; 0x10000
<_raw_spin_lock+68>: strex   r3, r2, [r0]
<_raw_spin_lock+72>: teq     r3, #0
<_raw_spin_lock+76>: bne     0xc0a3df84 <_raw_spin_lock+60>
<_raw_spin_lock+80>: uxth    r2, r1
<_raw_spin_lock+84>: cmp     r2, r1, lsr #16

在arm架构下，spin_lock拿锁的函数（循坏体），被替换为ldrex、strex，如下：

<_raw_spin_lock+60>: ldrex   r1, [r0]
<_raw_spin_lock+64>: add     r2, r1, #65536  ; 0x10000
<_raw_spin_lock+68>: strex   r3, r2, [r0]
<_raw_spin_lock+72>: teq     r3, #0
<_raw_spin_lock+76>: bne     0xc0a3df84 <_raw_spin_lock+60>

strex失败，导致core0一直重复获取锁，r3为strex指令的返回值：

c0 r4  : eeaa6800, r3 : 00000001, r2 : 01ec01eb

ARM 文档说明

DDI0487F_b_armv8_arm.pdf
https://developer.arm.com/documentation/ddi0487/latest/

STREX
Store Register Exclusive calculates an address from a base register value and an immediate offset, stores a word from a register to the calculated address if the PE has exclusive access to the memory at that address, and returns a status value of 0 if the store was successful, or of 1 if no store was performed.

DHT0008A_arm_synchronization_primitives
https://developer.arm.com/documentation/dht0008/a/arm-synchronization-primitives/exclusive-accesses/ldrex-and-strex

LDREX and STREX
The LDREX and STREX instructions split the operation of atomically updating memory into two separate steps. Together, they provide atomic updates in conjunction with exclusive monitors that track exclusive memory accesses, see Exclusive monitors. Load-Exclusive and Store-Exclusive must only access memory regions marked as Normal.

LDREX
The LDREX instruction loads a word from memory, initializing the state of the exclusive monitor(s) to track the synchronization operation. For example, LDREX R1, [R0] performs a Load-Exclusive from the address in R0, places the value into R1 and updates the exclusive monitor(s).

STREX
The STREX instruction performs a conditional store of a word to memory. If the exclusive monitor(s) permit the store, the operation updates the memory location and returns the value 0 in the destination register, indicating that the operation succeeded. If the exclusive monitor(s) do not permit the store, the operation does not update the memory location and returns the value 1 in the destination register. This makes it possible to implement conditional execution paths based on the success or failure of the memory operation. For example, STREX R2, R1, [R0] performs a Store-Exclusive operation to the address in R0, conditionally storing the value from R1 and indicating success or failure in R2.

Alternative exclusive access sizes
The ARMv6K architecture introduced byte, halfword and doubleword variants of LDREX and STREX:

LDREXB and STREXB
LDREXH and STREXH
LDREXD and STREXD.
The ARMv7 architecture added these to the Thumb instruction set in the A and R profiles. ARMv7-M supports the byte and halfword but not the doubleword variants. ARMv6-M does not support exclusive accesses.

The architecture requires that each Load-Exclusive instruction must be used only with the corresponding Store-Exclusive instruction, for example LDREXB must only be used with STREXB.

ldrex strex相关推荐

Android系统中提供的原子操作
代码的实现位于文件system/core/include/cutils中 http://androidxref.com/4.4.3_r1.1/xref/system/core/include/cuti ...
ARM下的原子操作实现原理
ARM下的原子操作实现原理本文的重点是学习C内嵌汇编的语法和ldrex/strex指令. 1.atomic_t类型定义 typedef struct { int counter; } ato ...
cas无法使用_一文彻底搞懂CAS实现原理
本文导读: 前言如何保障线程安全 CAS原理剖析 CPU如何保证原子操作解密CAS底层指令小结前言日常编码过程中,基本不会直接用到 CAS 操作,都是通过一些JDK 封装好的并发工具类来使用 ...
linux 外部协议请求,ARM架构和总线协议如何支持Linux原子操作?
这篇文章探讨ARM架构和总线协议如何来支持的.对于某款ARM处理器和总线CCI,CCN和CMN产品的具体实现,属于实现层面的微架构,一般需要NDA,这里不予讨论. 顺便提一下,在ARMv8 架构下对应 ...
浅谈单片机中变量访问的互斥
本文以stm32为硬件平台,浅谈一下变量访问的互斥. 假设是裸板,主程序和中断服务程序都对某个变量进行修改,那么很可能造成数据的不一致.查了一下,主要有两种解决方法:第一种是关中断,在访问公共资源的时 ...
Linux内核ARM构架中原子变量的底层实现研究
前段时间重新研究了一下Linux的并发控制机制,对于内核的自旋锁.互斥锁.信号量等机制及其变体做了底层代码上的研究.因为只有从原理上理解了这些机制,在编写驱动的时候才会记得应该注意什么.这些机制基本都 ...
Cortex-M3存储器系统
一.存储器系统的功能概览 1.Cortex-CM3存储器系统功能 1).存储器映射是预定义的,并且还规定好了那个位置使用那条总线. 2).Cortex-CM3的存储器系统支持"位带&q ...
内核常见锁的机制与实现分析1
今天讨论下内核常见锁的机制与实现分析. 第一个问题内核何时会发生临界资源的竞争访问? 对于非抢占UP(uni processor)内核只有一种情况会发生竞争, 即高优先级异常/中断处理函数抢占内核线程 ...
S5PV210体系结构与接口02：ARM编程模型汇编指令
目录 1. ARM的基本设定 1.1 ARM数据类型 1.1.1 基本数据类型 1.1.2 浮点数据类型 1.1.3 存储器大小端 1.2 支持的指令集 2. Cortex-A8编程模型 2.1 处理 ...

ldrex strex

目录