golang-exec cmd data race

问题背景：bytes.Buffer grow panic

panic: runtime error: slice bounds out of range [:1024] with capacity 512
goroutine 428529 [running]:
bytes.(*Buffer).grow(0xc0001e86f0, 0x200, 0x0)
#011/usr/local/go/src/bytes/buffer.go:148 +0x297
bytes.(*Buffer).ReadFrom(0xc0001e86f0, 0x80e000, 0xc0006aa2e8, 0x7fdd0865d028, 0xc0001e86f0, 0xc000449701)
#011/usr/local/go/src/bytes/buffer.go:202 +0x48
io.copyBuffer(0x80d980, 0xc0001e86f0, 0x80e000, 0xc0006aa2e8, 0x0, 0x0, 0x0, 0x4073b5, 0xc000948ea0, 0xc0004497b0)
#011/usr/local/go/src/io/io.go:395 +0x2ff
io.Copy(...)
#011/usr/local/go/src/io/io.go:368
os/exec.(*Cmd).writerDescriptor.func1(0xc000948ea0, 0xc0004497b0)
#011/usr/local/go/src/os/exec/exec.go:311 +0x65
os/exec.(*Cmd).Start.func1(0xc000c398c0, 0xc000f6ad60)
#011/usr/local/go/src/os/exec/exec.go:441 +0x27
created by os/exec.(*Cmd).Start
#011/usr/local/go/src/os/exec/exec.go:440 +0x629

执行exec.Cmd 中 write bytes.Buffer grow, 出现panic，常理标准库中应该不会出现这种错误；

google github 上让查data-race。比较符合slice 越界情况，除非出现多个线程同时操作buffer。

查看 writerDescripto 要么写 stdout，要么写 stderr，
再看代码：

多个线程 cmd.start 操作同一个 stderr，所以会出现竞态现象

// Pipe stdout of each command into stdin of next
func AssemblePipes(cmds []*exec.Cmd, stdin io.Reader, stdout io.Writer, stderr io.Writer) []*exec.Cmd {cmds[0].Stdin = stdincmds[0].Stderr = stderr// assemble pipesfor i, c := range cmds {if i < len(cmds)-1 {cmds[i+1].Stdin, _ = c.StdoutPipe()cmds[i+1].Stderr = stderr} else {c.Stdout = stdoutc.Stderr = stderr}}return cmds
}

使用 go build -race （后面有必要做一个检测）执行：

WARNING: DATA RACE
Write at 0x00c0000121a0 by goroutine 82:bytes.(*Buffer).ReadFrom()/usr/local/go/src/bytes/buffer.go:200 +0x48io.copyBuffer()/usr/local/go/src/io/io.go:388 +0x3faos/exec.(*Cmd).writerDescriptor.func1()/usr/local/go/src/io/io.go:364 +0x7aos/exec.(*Cmd).Start.func1()/usr/local/go/src/os/exec/exec.go:435 +0x34Previous write at 0x00c0000121a0 by goroutine 75:bytes.(*Buffer).ReadFrom()/usr/local/go/src/bytes/buffer.go:200 +0x48io.copyBuffer()/usr/local/go/src/io/io.go:388 +0x3faos/exec.(*Cmd).writerDescriptor.func1()/usr/local/go/src/io/io.go:364 +0x7aos/exec.(*Cmd).Start.func1()/usr/local/go/src/os/exec/exec.go:435 +0x34Goroutine 82 (running) created at:

WARNING: DATA RACE
Read at 0x00c000012180 by goroutine 82:bytes.(*Buffer).grow()/usr/local/go/src/bytes/buffer.go:73 +0x53bytes.(*Buffer).ReadFrom()/usr/local/go/src/bytes/buffer.go:202 +0x7cio.copyBuffer()/usr/local/go/src/io/io.go:388 +0x3faos/exec.(*Cmd).writerDescriptor.func1()/usr/local/go/src/io/io.go:364 +0x7aos/exec.(*Cmd).Start.func1()/usr/local/go/src/os/exec/exec.go:435 +0x34Previous write at 0x00c000012180 by goroutine 75:bytes.(*Buffer).grow()/usr/local/go/src/bytes/buffer.go:144 +0x23cbytes.(*Buffer).ReadFrom()/usr/local/go/src/bytes/buffer.go:202 +0x7cio.copyBuffer()/usr/local/go/src/io/io.go:388 +0x3faos/exec.(*Cmd).writerDescriptor.func1()/usr/local/go/src/io/io.go:364 +0x7aos/exec.(*Cmd).Start.func1()/usr/local/go/src/os/exec/exec.go:435 +0x34

能否确定是多个cmd 操作stderr 的问题呢？fmt 打印指针基本可以确定。
遗憾的是：不能用动态追踪追踪下 interface 的值：（研究课题）
https://medium.com/@mkevac
https://medium.com/bumble-tech/bpf-and-go-modern-forms-of-introspection-in-linux-6b9802682223
http://www.brendangregg.com/blog/2017-01-31/golang-bcc-bpf-function-tracing.html
https://github.com/grantseltzer/weaver/issues/15
https://riboseyim.github.io/2017/06/27/DTrace_bcc/

https://blog.pixielabs.ai/ebpf-function-tracing/post/
https://github.com/pixie-labs/pixie-demos/blob/main/simple-gotracing/http_trace_uprobe/utils.go
https://blog.pixielabs.ai/ebpf-http-tracing/
https://www.grant.pizza/blog/tracing-go-functions-with-ebpf-part-2/
https://github.com/iovisor/bcc/issues/1320
https://github.com/sematext/uprobe-http-tracer/blob/master/tracer/tracer.go
https://sematext.com/blog/ebpf-userland-apps/
https://www.slideshare.net/RayJenkins1/understanding-ebpf-in-a-hurry-149197981
https://blog.aquasec.com/intro-ebpf-tracing-containers
https://brunocalza.me/how-buffer-pool-works-an-implementation-in-go/
http://www.brendangregg.com/blog/2017-01-31/golang-bcc-bpf-function-tracing.html
https://pkg.go.dev/github.com/cilium/ebpf
https://medium.com/bumble-tech/bpf-and-go-modern-forms-of-introspection-in-linux-6b9802682223
https://www.linuxjournal.com/content/bpf-observability-getting-started-quickly
https://networkop.co.uk/post/2021-03-ebpf-intro/
https://chowdera.com/2021/04/20210401151937493O.html
https://news.ycombinator.com/item?id=24872974
https://speakerdeck.com/leodido/go-ebpf-superpowers
https://github.com/golang/go/issues/13398

golang-exec cmd data race相关推荐

golang exec cmd pipeline zombile 进程
问题背景:基于golang 实现了一个自助化配置的通用命令任务管理模块,这样运营人员可以直接通过修改配置,完成 if do 的命令逻辑:当磁盘满时,做什么?当网卡down 时做什么:并且可以支持各类环 ...
golang的 data race 分析
golang的data race 一.名词解析 1.data race: Any race is a bug 定义: ①多个线程(协程)对于同一个变量.②同时地.③进行读/写操作.并且④至少有一个线程 ...
golang data race 竞态条件
golang race condition 竞态条件 data race race condition golang race detector golang的协程机制使得编写并发代码变得非常容易,但 ...
golang 记一次data race排查过程
golang 记一次data race排查过程背景 data race 现场解决思路经验总结 data race在写并发代码时候经常遇到,相关基础概念的介绍可以参考之前一篇文章:golang d ...
数据竞争（data race）问题分析的利器——valgrind的Helgrind
数据竞争(data race)是指在非线程安全的情况下,多线程对同一个地址空间进行写操作.一般来说,我们都会通过线程同步方法来保证数据的安全,比如采用互斥量或者读写锁.但是由于某些笔误或者设计的缺陷, ...
数据争用(data race) 和竞态条件(race condition)
在有关多线程编程的话题中,数据争用(data race) 和竞态条件(race condition)是两个经常被提及的名词,它们两个有着相似的名字,也是我们在并行编程中极力避免出现的.但在处理实际问题 ...
go 并发编程之数据竞争 data race （三）
前言两个或多个 goroutine 访问同一个资源(如变量或数据结构),并尝试对该资源进行读写而不考虑其他 goroutine.这种类型的代码可以创建你见过的最疯狂和最随机的 bug.通常需要大量的 ...
别混淆数据争用(data race) 和竞态条件(race condition)
在有关多线程编程的话题中,数据争用(data race) 和竞态条件(race condition)是两个经常被提及的名词,它们两个有着相似的名字,也是我们在并行编程中极力避免出现的.但在处理实际问题 ...
go语言踩坑：data race导致的输出结果与预期结果不一致
一.第一个例子 package mainimport ("fmt""sync""time" )type person struct {Nam ...

golang-exec cmd data race

golang-exec cmd data race相关推荐

最新文章

热门文章