Data Wrangling数据处理

  • `|`的用法
  • sed:stream editor
  • wc
  • sort
  • uniq
  • paste
  • awk
  • R
  • BC
  • gnuplot
  • tee
  • Wrangling binary data
  • Regular expressions
    • capture groups
  • Exercise
    • 1. Take this [short interactive regex tutorial](https://regexone.com/).
    • 2.Find the number of words
    • 3. Use `man sed`
    • 4. boot time
    • 5.
    • 6.

https://missing.csail.mit.edu/2020/data-wrangling/

|的用法

  • ssh myserver journalctl | grep sshd | less | 将ssh myserver journalctl 的输出传送为grep的输入，再将grep的输出传送给less。less 是一个 Unix 命令行工具，用于浏览和搜索文本文件。它可以让你在命令行界面中分页查看文本文件，并提供许多方便的功能，如搜索、跳转到指定行、调整字体大小等。你可以在 less 中使用方向键移动光标、按 q 退出、按 / 开始搜索等。
  • ssh myserver 'journalctl | grep sshd | grep "Disconnected from"' > ssh.log 为什么使用单引号’ ‘将journalctl | grep sshd | grep “Disconnected from”’ > ssh.log引起来?Do the filtering on the remote server, and then massage the data locally.

sed:stream editor

  In it, you basically give short commands for how to modify the file, rather than manipulate its contents directly (although you can do that too).There are tons of commands, but one of the most common ones is s: substitution.For example, we can write:

ssh myserver journalctl| grep sshd| grep "Disconnected from"| sed 's/.*Disconnected from //'

  The s command is written on the form: s/REGEX/SUBSTITUTION/, where REGEX is the regular expression you want to search for, and SUBSTITUTION is the text you want to substitute matching text with.
  sed’s regular expressions are somewhat weird(奇怪、过时), and will require you to put a \ before most of these to give them their special meaning. Or you can pass -E.比如:sed -E 's/(ab)//g'或sed 's/\(ab\)//g' 去除所有的ab字符串。
  sed can do all sorts of other interesting things, like injecting text (with the i command), explicitly printing lines (with the p command), selecting lines by index, and lots of other things. Check man sed!

wc

  一个word count程序,用于统计字数。
wc -l表示统计行数。

sort

  用于排序，默认将每行的内容按照字典顺序排列。
sort -n表示按照数字顺序排序。
-k1,1 means “sort by only the first whitespace-separated column”. The ,n part says “sort until the n-th field, where the default is the end of the line.
-r表示反序排列

uniq

  对于一个已排序的行列表，uniq会使内容唯一化,删除重复内容。
uniq -c还会统计每种内容重复的次数。以重复次数 重复内容的格式输出。

paste

  paste -sd,lets you combine lines (-s) by a given single-character delimiter (-d; , in this case).

awk

| awk '$1 == 1 && $2 ~ /^c[^ ]*e$/ { print $2 }' First, what does {print $2} do? awk programs take the form of an optional pattern plus a block saying what to do if the pattern matches a given line. The default pattern (which we used above) matches all lines. Inside the block, $0 is set to the entire line’s contents, and $1 through $n are set to the n-th field of that line, when separated by the awk field separator (whitespace by default, change with -F). In this case, we’re saying that, for every line, print the contents of the second field!
  First, notice that we now have a pattern (the stuff that goes before {...}). The pattern says that the first field of the line should be equal to 1 (that’s the count from uniq -c), and that the second field should match the given regular expression. And the block just says to print the username.

awk 'BEGIN { rows = 0 } $1 == 1 && $2 ~ /^c[^ ]*e$/ { rows += $1 } END { print rows }  BEGIN is a pattern that matches the start of the input (and END matches the end). Now, the per-line block just adds the count from the first field (although it’ll always be 1 in this case), and then we print it out at the end.

R

  another (weird) programming language
| awk '{print $1}' | R --no-echo -e 'x <- scan(file="stdin", quiet=TRUE); summary(x)'  summary prints summary statistics for a vector, and we created a vector containing the input stream of numbers

BC

  全称"Berkeley Calculater"。
echo "1+2" | bc 计算1+2，输出3。

gnuplot

  简单画图工具
| sort -nk1,1 | tail -n10 | gnuplot -p -e 'set boxwidth 0.5; plot "-" using 1:xtic(2) with boxes'

tee

  把输出保存到文件中，同时在屏幕上看到输出内容。格式：tee [选项] 文件名
-a 追加写入文件，而非覆盖写入
-i 忽略中断信号

Wrangling binary data

  For example, we can use ffmpeg to capture an image from our camera, convert it to grayscale, compress it, send it to a remote machine over SSH, decompress it there, make a copy, and then display it.ffmpeg -loglevel panic -i /dev/video0 -frames 1 -f image2 - | convert - -colorspace gray - | gzip | ssh mymachine 'gzip -d | tee copy.jpg | env DISPLAY=:0 feh -'

Regular expressions

  Regular expressions are usually (though not always) surrounded by /。例如/.*Disconnected from /就是一个 Regular expressions。正则表达式debug网站

  • .匹配除换行符以外的任意字符means “any single character” except newline
  • *匹配前面的字符0次或多次zero or more of the preceding match
  • +匹配前面的字符1次或多次one or more of the preceding match
  • ?表示匹配前面的字符0次或1次

  * and + are, by default, “greedy”. They will match as much text as they can.
   比如: Jan 17 03:13:00 thesquareplanet.com sshd[2631]: Disconnected from invalid user Disconnected from 46.97.239.16 port 55920 [preauth] 这条字符串经过 sed 's/.*Disconnected from //'后变为46.97.239.16 port 55920 [preauth]
  In some regular expression implementations, you can just suffix * or + with a ? to make them non-greedy, but sadly sed doesn’t support that. We could switch to perl’s command-line mode though, which does support that construct:perl -pe 's/.*?Disconnected from //'

  • \d任意数字
  • \w任意字母或数字
  • \s任意空白字符（包括空格、制表符、换行符等）

大写表示非，如\D表示非数字

  • [abc] any one character of a, b, and c.[a-z]所有小写字母,[^0-9]所有非数字字符(包括换行号)
  • [^ ]是正则表达式中的一种特殊字符，表示非空格字符。在方括号[]中，^表示非，所以[^ ]就是非空格字符的意思。
  • (RX1|RX2) either something that matches RX1 or RX2。|表示或的关系
  • ^ $行首、行尾。^a匹配行首的a，a$匹配行尾的a。(在vim中,^是跳转到行首第一个非空字符,0才是跳到行尾)
  • \b表示匹配一个单词的边界( matches the boundary between a word and a non-word character)，在匹配整个单词时特别有用，比如\w+\b。
  • {和}表示匹配前面的字符特定的次数。b{4}表示bbbb,a{2,4}表示2到4个a,c{3,}表示3个或更多个c
  • (和)capture groups
  • (和)? non-capturing group

capture groups

  Any text matched by a regex surrounded by parentheses(括号) is stored in a numbered capture group. These are available in the substitution (and in some engines, even in the pattern itself!) as \1, \2, \3, etc.
| sed -E 's/.*Disconnected from (invalid |authenticating )?user (.*) [^ ]+ port [0-9]+( \[preauth\])?$/\2/' \2表示(.*)所匹配的内容。

Exercise

1. Take this short interactive regex tutorial.

2.Find the number of words

  Find the number of words(in /usr/share/dict/words) that contain at least three as and don’t have a 's ending. What are the three most common last two letters of those words? sed’s y command, or the tr program, may help you with case insensitivity. How many of those two-letter combinations are there? And for a challenge: which combinations do not occur?

  1. cat words | awk '$0 ~ /(.*?[aA]){3,}[A-Za-z]*$/{print $0 }' | sed -E 's/.*?(\w\w)/\1/' | sort | uniq -c | sort -n -k1,1 -r | wc -l输出两字母组合的种类,去掉末尾的|wc -l可以输出两字母组合按出现次数排列的列表。
  2. challenge:comm -23 <(echo {a..z}{a..z} | tr ' ' '\n' | sort) <(grep -i -E '(.*?[aA]){3,}[A-Za-z]*$' words | rev | cut -c1-2 | sort | uniq)

3. Use man sed

  To do in-place substitution it is quite tempting to do something like sed s/REGEX/SUBSTITUTION/ input.txt > input.txt. However this is a bad idea, why? Is this particular to sed? Use man sed to find out how to accomplish this.

  sed s/REGEX/SUBSTITUTION/ input.txt > input.txt原来的文件会被覆盖，导致数据丢失。
  使用 -i 参数，这样可以在文件本身上进行替换,如sed -i 's/REGEX/SUBSTITUTION/' input.txt。
  使用 -i[SUFFIX]的形式, 其中SUFFIX是可选的并且表示备份文件的后缀，默认是没有后缀。 如:sed -i.bak 's/old/new/' input.txt会替换文件input.txt中的匹配到的的 “old” 为 “new”,并且将修改前的文件备份到一个名为input.txt.bak的文件.这样就可以在不丢失数据的情况下完成文件内替换操作。
  这种问题不仅限于使用sed命令，其他类似的命令，如awk,perl,grep 等也有类似的问题，需要注意不要把文件重定向到自身。

4. boot time

  Find your average, median, and max system boot time over the last ten boots. Use journalctl on Linux and log show on macOS, and look for log timestamps near the beginning and end of each boot. On Linux, they may look something like:
Logs begin at ...andsystemd[577]: Startup finished in ...
On macOS, look for:
=== system boot:andPrevious shutdown cause: 5

journalctl --list-boots | sed -E 's/^.{51}(.{8}).{20}(.{8}).{4}$/\1 \2/'|awk '{split($1, a, ":"); split($2, b, ":"); if (b[3]<a[3]) {b[2]--; b[3]+=60} if (b[2]<a[2]) {b[1]--; b[2]+=60} if (b[1]<a[1]) {b[1]+=24} print ((b[1]-a[1])*3600 + (b[2]-a[2])*60 + (b[3]-a[3]))}' | sort -nr|awk '{a[NR]=$0;s++}END{if (s%2==1) print a[int(s/2)+1]; else print (a[s/2]+a[s/2+1])/2}'
  In awk, NR is a predefined variable that stands for “number of records” and it keeps track of the number of input records that have been processed so far. It is incremented by one for each input record read, starting at 1 for the first record. For example, when awk reads the first line of an input file, NR is set to 1, when it reads the second line, NR is set to 2, and so on. In the above command, I’m using NR as an index to store the numbers in an array a and also using it to get the middle element of the array. For example, if the input is a list of numbers separated by new line, awk read the first number, NR will be 1, it stores the number in the first element of the array a[1], when it reads the second number, NR will be 2, it stores the number in the second element of the array a[2], and so on.

5.

Look for boot messages that are not shared between your past three reboots (see journalctl’s -b flag). Break this task down into multiple steps. First, find a way to get just the logs from the past three boots. There may be an applicable flag on the tool you use to extract the boot logs, or you can use sed ‘0,/STRING/d’ to remove all lines previous to one that matches STRING. Next, remove any parts of the line that always varies (like the timestamp). Then, de-duplicate the input lines and keep a count of each one (uniq is your friend). And finally, eliminate any line whose count is 3 (since it was shared among all the boots).

6.

Find an online data set. Fetch it using curl and extract out just two columns of numerical data. If you’re fetching HTML data, pup might be helpful. For JSON data, try jq. Find the min and max of one column in a single command, and the difference of the sum of each column in another.

Data Wrangling数据处理相关推荐

  1. Data Wrangling

    数据整理(Data Wrangling) 数据整理(Data Wrangling)可归纳为以下三步: - 数据收集(Gather) - 数据评估(Assess) - 数据清理(Clean) 数据收集( ...

  2. python选课系统_【精选】在Monash读Data Science，人人都拥有这样一份选课指南。

    点击上方"蓝字",关注最适合你的学习咨询 前言 1.课程难度因人而异,课程作业也可能每学期变动,所以大家结合个人实际情况参考借鉴. 2.本指南系列只描述了比较最主流的课,冷门课程资 ...

  3. ssis组件_用于SSIS的Melissa Data Quality免费组件

    ssis组件 In this article, we will talk briefly about data quality in SQL Server. Then, we will give a ...

  4. python解题时间_1小时还是30秒？Python给你的另一种数据处理选择

    原标题:1小时还是30秒?Python给你的另一种数据处理选择 引子 想象一下,你每周都要手动重复同一过程,比如从多个来源复制数据并粘贴到一个电子表格中,用于后续处理.这项任务可能每周都需要花费一两个 ...

  5. R语言统计入门课程推荐——生物科学中的数据分析Data Analysis for the Life Sciences

    Data Analysis for the Life Sciences是哈佛大学PH525x系列课程--生物医学中的数据分析(PH525x series - Biomedical Data Scien ...

  6. RxSwift之深入解析URLSession的数据请求和数据处理

    一.请求网络数据 ① 通过 rx.response 请求数据 如下所示,通过豆瓣提供的音乐频道列表接口获取数据,并将返回结果输出到控制台中: "https://www.douban.com/ ...

  7. 数据科学家数据分析师_使您的分析师和数据科学家在数据处理方面保持一致

    数据科学家数据分析师 According to a recent survey conducted by Dimensional Research, only 50 percent of data a ...

  8. data mining (foreign blogs)

    出处:http://blog.csdn.net/shuimuqingyi/article/details/8698607 国外数据挖掘方面的经典博客 总体感觉数据挖掘行业在国内尚没有收到足够重视,国内 ...

  9. data mining blog (foreign)

    国外数据挖掘方面的经典博客 总体感觉数据挖掘行业在国内尚没有收到足够重视,国内的相关博客的内容也不够丰富,下面列出了一些国外数据挖掘方面的经典博客.数据挖掘是一个有趣的以及具有足够学术价值和商业价值的 ...

最新文章

  1. 卡尔曼滤波器的特殊案例
  2. Safari回传值给应用程序
  3. linux下使用automake、autoconf生成configure文件
  4. 通过C学Python（3）列表的常用方法
  5. excel切片器_如何在Excel表格中使用切片器
  6. php 取数组数量_PHP array_slice()：截取数组的一部分
  7. 37岁程序员失业投500份简历就3次面试猎头：超35岁不要
  8. @JsonIgnore和@JsonSerialize 的 区别
  9. MS SqL2000 数据库置疑状态的解决方法[转]
  10. 用flashAS3.0做一个连线题
  11. 如何卸载 STEP 7-MicroWIN编程软件？
  12. 二手笔记本电脑电池测试软件,笔记本电池修复软件Battery Doubler V1.2.1免费已注册版...
  13. Js字符串转json
  14. 用 Python 分析某医院药品销售案例
  15. 关于Word插入图片闪退
  16. 蜗牛星际b款装服务器系统,蜗牛星际B款单网口折腾
  17. 浩辰CAD给排水2021安装教程
  18. 非财务人员的财务培训教(一.二)------财务基础知识
  19. 运行python的两种方式磁盘式_day03-python-学习笔记
  20. 高人指路！如何借力国家标准让企业项目管理体制建设更容易？

热门文章

  1. 样本间的距离或者相似度度量方法
  2. docker加速器添加启动参数报错
  3. 仿抖音短视频APP源码，滚动视图
  4. 企业从无到有，究竟靠什么？
  5. 输入关键词自动获取相关图片素材的方法
  6. PC端抖音HTML布局,抖音有PC端了，终于可以发布无损视频了，支持15min内的作品...
  7. kvm桥接网络配置linux,KVM 虚拟机网卡桥接网络配置
  8. 使用HTML+CSS写一个静态页面
  9. 如何修改安卓界面上的图片和文字
  10. java web 第六天