原理分析:

1.安装服务:

fluentd 安装

https://github.com/heheliu321/kubernetes/tree/master/cluster/addons/fluentd-elasticsearch

创建configmap资源(key --filename)

修改fluentd-es-configmap.yaml中的es地址,其他不用动

 output.conf: |-<match **>@id elasticsearch@type elasticsearch@log_level infotype_name _docinclude_tag_key truehost 192.168.13.1port 9200logstash_format true<buffer>@type filepath /var/log/fluentd-buffers/kubernetes.system.bufferflush_mode intervalretry_type exponential_backoffflush_thread_count 2flush_interval 5sretry_foreverretry_max_interval 30chunk_limit_size 2Mqueue_limit_length 8overflow_action block</buffer></match>
root@ubuntu-128:~# kubectl apply -f fluentd-es-configmap.yaml

将pod通过configmaps将 配置信息加载到pod中

root@ubuntu-128:~# kubectl apply -f fluentd-es-ds.yaml

其中fluentd需要镜像 quay.io/fluentd_elasticsearch/fluentd:v2.6.0作为插件,完成将pod日志上传到es

 docker pull quay.io/fluentd_elasticsearch/fluentd:v2.6.0

查看可以知道ubutu-130的fluentd成功运行,

root@ubuntu-128:/home/itcast/working/fluentd# kubectl get pods --namespace=kube-system -o wide
NAME                                 READY   STATUS        RESTARTS   AGE    IP               NODE         NOMINATED NODE   READINESS GATES
coredns-59b69b999c-drvmt             1/1     Running       1          118m   10.244.0.31      ubuntu-128   <none>           <none>
coredns-59b69b999c-jhx29             1/1     Running       0          34m    10.244.1.124     ubuntu-130   <none>           <none>
etcd-ubuntu-128                      1/1     Running       8          4d1h   192.168.13.128   ubuntu-128   <none>           <none>
fluentd-es-v2.6.0-cl6f8              1/1     Running       0          20m    10.244.1.125     ubuntu-130   <none>           <none>
kube-apiserver-ubuntu-128            1/1     Running       12         4d1h   192.168.13.128   ubuntu-128   <none>           <none>
kube-controller-manager-ubuntu-128   1/1     Running       72         23m    192.168.13.128   ubuntu-128   <none>           <none>
kube-flannel-ds-amd64-6fnlw          1/1     Running       0          20m    192.168.13.128   ubuntu-128   <none>           <none>
kube-flannel-ds-amd64-7b54h          1/1     Terminating   5          4d     192.168.13.129   ubuntu-129   <none>           <none>
kube-flannel-ds-amd64-cxwsg          1/1     Running       14         4d     192.168.13.130   ubuntu-130   <none>           <none>
kube-proxy-6hb9w                     1/1     Running       8          4d1h   192.168.13.128   ubuntu-128   <none>           <none>
kube-proxy-wgk4w                     1/1     Running       4          4d     192.168.13.129   ubuntu-129   <none>           <none>
kube-proxy-xqgrw                     1/1     Running       10         4d     192.168.13.130   ubuntu-130   <none>           <none>
kube-scheduler-ubuntu-128            1/1     Running       66         4d1h   192.168.13.128   ubuntu-128   <none>           <none>
tiller-deploy-7b9c5bf9c4-zvxq5       1/1     Running       7          2d2h   10.244.1.111     ubuntu-130   <none>           <none>

测试

这里讲一下日志的细节,日志记录了大量的操作系统细节,重点注意一下,

root@ubuntu-128:/home/itcast/working/fluentd# kubectl logs fluentd-es-v2.6.0-cl6f8 --namespace=kube-system
2019-08-16 15:27:16 +0000 [warn]: [elasticsearch] failed to flush the buffer. retry_time=0 next_retry_seconds=2019-08-16 15:27:17 +0000 chunk="5903d9f6da39eac27f4f331803e25de2" error_class=Fluent::Plugin::ElasticsearchOutput::RecoverableRequestFailure error="could not push logs to Elasticsearch cluster ({:host=>\"192.168.13.1\", :port=>9200, :scheme=>\"http\"}): read timeout reached"2019-08-16 15:27:16 +0000 [warn]: /var/lib/gems/2.3.0/gems/fluent-plugin-elasticsearch-3.5.2/lib/fluent/plugin/out_elasticsearch.rb:757:in `rescue in send_bulk'2019-08-16 15:27:16 +0000 [warn]: /var/lib/gems/2.3.0/gems/fluent-plugin-elasticsearch-3.5.2/lib/fluent/plugin/out_elasticsearch.rb:733:in `send_bulk'2019-08-16 15:27:16 +0000 [warn]: /var/lib/gems/2.3.0/gems/fluent-plugin-elasticsearch-3.5.2/lib/fluent/plugin/out_elasticsearch.rb:627:in `block in write'2019-08-16 15:27:16 +0000 [warn]: /var/lib/gems/2.3.0/gems/fluent-plugin-elasticsearch-3.5.2/lib/fluent/plugin/out_elasticsearch.rb:626:in `each'2019-08-16 15:27:16 +0000 [warn]: /var/lib/gems/2.3.0/gems/fluent-plugin-elasticsearch-3.5.2/lib/fluent/plugin/out_elasticsearch.rb:626:in `write'2019-08-16 15:27:16 +0000 [warn]: /var/lib/gems/2.3.0/gems/fluentd-1.5.1/lib/fluent/plugin/output.rb:1125:in `try_flush'2019-08-16 15:27:16 +0000 [warn]: /var/lib/gems/2.3.0/gems/fluentd-1.5.1/lib/fluent/plugin/output.rb:1425:in `flush_thread_run'2019-08-16 15:27:16 +0000 [warn]: /var/lib/gems/2.3.0/gems/fluentd-1.5.1/lib/fluent/plugin/output.rb:454:in `block (2 levels) in start'2019-08-16 15:27:16 +0000 [warn]: /var/lib/gems/2.3.0/gems/fluentd-1.5.1/lib/fluent/plugin_helper/thread.rb:78:in `block in thread_create'
2019-08-16 15:27:17 +0000 [warn]: [elasticsearch] retry succeeded. chunk_id="5903d9f6da39eac27f4f331803e25de2"
2019-08-16 15:28:20 +0000 [warn]: dump an error event: error_class=Fluent::Plugin::ConcatFilter::TimeoutError error="Timeout flush: kernel:default" location=nil tag="kernel" time=2019-08-16 15:28:20.403295017 +0000 record={"boot_id"=>"417c6389e89a4793bc868854e9608715", "machine_id"=>"7e7d8472ba514c0daf3ab2ad126bcf6e", "hostname"=>"ubuntu-130", "source_monotonic_timestamp"=>"0", "transport"=>"kernel", "priority"=>"5", "syslog_facility"=>"0", "syslog_identifier"=>"kernel", "message"=>"random: get_random_bytes called from start_kernel+0x42/0x50d with crng_init=0Linux version 4.13.0-36-generic (buildd@lgw01-amd64-033) (gcc version 5.4.0 20160609 (Ubuntu 5.4.0-6ubuntu1~16.04.9)) #40~16.04.1-Ubuntu SMP Fri Feb 16 23:25:58 UTC 2018 (Ubuntu 4.13.0-36.40~16.04.1-generic 4.13.13)Command line: BOOT_IMAGE=/boot/vmlinuz-4.13.0-36-generic root=UUID=9d46940e-386b-4c36-aa05-c1974cb537ac ro quiet splashKERNEL supported cpus:  Intel GenuineIntel  AMD AuthenticAMD  Centaur CentaurHaulsDisabled fast string operationsx86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers'x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers'x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers'x86/fpu: xstate_offset[2]:  576, xstate_sizes[2]:  256x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format.e820: BIOS-provided physical RAM map:BIOS-e820: [mem 0x0000000000000000-0x000000000009e7ff] usableBIOS-e820: [mem 0x000000000009e800-0x000000000009ffff] reservedBIOS-e820: [mem 0x00000000000dc000-0x00000000000fffff] reservedBIOS-e820: [mem 0x0000000000100000-0x000000007fedffff] usableBIOS-e820: [mem 0x000000007fee0000-0x000000007fefefff] ACPI dataBIOS-e820: [mem 0x000000007feff000-0x000000007fefffff] ACPI NVSBIOS-e820: [mem 0x000000007ff00000-0x000000007fffffff] usableBIOS-e820: [mem 0x00000000f0000000-0x00000000f7ffffff] reservedBIOS-e820: [mem 0x00000000fec00000-0x00000000fec0ffff] reservedBIOS-e820: [mem 0x00000000fee00000-0x00000000fee00fff] reservedBIOS-e820: [mem 0x00000000fffe0000-0x00000000ffffffff] reservedNX (Execute Disable) protection: activerandom: fast init doneSMBIOS 2.7 present.DMI: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 07/02/2015Hypervisor detected: VMwarevmware: TSC freq read from hypervisor : 2494.420 MHzvmware: Host bus clock speed read from hypervisor : 66000000 Hzvmware: using sched offset of 6062854211 nse820: update [mem 0x00000000-0x00000fff] usable ==> reservede820: remove [mem 0x000a0000-0x000fffff] usablee820: last_pfn = 0x80000 max_arch_pfn = 0x400000000MTRR default type: uncachableMTRR fixed ranges enabled:  00000-9FFFF write-back  A0000-BFFFF uncachable  C0000-CFFFF write-protect  D0000-EFFFF uncachable  F0000-FFFFF write-protectMTRR variable ranges enabled:  0 base 000C0000000 mask 3FFC0000000 uncachable  1 base 00000000000 mask 3FF00000000 write-back  2 disabled  3 disabled  4 disabled  5 disabled  6 disabled  7 disabledx86/PAT: Configuration [0-7]: WB  WC  UC- UC  WB  WC  UC- WT  total RAM covered: 3072MFound optimal setting for mtrr clean up gran_size: 64K \tchunk_size: 64K \tnum_reg: 2  \tlose cover RAM: 0Gfound SMP MP-table at [mem 0x000f6a80-0x000f6a8f] mapped at [ffff927b800f6a80]Scanning 1 areas for low memory corruptionBase memory trampoline at [ffff927b80098000] 98000 size 24576BRK [0x3d72b000, 0x3d72bff
9-08-16 15:35:07 +0000 [warn]: [elasticsearch] retry succeeded. chunk_id="5903dbb635ba93a5b7402539ca7ba3bd"
2019-08-16 15:45:50 +0000 [warn]: [elasticsearch] failed to flush the buffer. retry_time=0 next_retry_seconds=2019-08-16 15:45:51 +0000 chunk="5903de18c168d1484e7409f0d90e36c3" error_class=Fluent::Plugin::ElasticsearchOutput::RecoverableRequestFailure error="could not push logs to Elasticsearch cluster ({:host=>\"192.168.13.1\", :port=>9200, :scheme=>\"http\"}): read timeout reached"2019-08-16 15:45:50 +0000 [warn]: suppressed same stacktrace
2019-08-16 15:45:59 +0000 [warn]: [elasticsearch] failed to flush the buffer. retry_time=1 next_retry_seconds=2019-08-16 15:45:59 +0000 chunk="5903de1d5506de45bc10846751a5e79f" error_class=Fluent::Plugin::ElasticsearchOutput::RecoverableRequestFailure error="could not push logs to Elasticsearch cluster ({:host=>\"192.168.13.1\", :port=>9200, :scheme=>\"http\"}): read timeout reached"2019-08-16 15:45:59 +0000 [warn]: suppressed same stacktrace
2019-08-16 15:45:59 +0000 [warn]: [elasticsearch] failed to flush the buffer. retry_time=2 next_retry_seconds=2019-08-16 15:46:01 +0000 chunk="5903de18c168d1484e7409f0d90e36c3" error_class=Fluent::Plugin::ElasticsearchOutput::RecoverableRequestFailure error="could not push logs to Elasticsearch cluster ({:host=>\"192.168.13.1\", :port=>9200, :scheme=>\"http\"}): read timeout reached"2019-08-16 15:45:59 +0000 [warn]: suppressed same stacktrace
2019-08-16 15:46:02 +0000 [warn]: [elasticsearch] retry succeeded. chunk_id="5903de18c168d1484e7409f0d90e36c3"

在ubuntu-130中 使用tcpdump可以看到fluentd将日志推送到ES

root@ubuntu-130:~# tcpdump -i any port 9200 -s 0
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on any, link-type LINUX_SLL (Linux cooked), capture size 262144 bytes
23:52:21.601934 IP 10.244.1.125.57142 > 192.168.13.1.9200: Flags [S], seq 927584757, win 28200, options [mss 1410,sackOK,TS val 4040995733 ecr 0,nop,wscale 7], length 0
23:52:21.601989 IP 10.244.1.125.57142 > 192.168.13.1.9200: Flags [S], seq 927584757, win 28200, options [mss 1410,sackOK,TS val 4040995733 ecr 0,nop,wscale 7], length 0
23:52:21.602018 IP ubuntu-130.57142 > 192.168.13.1.9200: Flags [S], seq 927584757, win 28200, options [mss 1410,sackOK,TS val 4040995733 ecr 0,nop,wscale 7], length 0
23:52:21.602515 IP 192.168.13.1.9200 > ubuntu-130.57142: Flags [S.], seq 1478163250, ack 927584758, win 8192, options [mss 1460,nop,wscale 8,sackOK,TS val 1042016 ecr 4040995733], length 0
23:52:21.602543 IP 192.168.13.1.9200 > 10.244.1.125.57142: Flags [S.], seq 1478163250, ack 927584758, win 8192, options [mss 1460,nop,wscale 8,sackOK,TS val 1042016 ecr 4040995733], length 0
23:52:21.602550 IP 192.168.13.1.9200 > 10.244.1.125.57142: Flags [S.], seq 1478163250, ack 927584758, win 8192, options [mss 1460,nop,wscale 8,sackOK,TS val 1042016 ecr 4040995733], length 0
23:52:21.602617 IP 10.244.1.125.57142 > 192.168.13.1.9200: Flags [.], ack 1, win 221, options [nop,nop,TS val 4040995734 ecr 1042016], length 0
23:52:21.623993 IP 192.168.13.1.9200 > ubuntu-130.57142: Flags [P.], seq 1:550, ack 2137, win 256, options [nop,nop,TS val 1042018 ecr 4040995738], length 549
23:52:21.624050 IP 192.168.13.1.9200 > 10.244.1.125.57142: Flags [P.], seq 1:550, ack 2137, win 256, options [nop,nop,TS val 1042018 ecr 4040995738], length 549
23:52:21.624057 IP 192.168.13.1.9200 > 10.244.1.125.57142: Flags [P.], seq 1:550, ack 2137, win 256, options [nop,nop,TS val 1042018 ecr 4040995738], length 549
23:52:21.624110 IP 10.244.1.125.57142 > 192.168.13.1.9200: Flags [.], ack 550, win 229, options [nop,nop,TS val 4040995755 ecr 1042018], length 0
23:52:21.624119 IP 10.244.1.125.57142 > 192.168.13.1.9200: Flags [.], ack 550, win 229, options [nop,nop,TS val 4040995755 ecr 1042018], length 0
23:52:21.624131 IP ubuntu-130.57142 > 192.168.13.1.9200: Flags [.], ack 550, win 229, options [nop,nop,TS val 4040995755 ecr 1042018], length 0
23:52:21.626041 IP 10.244.1.125.57142 > 192.168.13.1.9200: Flags [F.], seq 2137, ack 550, win 229, options [nop,nop,TS val 4040995757 ecr 1042018], length 0
23:52:21.626089 IP 10.244.1.125.57142 > 192.168.13.1.9200: Flags [F.], seq 2137, ack 550, win 229, options [nop,nop,TS val 4040995757 ecr 1042018], length 0
23:52:21.626117 IP ubuntu-130.57142 > 192.168.13.1.9200: Flags [F.], seq 2137, ack 550, win 229, options [nop,nop,TS val 4040995757 ecr 1042018], length 0
23:52:21.626136 IP 192.168.13.1.9200 > ubuntu-130.57144: Flags [P.], seq 1846136596:1846137571, ack 3099668854, win 256, options [nop,nop,TS val 1042018 ecr 4040995738], length 975
23:52:21.626144 IP 192.168.13.1.9200 > 10.244.1.125.57144: Flags [P.], seq 1846136596:1846137571, ack 3099668854, win 256, options [nop,nop,TS val 1042018 ecr 4040995738], length 975
23:52:21.626147 IP 192.168.13.1.9200 > 10.244.1.125.57144: Flags [P.], seq 0:975, ack 1, win 256, options [nop,nop,TS val 1042018 ecr 4040995738], length 975
23:52:21.626373 IP 10.244.1.125.57144 > 192.168.13.1.9200: Flags [.], ack 975, win 236, options [nop,nop,TS val 4040995758 ecr 1042018], length 0
23:52:21.626394 IP 10.244.1.125.57144 > 192.168.13.1.9200: Flags [.], ack 975, win 236, options [nop,nop,TS val 4040995758 ecr 1042018], length 0
23:52:21.626404 IP ubuntu-130.57144 > 192.168.13.1.9200: Flags [.], ack 975, win 236, options [nop,nop,TS val 4040995758 ecr 1042018], length 0
23:52:21.626523 IP 192.168.13.1.9200 > ubuntu-130.57142: Flags [.], ack 2138, win 256, options [nop,nop,TS val 1042018 ecr 4040995757], length 0
23:52:21.626538 IP 192.168.13.1.9200 > 10.244.1.125.57142: Flags [.], ack 2138, win 256, options [nop,nop,TS val 1042018 ecr 4040995757], length 0
23:52:21.626542 IP 192.168.13.1.9200 > 10.244.1.125.57142: Flags [.], ack 2138, win 256, options [nop,nop,TS val 1042018 ecr 4040995757], length 0
23:52:21.626994 IP 192.168.13.1.9200 > ubuntu-130.57142: Flags [R.], seq 550, ack 2138, win 0, length 0
23:52:21.629256 IP 10.244.1.125.57144 > 192.168.13.1.9200: Flags [F.], seq 1, ack 975, win 236, options [nop,nop,TS val 4040995761 ecr 1042018], length 0
23:52:21.629300 IP 10.244.1.125.57144 > 192.168.13.1.9200: Flags [F.], seq 1, ack 975, win 236, options [nop,nop,TS val 4040995761 ecr 1042018], length 0
23:52:21.629326 IP ubuntu-130.57144 > 192.168.13.1.9200: Flags [F.], seq 1, ack 975, win 236, options [nop,nop,TS val 4040995761 ecr 1042018], length 0
23:52:21.630715 IP 192.168.13.1.9200 > ubuntu-130.57144: Flags [.], ack 2, win 256, options [nop,nop,TS val 1042018 ecr 4040995761], length 0
23:52:21.630800 IP 192.168.13.1.9200 > 10.244.1.125.57144: Flags [.], ack 2, win 256, options [nop,nop,TS val 1042018 ecr 4040995761], length 0
23:52:21.630808 IP 192.168.13.1.9200 > 10.244.1.125.57144: Flags [.], ack 2, win 256, options [nop,nop,TS val 1042018 ecr 4040995761], length 0
23:52:21.630814 IP 192.168.13.1.9200 > ubuntu-130.57144: Flags [R.], seq 975, ack 2, win 0, length 0
23:52:21.630823 IP 192.168.13.1.9200 > 10.244.1.125.57144: Flags [R.], seq 975, ack 2, win 0, length 0
23:52:21.630825 IP 192.168.13.1.9200 > 10.244.1.125.57144: Flags [R.], seq 975, ack 2, win 0, length 0

查看es节点中index

http://192.168.13.1:9200/_cat/indices?v
_______________________________________________________________________________________
health status index               uuid                   pri rep docs.count docs.deleted store.size pri.store.size
yellow open   logstash-2019.08.16 y7KXQWwXQP6EZ-r5MMGzrQ   5   1      10399            0      5.9mb          5.9mb

查看ES的logstash-2019.08.16索引数据

http://192.168.13.1:9200/logstash-2019.08.16/_search
——————————————————————————————————————————————————————————
{"took": 2,"timed_out": false,"_shards": {"total": 5,"successful": 5,"skipped": 0,"failed": 0},"hits": {"total": 10441,"max_score": 1,"hits": [{"_index": "logstash-2019.08.16","_type": "_doc","_id": "8wIIm2wBYm28bAuFxTQo","_score": 1,"_source": {"stream": "stdout","docker": {"container_id": "f57a82ec41244abc0aba2b846070494f021250d44789fd0a81bc5f2ae92d1fba"},"kubernetes": {"container_name": "istio-proxy","namespace_name": "istio-system","pod_name": "istio-policy-7bc45bb8f9-4lwjp","container_image": "istio/proxyv2:1.0.3","container_image_id": "docker-pullable://istio/proxyv2@sha256:947348e2039b8b0e356e843ae263dc0c3d50abbf6cfe9d676446353f85b9ccb7","pod_id": "29054276-bea1-11e9-ad52-000c295a7d67","host": "ubuntu-130","labels": {"app": "policy","istio": "mixer","istio-mixer-type": "policy","pod-template-hash": "7bc45bb8f9"},"master_url": "https://10.96.0.1:443/api","namespace_id": "8b3de99f-be97-11e9-9d80-000c295a7d67"},"message": "discoveryRefreshDelay: 1s\n","@timestamp": "2019-08-16T15:25:31.383692784+00:00","tag": "kubernetes.var.log.containers.istio-policy-7bc45bb8f9-4lwjp_istio-system_istio-proxy-f57a82ec41244abc0aba2b846070494f021250d44789fd0a81bc5f2ae92d1fba.log"}},{"_index": "logstash-2019.08.16","_type": "_doc","_id": "-AIIm2wBYm28bAuFxTQo","_score": 1,"_source": {"stream": "stdout","docker": {"container_id": "f57a82ec41244abc0aba2b846070494f021250d44789fd0a81bc5f2ae92d1fba"},"kubernetes": {"container_name": "istio-proxy","namespace_name": "istio-system","pod_name": "istio-policy-7bc45bb8f9-4lwjp","container_image": "istio/proxyv2:1.0.3","container_image_id": "docker-pullable://istio/proxyv2@sha256:947348e2039b8b0e356e843ae263dc0c3d50abbf6cfe9d676446353f85b9ccb7","pod_id": "29054276-bea1-11e9-ad52-000c295a7d67","host": "ubuntu-130","labels": {"app": "policy","istio": "mixer","istio-mixer-type": "policy","pod-template-hash": "7bc45bb8f9"},"master_url": "https://10.96.0.1:443/api","namespace_id": "8b3de99f-be97-11e9-9d80-000c295a7d67"},"message": "\n","@timestamp": "2019-08-16T15:25:31.387587739+00:00","tag": "kubernetes.var.log.containers.istio-policy-7bc45bb8f9-4lwjp_istio-system_istio-proxy-f57a82ec41244abc0aba2b846070494f021250d44789fd0a81bc5f2ae92d1fba.log"}},{"_index": "logstash-2019.08.16","_type": "_doc","_id": "_QIIm2wBYm28bAuFxTQo","_score": 1,"_source": {"stream": "stdout","docker": {"container_id": "f57a82ec41244abc0aba2b846070494f021250d44789fd0a81bc5f2ae92d1fba"},"kubernetes": {"container_name": "istio-proxy","namespace_name": "istio-system","pod_name": "istio-policy-7bc45bb8f9-4lwjp","container_image": "istio/proxyv2:1.0.3","container_image_id": "docker-pullable://istio/proxyv2@sha256:947348e2039b8b0e356e843ae263dc0c3d50abbf6cfe9d676446353f85b9ccb7","pod_id": "29054276-bea1-11e9-ad52-000c295a7d67","host": "ubuntu-130","labels": {"app": "policy","istio": "mixer","istio-mixer-type": "policy","pod-template-hash": "7bc45bb8f9"},"master_url": "https://10.96.0.1:443/api","namespace_id": "8b3de99f-be97-11e9-9d80-000c295a7d67"},"message": "  address:\n","@timestamp": "2019-08-16T15:25:31.393629656+00:00","tag": "kubernetes.var.log.containers.istio-policy-7bc45bb8f9-4lwjp_istio-system_istio-proxy-f57a82ec41244abc0aba2b846070494f021250d44789fd0a81bc5f2ae92d1fba.log"}},{"_index": "logstash-2019.08.16","_type": "_doc","_id": "AwIIm2wBYm28bAuFxTUo","_score": 1,"_source": {"stream": "stdout","docker": {"container_id": "f57a82ec41244abc0aba2b846070494f021250d44789fd0a81bc5f2ae92d1fba"},"kubernetes": {"container_name": "istio-proxy","namespace_name": "istio-system","pod_name": "istio-policy-7bc45bb8f9-4lwjp","container_image": "istio/proxyv2:1.0.3","container_image_id": "docker-pullable://istio/proxyv2@sha256:947348e2039b8b0e356e843ae263dc0c3d50abbf6cfe9d676446353f85b9ccb7","pod_id": "29054276-bea1-11e9-ad52-000c295a7d67","host": "ubuntu-130","labels": {"app": "policy","istio": "mixer","istio-mixer-type": "policy","pod-template-hash": "7bc45bb8f9"},"master_url": "https://10.96.0.1:443/api","namespace_id": "8b3de99f-be97-11e9-9d80-000c295a7d67"},"message": "  stats_tags:\n","@timestamp": "2019-08-16T15:25:31.397690383+00:00","tag": "kubernetes.var.log.containers.istio-policy-7bc45bb8f9-4lwjp_istio-system_istio-proxy-f57a82ec41244abc0aba2b846070494f021250d44789fd0a81bc5f2ae92d1fba.log"}},{"_index": "logstash-2019.08.16","_type": "_doc","_id": "FgIIm2wBYm28bAuFxTUo","_score": 1,"_source": {"stream": "stdout","docker": {"container_id": "f57a82ec41244abc0aba2b846070494f021250d44789fd0a81bc5f2ae92d1fba"},"kubernetes": {"container_name": "istio-proxy","namespace_name": "istio-system","pod_name": "istio-policy-7bc45bb8f9-4lwjp","container_image": "istio/proxyv2:1.0.3","container_image_id": "docker-pullable://istio/proxyv2@sha256:947348e2039b8b0e356e843ae263dc0c3d50abbf6cfe9d676446353f85b9ccb7","pod_id": "29054276-bea1-11e9-ad52-000c295a7d67","host": "ubuntu-130","labels": {"app": "policy","istio": "mixer","istio-mixer-type": "policy","pod-template-hash": "7bc45bb8f9"},"master_url": "https://10.96.0.1:443/api","namespace_id": "8b3de99f-be97-11e9-9d80-000c295a7d67"},"message": "    connect_timeout: 0.250s\n","@timestamp": "2019-08-16T15:25:31.522235510+00:00","tag": "kubernetes.var.log.containers.istio-policy-7bc45bb8f9-4lwjp_istio-system_istio-proxy-f57a82ec41244abc0aba2b846070494f021250d44789fd0a81bc5f2ae92d1fba.log"}},{"_index": "logstash-2019.08.16","_type": "_doc","_id": "GQIIm2wBYm28bAuFxTUo","_score": 1,"_source": {"stream": "stdout","docker": {"container_id": "f57a82ec41244abc0aba2b846070494f021250d44789fd0a81bc5f2ae92d1fba"},"kubernetes": {"container_name": "istio-proxy","namespace_name": "istio-system","pod_name": "istio-policy-7bc45bb8f9-4lwjp","container_image": "istio/proxyv2:1.0.3","container_image_id": "docker-pullable://istio/proxyv2@sha256:947348e2039b8b0e356e843ae263dc0c3d50abbf6cfe9d676446353f85b9ccb7","pod_id": "29054276-bea1-11e9-ad52-000c295a7d67","host": "ubuntu-130","labels": {"app": "policy","istio": "mixer","istio-mixer-type": "policy","pod-template-hash": "7bc45bb8f9"},"master_url": "https://10.96.0.1:443/api","namespace_id": "8b3de99f-be97-11e9-9d80-000c295a7d67"},"message": "    - socket_address:\n","@timestamp": "2019-08-16T15:25:31.523743382+00:00","tag": "kubernetes.var.log.containers.istio-policy-7bc45bb8f9-4lwjp_istio-system_istio-proxy-f57a82ec41244abc0aba2b846070494f021250d44789fd0a81bc5f2ae92d1fba.log"}},{"_index": "logstash-2019.08.16","_type": "_doc","_id": "GwIIm2wBYm28bAuFxTUp","_score": 1,"_source": {"stream": "stdout","docker": {"container_id": "f57a82ec41244abc0aba2b846070494f021250d44789fd0a81bc5f2ae92d1fba"},"kubernetes": {"container_name": "istio-proxy","namespace_name": "istio-system","pod_name": "istio-policy-7bc45bb8f9-4lwjp","container_image": "istio/proxyv2:1.0.3","container_image_id": "docker-pullable://istio/proxyv2@sha256:947348e2039b8b0e356e843ae263dc0c3d50abbf6cfe9d676446353f85b9ccb7","pod_id": "29054276-bea1-11e9-ad52-000c295a7d67","host": "ubuntu-130","labels": {"app": "policy","istio": "mixer","istio-mixer-type": "policy","pod-template-hash": "7bc45bb8f9"},"master_url": "https://10.96.0.1:443/api","namespace_id": "8b3de99f-be97-11e9-9d80-000c295a7d67"},"message": "        address: 127.0.0.1\n","@timestamp": "2019-08-16T15:25:31.524735918+00:00","tag": "kubernetes.var.log.containers.istio-policy-7bc45bb8f9-4lwjp_istio-system_istio-proxy-f57a82ec41244abc0aba2b846070494f021250d44789fd0a81bc5f2ae92d1fba.log"}},{"_index": "logstash-2019.08.16","_type": "_doc","_id": "HwIIm2wBYm28bAuFxTUp","_score": 1,"_source": {"stream": "stdout","docker": {"container_id": "f57a82ec41244abc0aba2b846070494f021250d44789fd0a81bc5f2ae92d1fba"},"kubernetes": {"container_name": "istio-proxy","namespace_name": "istio-system","pod_name": "istio-policy-7bc45bb8f9-4lwjp","container_image": "istio/proxyv2:1.0.3","container_image_id": "docker-pullable://istio/proxyv2@sha256:947348e2039b8b0e356e843ae263dc0c3d50abbf6cfe9d676446353f85b9ccb7","pod_id": "29054276-bea1-11e9-ad52-000c295a7d67","host": "ubuntu-130","labels": {"app": "policy","istio": "mixer","istio-mixer-type": "policy","pod-template-hash": "7bc45bb8f9"},"master_url": "https://10.96.0.1:443/api","namespace_id": "8b3de99f-be97-11e9-9d80-000c295a7d67"},"message": "      - max_connections: 100000\n","@timestamp": "2019-08-16T15:25:31.527531467+00:00","tag": "kubernetes.var.log.containers.istio-policy-7bc45bb8f9-4lwjp_istio-system_istio-proxy-f57a82ec41244abc0aba2b846070494f021250d44789fd0a81bc5f2ae92d1fba.log"}},{"_index": "logstash-2019.08.16","_type": "_doc","_id": "IAIIm2wBYm28bAuFxTUp","_score": 1,"_source": {"stream": "stdout","docker": {"container_id": "f57a82ec41244abc0aba2b846070494f021250d44789fd0a81bc5f2ae92d1fba"},"kubernetes": {"container_name": "istio-proxy","namespace_name": "istio-system","pod_name": "istio-policy-7bc45bb8f9-4lwjp","container_image": "istio/proxyv2:1.0.3","container_image_id": "docker-pullable://istio/proxyv2@sha256:947348e2039b8b0e356e843ae263dc0c3d50abbf6cfe9d676446353f85b9ccb7","pod_id": "29054276-bea1-11e9-ad52-000c295a7d67","host": "ubuntu-130","labels": {"app": "policy","istio": "mixer","istio-mixer-type": "policy","pod-template-hash": "7bc45bb8f9"},"master_url": "https://10.96.0.1:443/api","namespace_id": "8b3de99f-be97-11e9-9d80-000c295a7d67"},"message": "        max_pending_requests: 100000\n","@timestamp": "2019-08-16T15:25:31.528025002+00:00","tag": "kubernetes.var.log.containers.istio-policy-7bc45bb8f9-4lwjp_istio-system_istio-proxy-f57a82ec41244abc0aba2b846070494f021250d44789fd0a81bc5f2ae92d1fba.log"}},{"_index": "logstash-2019.08.16","_type": "_doc","_id": "JQIIm2wBYm28bAuFxTUp","_score": 1,"_source": {"stream": "stdout","docker": {"container_id": "f57a82ec41244abc0aba2b846070494f021250d44789fd0a81bc5f2ae92d1fba"},"kubernetes": {"container_name": "istio-proxy","namespace_name": "istio-system","pod_name": "istio-policy-7bc45bb8f9-4lwjp","container_image": "istio/proxyv2:1.0.3","container_image_id": "docker-pullable://istio/proxyv2@sha256:947348e2039b8b0e356e843ae263dc0c3d50abbf6cfe9d676446353f85b9ccb7","pod_id": "29054276-bea1-11e9-ad52-000c295a7d67","host": "ubuntu-130","labels": {"app": "policy","istio": "mixer","istio-mixer-type": "policy","pod-template-hash": "7bc45bb8f9"},"master_url": "https://10.96.0.1:443/api","namespace_id": "8b3de99f-be97-11e9-9d80-000c295a7d67"},"message": "    - pipe:\n","@timestamp": "2019-08-16T15:25:31.532554059+00:00","tag": "kubernetes.var.log.containers.istio-policy-7bc45bb8f9-4lwjp_istio-system_istio-proxy-f57a82ec41244abc0aba2b846070494f021250d44789fd0a81bc5f2ae92d1fba.log"}}]}
}
http://192.168.13.1:9200/logstash-2019.08.16/_doc/HwIIm2wBYm28bAuFxTUp
——————————————————————————————————————————————————————————————————————
{"_index": "logstash-2019.08.16","_type": "_doc","_id": "HwIIm2wBYm28bAuFxTUp","_version": 1,"found": true,"_source": {"stream": "stdout","docker": {"container_id": "f57a82ec41244abc0aba2b846070494f021250d44789fd0a81bc5f2ae92d1fba"},"kubernetes": {"container_name": "istio-proxy","namespace_name": "istio-system","pod_name": "istio-policy-7bc45bb8f9-4lwjp","container_image": "istio/proxyv2:1.0.3","container_image_id": "docker-pullable://istio/proxyv2@sha256:947348e2039b8b0e356e843ae263dc0c3d50abbf6cfe9d676446353f85b9ccb7","pod_id": "29054276-bea1-11e9-ad52-000c295a7d67","host": "ubuntu-130","labels": {"app": "policy","istio": "mixer","istio-mixer-type": "policy","pod-template-hash": "7bc45bb8f9"},"master_url": "https://10.96.0.1:443/api","namespace_id": "8b3de99f-be97-11e9-9d80-000c295a7d67"},"message": "      - max_connections: 100000\n","@timestamp": "2019-08-16T15:25:31.527531467+00:00","tag": "kubernetes.var.log.containers.istio-policy-7bc45bb8f9-4lwjp_istio-system_istio-proxy-f57a82ec41244abc0aba2b846070494f021250d44789fd0a81bc5f2ae92d1fba.log"}
}

从日志看,他貌似只收集kube-system日志,没有收集业务日志(比如demployment-nginx),后台思考是因为demployment-nginx在default namespace命令空间下

root@ubuntu-128:/home/itcast/working/deployment# kubectl get pod --namespace=kube-system -o wide
NAME                                 READY   STATUS    RESTARTS   AGE     IP               NODE         NOMINATED NODE   READINESS GATES
coredns-59b69b999c-drvmt             1/1     Running   18         13h     10.244.0.35      ubuntu-128   <none>           <none>
coredns-59b69b999c-qllxs             1/1     Running   0          140m    10.244.1.145     ubuntu-130   <none>           <none>
etcd-ubuntu-128                      1/1     Running   12         4d12h   192.168.13.128   ubuntu-128   <none>           <none>
fluentd-es-v2.6.0-l86vd              1/1     Running   0          68m     10.244.1.148     ubuntu-130   <none>           <none>
kube-apiserver-ubuntu-128            1/1     Running   17         4d12h   192.168.13.128   ubuntu-128   <none>           <none>
kube-controller-manager-ubuntu-128   1/1     Running   110        12h     192.168.13.128   ubuntu-128   <none>           <none>
kube-flannel-ds-amd64-6fnlw          1/1     Running   5          11h     192.168.13.128   ubuntu-128   <none>           <none>
kube-flannel-ds-amd64-cxwsg          1/1     Running   16         4d12h   192.168.13.130   ubuntu-130   <none>           <none>
kube-proxy-6hb9w                     1/1     Running   12         4d12h   192.168.13.128   ubuntu-128   <none>           <none>
kube-proxy-xqgrw                     1/1     Running   11         4d12h   192.168.13.130   ubuntu-130   <none>           <none>
kube-scheduler-ubuntu-128            1/1     Running   101        151m    192.168.13.128   ubuntu-128   <none>           <none>
nginx-deployment-76bf4969df-7h5mw    1/1     Running   0          33m     10.244.1.149     ubuntu-130   <none>           <none>
tiller-deploy-7b9c5bf9c4-zvxq5       1/1     Running   9          2d13h   10.244.1.136     ubuntu-130   <none>           <none>

因此在kube-system创建nginx的deployment,成功看到了业务日志

手动使用curl触发日志,

<h1>Welcome to nginx!</h1>
<p>If you see this page, the nginx web server is successfully installed and
working. Further configuration is required.</p><p>For online documentation and support please refer to
<a href="http://nginx.org/">nginx.org</a>.<br/>
Commercial support is available at
<a href="http://nginx.com/">nginx.com</a>.</p><p><em>Thank you for using nginx.</em></p>
</body>
</html>
root@ubuntu-130:~# curl http://10.244.1.149:80
<!DOCTYPE html>
<html>
<head>
<title>Welcome to nginx!</title>
<style>body {width: 35em;margin: 0 auto;font-family: Tahoma, Verdana, Arial, sans-serif;}
</style>
</head>
<body>
<h1>Welcome to nginx!</h1>
<p>If you see this page, the nginx web server is successfully installed and
working. Further configuration is required.</p><p>For online documentation and support please refer to
<a href="http://nginx.org/">nginx.org</a>.<br/>
Commercial support is available at
<a href="http://nginx.com/">nginx.com</a>.</p><p><em>Thank you for using nginx.</em></p>
</body>
</html>
http://192.168.13.1:9200/fluentd/_doc/upWKnWwBQIky4GgXOsKV
+——————————————————————————————————————————————————————{"_index": "fluentd","_type": "_doc","_id": "upWKnWwBQIky4GgXOsKV","_version": 1,"found": true,"_source": {"stream": "stdout","docker": {"container_id": "77e1bccb80a2470aeca60e9a376bb832935e53ee676292cf50b93a94e9417ffa"},"kubernetes": {"container_name": "nginx","namespace_name": "kube-system","pod_name": "nginx-deployment-76bf4969df-7h5mw","container_image": "nginx:1.7.9","container_image_id": "docker-pullable://nginx@sha256:e3456c851a152494c3e4ff5fcc26f240206abac0c9d794affb40e0714846c451","pod_id": "969d8555-c099-11e9-a6da-000c295a7d67","host": "ubuntu-130","labels": {"app": "nginx","pod-template-hash": "76bf4969df"},"master_url": "https://10.96.0.1:443/api","namespace_id": "3ef5b06e-bd0f-11e9-b463-000c295a7d67"},"message": "10.244.1.1 - - [17/Aug/2019:03:06:15 +0000] \"GET / HTTP/1.1\" 200 612 \"-\" \"curl/7.47.0\" \"-\"\n","tag": "kubernetes.var.log.containers.nginx-deployment-76bf4969df-7h5mw_kube-system_nginx-77e1bccb80a2470aeca60e9a376bb832935e53ee676292cf50b93a94e9417ffa.log"}
}

精准范围查询

POST http://192.168.13.1:9200/fluentd/_search
_______________________________________________
{"query":{"query_string":{"query":"kubernetes.var.log.containers.nginx-deployment-76bf4969df-7h5mw_kube-system_nginx-77e1bccb80a2470aeca60e9a376bb832935e53ee676292cf50b93a94e9417ffa.log"}},"from":0,"size":10
}
——————————————————————————————————————————————————————
{"took": 69,"timed_out": false,"_shards": {"total": 5,"successful": 5,"skipped": 0,"failed": 0},"hits": {"total": 3,"max_score": 23.224003,"hits": [{"_index": "fluentd","_type": "_doc","_id": "upWKnWwBQIky4GgXOsKV","_score": 23.224003,"_source": {"stream": "stdout","docker": {"container_id": "77e1bccb80a2470aeca60e9a376bb832935e53ee676292cf50b93a94e9417ffa"},"kubernetes": {"container_name": "nginx","namespace_name": "kube-system","pod_name": "nginx-deployment-76bf4969df-7h5mw","container_image": "nginx:1.7.9","container_image_id": "docker-pullable://nginx@sha256:e3456c851a152494c3e4ff5fcc26f240206abac0c9d794affb40e0714846c451","pod_id": "969d8555-c099-11e9-a6da-000c295a7d67","host": "ubuntu-130","labels": {"app": "nginx","pod-template-hash": "76bf4969df"},"master_url": "https://10.96.0.1:443/api","namespace_id": "3ef5b06e-bd0f-11e9-b463-000c295a7d67"},"message": "10.244.1.1 - - [17/Aug/2019:03:06:15 +0000] \"GET / HTTP/1.1\" 200 612 \"-\" \"curl/7.47.0\" \"-\"\n","tag": "kubernetes.var.log.containers.nginx-deployment-76bf4969df-7h5mw_kube-system_nginx-77e1bccb80a2470aeca60e9a376bb832935e53ee676292cf50b93a94e9417ffa.log"}},{"_index": "fluentd","_type": "_doc","_id": "u5WKnWwBQIky4GgXOsKV","_score": 22.271074,"_source": {"stream": "stdout","docker": {"container_id": "77e1bccb80a2470aeca60e9a376bb832935e53ee676292cf50b93a94e9417ffa"},"kubernetes": {"container_name": "nginx","namespace_name": "kube-system","pod_name": "nginx-deployment-76bf4969df-7h5mw","container_image": "nginx:1.7.9","container_image_id": "docker-pullable://nginx@sha256:e3456c851a152494c3e4ff5fcc26f240206abac0c9d794affb40e0714846c451","pod_id": "969d8555-c099-11e9-a6da-000c295a7d67","host": "ubuntu-130","labels": {"app": "nginx","pod-template-hash": "76bf4969df"},"master_url": "https://10.96.0.1:443/api","namespace_id": "3ef5b06e-bd0f-11e9-b463-000c295a7d67"},"message": "10.244.1.1 - - [17/Aug/2019:03:06:16 +0000] \"GET / HTTP/1.1\" 200 612 \"-\" \"curl/7.47.0\" \"-\"\n","tag": "kubernetes.var.log.containers.nginx-deployment-76bf4969df-7h5mw_kube-system_nginx-77e1bccb80a2470aeca60e9a376bb832935e53ee676292cf50b93a94e9417ffa.log"}},{"_index": "fluentd","_type": "_doc","_id": "vJWKnWwBQIky4GgXOsKV","_score": 22.271074,"_source": {"stream": "stdout","docker": {"container_id": "77e1bccb80a2470aeca60e9a376bb832935e53ee676292cf50b93a94e9417ffa"},"kubernetes": {"container_name": "nginx","namespace_name": "kube-system","pod_name": "nginx-deployment-76bf4969df-7h5mw","container_image": "nginx:1.7.9","container_image_id": "docker-pullable://nginx@sha256:e3456c851a152494c3e4ff5fcc26f240206abac0c9d794affb40e0714846c451","pod_id": "969d8555-c099-11e9-a6da-000c295a7d67","host": "ubuntu-130","labels": {"app": "nginx","pod-template-hash": "76bf4969df"},"master_url": "https://10.96.0.1:443/api","namespace_id": "3ef5b06e-bd0f-11e9-b463-000c295a7d67"},"message": "10.244.1.1 - - [17/Aug/2019:03:06:17 +0000] \"GET / HTTP/1.1\" 200 612 \"-\" \"curl/7.47.0\" \"-\"\n","tag": "kubernetes.var.log.containers.nginx-deployment-76bf4969df-7h5mw_kube-system_nginx-77e1bccb80a2470aeca60e9a376bb832935e53ee676292cf50b93a94e9417ffa.log"}}]}
}

k8s搭建fluentd+ES相关推荐

  1. centos7 如何安装部署k8s_五步教你如何使用k8s快速部署ES

    前言 今天小编打算用五步教大家如何使用k8s快速部署ES,有兴趣的小伙伴可以了解一下~ 由于是使用本地存储,所以需要先创建pv 1.创建存储类 local-elasticsearch.yaml kin ...

  2. 2019最新k8s集群搭建教程 (centos k8s 搭建)

    2019-k8s-centos 2019最新k8s集群搭建教程 (centos k8s 搭建) 网上全是要么过时的,要么残缺的,大多数都是2016年,2017年的文档,照着尝试了N次,各种卸了重装,最 ...

  3. k8s部署tomcat及web应用_k8s介绍 k8s搭建一个应用(mysql+tomcat)

    k8s介绍 k8s搭建一个应用(mysql+tomcat) kubetnetes介绍 官网 https://kubernetes.io/ Kubernetes也就是k8s Kubernetes是一个开 ...

  4. K8S搭建自动化部署环境(三)Jenkins下载、安装和启动

    各位大佬,前文如下: K8S搭建自动化部署环境(一)安装Kubernetes K8S搭建自动化部署环境(二)安装K8S管理工具Kuboard V3 一.jenkins 下载 jenkins下载地址:h ...

  5. k8s搭建--裸机搭建(Bare Metal)

    本篇文章从新建虚拟机开始,教大家在linux系统中的k8s搭建,博主本人搭建了三个节点,一个master和两个node节点,其实一个主节点一个工作节点就足够了,搭建过程供大家参考. 第一步,新建虚拟机 ...

  6. 一文学会 PC搭建opengl es C++开发环境

    Windows搭建opengl es C++开发环境 之前编写Native的OpenGL ES实现代码时,使用的编译工具无论是Android Studio还是eclipse都给我一种:用txt文档中写 ...

  7. 《Centos7——k8s搭建》

    目录 k8s搭建 1.关闭防火墙 2.时间同步 3.修改主机名(三台分别修改) 4. 编辑hosts文件 5: 安装指定版本的docker 6:运行docker 7:docker镜像加速 8:上传k8 ...

  8. K8S搭建自动化部署环境(四)Jenkins多分支流水线Blue Ocean的安装和使用

    各位大佬,前文如下: K8S搭建自动化部署环境(一)安装Kubernetes K8S搭建自动化部署环境(二)安装K8S管理工具Kuboard V3 K8S搭建自动化部署环境(三)Jenkins下载.安 ...

  9. k8s搭建部署--三个节点服务器(超详细)

    目录 k8s搭建部署--三个节点服务器 docker环境准备 安装kubeadm,kubelet和kubectl Centos添加kubernetes YUM软件源 ubuntu修改源 master主 ...

最新文章

  1. mvc 职能划分_【转】MVC中的操作如何分类(MVC层次的划分)
  2. 第十六届全国大学生智能车| AI视觉组新手入门教程
  3. SpringMVC中重定向
  4. Ubuntu:Ubuntu下安装Anaconda和Tensorflow的简介、入门、安装流程之详细攻略
  5. zlib 加密 java_JAVA自带Zlib+base64位压缩加密和解压解密
  6. mongodb插入数据_MongoDB插入
  7. Ubuntu下安装Chrome浏览器的方法
  8. cass连接复合线命令 lisp_南方CASS和AutoCAD快捷命令大全
  9. QT版用QLCDnumber显示时间
  10. 一文详解企业数据分类分级的推进路径
  11. 计算机基础知识面试题集合(包含计网OSI、TCP/IP、HTTP、TCP、UDP、三次握手、四次挥手、OS进程线程、死锁,常见数据结构及排序,Linux常用命令、数据库基础等。)
  12. 空气质量等级c语言编程,关于SDS011模块(空气中pm2.5及pm10)单片机c程序实现(链接附源码)...
  13. G1调优实践日记--G1HeapWastePercent和InitiatingHeapOccupancyPercent的应用
  14. 直播互动软件神器有哪些
  15. 与传统招聘方式相比,小程序招聘都有哪些优势?
  16. Java使用JAVE获取MP4播放时长
  17. linux终端 中文是数字,Linux中man命令及之后的数字
  18. ibm3500服务器型号,IBM x365 M2服务器的详细配置.doc
  19. 鱼眼摄像头的畸变矫正方法-python+opencv
  20. 中国联通2g信号停止服务器,中国联通关闭2G/3G服务?官方回应来了

热门文章

  1. java使用grok解析日志文件
  2. SWUST OJ 617: 班级课程成绩计算
  3. 计算机风景画作品,电脑风景画图片大全
  4. DB2 SQLSTATE:428C9,不能将ROWID列指定为INSERT或UPDATE的目标列。
  5. python判断火车票座位代码_12306 火车票监控Python代码
  6. PNG背景透明 转成 BMP背景透明
  7. Windows11 开机自动同步时间(开机时间不更新问题)
  8. sd卡座封装尺寸_SD卡座结构图及使用注意说明
  9. 闲谈JVM(二):浅析新老生代参数配置
  10. Cuda CuDnn Pytorch 安装 print(torch.cuda.is_available())输出报错。NVDIA 940MX