一文记录Telegraf解析nginx日志

文档记录Telegraf解析nginx access.log的相关内容

Tail Input Plugin

Tail Input Plugin 是 Telegraf项目 众多Inputs插件之一。Tail Input Plugin 监听一个日志文件并解析该文件的每个日志变更。默认情况下,tail 插件的行为类似于以下 Unix tail 命令

tail -F –lines=0 myfile.log

Telegraf收集nginx access.log 相当于Unix tail命令:

tail -F –lines=0 access.log

并使用grok格式来解析日志。除此之外,Telegraf Tail Input Plugin支持多种 Data Format细节东西可以跟下源码,下面直接记录运行配置。

配置

telegraf.conf

[agent] interval = “10s” round_interval = true metric_batch_size = 1000 metric_buffer_limit = 10000 collection_jitter = “0s” flush_interval = “10s” flush_jitter = “0s” precision = “” hostname = “” omit_hostname = false[[inputs.tail]] files = [“/Users/enic/github/telegraf/testdata/access.log”]# from_beginning = false from_beginning = true grok_patterns = [“%{COMMON_LOG_FORMAT}”] name_override = “nginx_access_log” grok_custom_pattern_files = [] grok_custom_patterns = ”’ NGINX_ACCESS_LOG %{IP:remote_addr} – (-|%{WORD:remote_user}) [%{HTTPDATE:time_local}] %{QS:request} %{NUMBER:status:int} %{NUMBER:body_bytes_sent:int} %{QS:referrer} %{QS:agent} %{IPORHOST:xforwardedfor} %{IPORHOST:host} %{QS:upstream_addr} (-|%{NUMBER:upstream_status:int}) %{BASE10NUM:request_time:float} (-|%{BASE10NUM:upstream_response_time:float}) %{QS:cookie_uin} %{QS:cookie_luin} ”’ grok_timezone = “Local” data_format = “grok”[[outputs.file]] files = [“/Users/enic/github/telegraf/testdata/outputs.log”]

access.log

10.10.10.10 – – [19/Apr/2022:03:17:34 +0800] “POST /user/login HTTP/1.1” 400 63 “-” “Go-http-client/1.1” “-“10.10.10.10 – – [19/Apr/2022:03:17:34 +0800] “POST /user/login HTTP/1.1” 400 63 “-” “Go-http-client/1.1” “-“10.10.10.10 – – [19/Apr/2022:03:17:34 +0800] “POST /user/login HTTP/1.1” 400 63 “-” “Go-http-client/1.1” “-“10.10.10.10 – – [19/Apr/2022:03:17:34 +0800] “POST /user/login HTTP/1.1” 400 63 “-” “Go-http-client/1.1” “-“10.10.10.10 – – [19/Apr/2022:03:17:34 +0800] “POST /user/login HTTP/1.1” 400 63 “-” “Go-http-client/1.1” “-“10.10.10.10 – – [19/Apr/2022:03:17:34 +0800] “POST /user/login HTTP/1.1” 400 63 “-” “Go-http-client/1.1” “-“10.10.10.10 – – [19/Apr/2022:03:17:44 +0800] “POST /qps HTTP/1.1” 504 3693 “-” “Go-http-client/1.1” “-“10.10.10.10 – – [19/Apr/2022:03:17:54 +0800] “POST /latency HTTP/1.1” 504 3693 “-” “Go-http-client/1.1” “-“10.10.10.10 – – [19/Apr/2022:03:18:34 +0800] “POST /user/login HTTP/1.1” 400 63 “-” “Go-http-client/1.1” “-“10.10.10.10 – – [19/Apr/2022:03:18:34 +0800] “POST /user/login HTTP/1.1” 400 63 “-” “Go-http-client/1.1” “-“10.10.10.10 – – [19/Apr/2022:03:18:34 +0800] “POST /user/login HTTP/1.1” 400 63 “-” “Go-http-client/1.1” “-“10.10.10.10 – – [19/Apr/2022:03:18:34 +0800] “POST /user/login HTTP/1.1” 400 63 “-” “Go-http-client/1.1” “-“10.10.10.10 – – [19/Apr/2022:03:18:34 +0800] “POST /user/login HTTP/1.1” 400 63 “-” “Go-http-client/1.1” “-“10.10.10.10 – – [19/Apr/2022:03:18:34 +0800] “POST /user/login HTTP/1.1” 400 63 “-” “Go-http-client/1.1” “-“10.10.10.100 – – [19/Apr/2022:03:18:37 +0800] “GET / HTTP/1.1” 200 1839 “-” “Expanse, a Palo Alto Networks company, searches across the global IPv4 space multiple times per day to identify customers’ presences on the Internet. If you would like to be excluded from our scans, please send IP addresses/domains to: scaninfo@paloaltonetworks.com” “-“10.10.10.10 – – [19/Apr/2022:03:18:44 +0800] “POST /qps HTTP/1.1” 504 3693 “-” “Go-http-client/1.1” “-“10.10.10.10 – – [19/Apr/2022:03:18:54 +0800] “POST /latency HTTP/1.1” 504 3693 “-” “Go-http-client/1.1” “-“10.10.10.10 – – [19/Apr/2022:03:19:34 +0800] “POST /user/login HTTP/1.1” 400 63 “-” “Go-http-client/1.1” “-“10.10.10.10 – – [19/Apr/2022:03:19:34 +0800] “POST /user/login HTTP/1.1” 400 63 “-” “Go-http-client/1.1” “-“10.10.10.10 – – [19/Apr/2022:03:19:34 +0800] “POST /user/login HTTP/1.1” 400 63 “-” “Go-http-client/1.1” “-“10.10.10.10 – – [19/Apr/2022:03:19:34 +0800] “POST /user/login HTTP/1.1” 400 63 “-” “Go-http-client/1.1” “-“10.10.10.10 – – [19/Apr/2022:03:18:34 +0800] “POST /user/login HTTP/1.1” 400 63 “-” “Go-http-client/1.1” “-“10.10.10.10 – – [19/Apr/2022:03:18:34 +0800] “POST /user/login HTTP/1.1” 400 63 “-” “Go-http-client/1.1” “-“10.10.10.10 – – [19/Apr/2022:03:18:34 +0800] “POST /user/login HTTP/1.1” 400 63 “-” “Go-http-client/1.1” “-“10.10.10.10 – – [19/Apr/2022:03:18:34 +0800] “POST /user/login HTTP/1.1” 400 63 “-” “Go-http-client/1.1” “-“10.10.10.10 – – [19/Apr/2022:03:19:34 +0800] “POST /user/login HTTP/1.1” 400 63 “-” “Go-http-client/1.1” “-“10.10.10.10 – – [19/Apr/2022:03:19:34 +0800] “POST /user/login HTTP/1.1” 400 63 “-” “Go-http-client/1.1” “-“10.10.10.10 – – [19/Apr/2022:03:19:34 +0800] “POST /user/login HTTP/1.1” 400 63 “-” “Go-http-client/1.1” “-“10.10.10.10 – – [19/Apr/2022:03:19:34 +0800] “POST /user/login HTTP/1.1” 400 63 “-” “Go-http-client/1.1” “-“10.10.10.10 – – [19/Apr/2022:03:18:34 +0800] “POST /user/login HTTP/1.1” 400 63 “-” “Go-http-client/1.1” “-“10.10.10.10 – – [19/Apr/2022:03:18:34 +0800] “POST /user/login HTTP/1.1” 400 63 “-” “Go-http-client/1.1” “-“10.10.10.10 – – [19/Apr/2022:03:18:34 +0800] “POST /user/login HTTP/1.1” 400 63 “-” “Go-http-client/1.1” “-“10.10.10.10 – – [19/Apr/2022:03:18:34 +0800] “POST /user/login HTTP/1.1” 400 63 “-” “Go-http-client/1.1” “-“10.10.10.10 – – [19/Apr/2022:03:18:34 +0800] “POST /user/login HTTP/1.1” 400 63 “-” “Go-http-client/1.1” “-“10.10.10.10 – – [19/Apr/2022:03:18:34 +0800] “POST /user/login HTTP/1.1” 400 63 “-” “Go-http-client/1.1” “-“10.10.10.10 – – [19/Apr/2022:03:18:34 +0800] “POST /user/login HTTP/1.1” 400 63 “-” “Go-http-client/1.1” “-“10.10.10.10 – – [19/Apr/2022:03:18:34 +0800] “POST /user/login HTTP/1.1” 400 63 “-” “Go-http-client/1.1” “-“10.10.10.10 – – [19/Apr/2022:03:18:34 +0800] “POST /user/login HTTP/1.1” 400 63 “-” “Go-http-client/1.1” “-“

操作

cd telegrafmake./telegraf –config=/Users/enic/github/telegraf/testdata/telegraf.conf

output展示

nginx_access_log,host=enicMBP,path=/Users/enic/github/telegraf/testdata/access.log,resp_code=400,verb=POST request=”/user/login”,http_version=1.1,ident=”-“,auth=”-“,resp_bytes=63i,client_ip=”10.10.10.10″ 1650309454000000000nginx_access_log,host=enicMBP,path=/Users/enic/github/telegraf/testdata/access.log,resp_code=400,verb=POST request=”/user/login”,resp_bytes=63i,client_ip=”10.10.10.10″,ident=”-“,auth=”-“,http_version=1.1 1650309454001000000nginx_access_log,host=enicMBP,path=/Users/enic/github/telegraf/testdata/access.log,resp_code=400,verb=POST ident=”-“,request=”/user/login”,auth=”-“,http_version=1.1,resp_bytes=63i,client_ip=”10.10.10.10″ 1650309454002000000nginx_access_log,host=enicMBP,path=/Users/enic/github/telegraf/testdata/access.log,resp_code=400,verb=POST http_version=1.1,client_ip=”10.10.10.10″,ident=”-“,resp_bytes=63i,auth=”-“,request=”/user/login” 1650309454003000000nginx_access_log,host=enicMBP,path=/Users/enic/github/telegraf/testdata/access.log,resp_code=400,verb=POST client_ip=”10.10.10.10″,auth=”-“,ident=”-“,request=”/user/login”,http_version=1.1,resp_bytes=63i 1650309454004000000nginx_access_log,host=enicMBP,path=/Users/enic/github/telegraf/testdata/access.log,resp_code=400,verb=POST ident=”-“,auth=”-“,request=”/user/login”,resp_bytes=63i,client_ip=”10.10.10.10″,http_version=1.1 1650309454005000000nginx_access_log,host=enicMBP,path=/Users/enic/github/telegraf/testdata/access.log,resp_code=504,verb=POST auth=”-“,request=”/qps”,http_version=1.1,resp_bytes=3693i,client_ip=”10.10.10.10″,ident=”-” 1650309464000000000nginx_access_log,host=enicMBP,path=/Users/enic/github/telegraf/testdata/access.log,resp_code=504,verb=POST ident=”-“,auth=”-“,request=”/latency”,http_version=1.1,resp_bytes=3693i,client_ip=”10.10.10.10″ 1650309474000000000nginx_access_log,host=enicMBP,path=/Users/enic/github/telegraf/testdata/access.log,resp_code=400,verb=POST auth=”-“,request=”/user/login”,resp_bytes=63i,client_ip=”10.10.10.10″,http_version=1.1,ident=”-” 1650309514000000000nginx_access_log,host=enicMBP,path=/Users/enic/github/telegraf/testdata/access.log,resp_code=400,verb=POST ident=”-“,auth=”-“,request=”/user/login”,client_ip=”10.10.10.10″,resp_bytes=63i,http_version=1.1 1650309514001000000nginx_access_log,host=enicMBP,path=/Users/enic/github/telegraf/testdata/access.log,resp_code=400,verb=POST resp_bytes=63i,client_ip=”10.10.10.10″,ident=”-“,auth=”-“,request=”/user/login”,http_version=1.1 1650309514002000000nginx_access_log,host=enicMBP,path=/Users/enic/github/telegraf/testdata/access.log,resp_code=400,verb=POST request=”/user/login”,resp_bytes=63i,ident=”-“,auth=”-“,http_version=1.1,client_ip=”10.10.10.10″ 1650309514003000000nginx_access_log,host=enicMBP,path=/Users/enic/github/telegraf/testdata/access.log,resp_code=400,verb=POST client_ip=”10.10.10.10″,auth=”-“,ident=”-“,request=”/user/login”,http_version=1.1,resp_bytes=63i 1650309514004000000nginx_access_log,host=enicMBP,path=/Users/enic/github/telegraf/testdata/access.log,resp_code=400,verb=POST resp_bytes=63i,ident=”-“,request=”/user/login”,client_ip=”10.10.10.10″,auth=”-“,http_version=1.1 1650309514005000000nginx_access_log,host=enicMBP,path=/Users/enic/github/telegraf/testdata/access.log,resp_code=200,verb=GET ident=”-“,request=”/”,resp_bytes=1839i,auth=”-“,http_version=1.1,client_ip=”10.10.10.100″ 1650309517000000000nginx_access_log,host=enicMBP,path=/Users/enic/github/telegraf/testdata/access.log,resp_code=504,verb=POST request=”/qps”,auth=”-“,ident=”-“,http_version=1.1,resp_bytes=3693i,client_ip=”10.10.10.10″ 1650309524000000000nginx_access_log,host=enicMBP,path=/Users/enic/github/telegraf/testdata/access.log,resp_code=504,verb=POST resp_bytes=3693i,auth=”-“,request=”/latency”,http_version=1.1,client_ip=”10.10.10.10″,ident=”-” 1650309534000000000nginx_access_log,host=enicMBP,path=/Users/enic/github/telegraf/testdata/access.log,resp_code=400,verb=POST http_version=1.1,resp_bytes=63i,auth=”-“,client_ip=”10.10.10.10″,ident=”-“,request=”/user/login” 1650309574000000000nginx_access_log,host=enicMBP,path=/Users/enic/github/telegraf/testdata/access.log,resp_code=400,verb=POST ident=”-“,auth=”-“,resp_bytes=63i,client_ip=”10.10.10.10″,request=”/user/login”,http_version=1.1 1650309574001000000nginx_access_log,host=enicMBP,path=/Users/enic/github/telegraf/testdata/access.log,resp_code=400,verb=POST ident=”-“,request=”/user/login”,auth=”-“,http_version=1.1,resp_bytes=63i,client_ip=”10.10.10.10″ 1650309574002000000nginx_access_log,host=enicMBP,path=/Users/enic/github/telegraf/testdata/access.log,resp_code=400,verb=POST auth=”-“,request=”/user/login”,client_ip=”10.10.10.10″,ident=”-“,http_version=1.1,resp_bytes=63i 1650309574003000000nginx_access_log,host=enicMBP,path=/Users/enic/github/telegraf/testdata/access.log,resp_code=400,verb=POST http_version=1.1,resp_bytes=63i,client_ip=”10.10.10.10″,auth=”-“,request=”/user/login”,ident=”-” 1650309514000000000nginx_access_log,host=enicMBP,path=/Users/enic/github/telegraf/testdata/access.log,resp_code=400,verb=POST ident=”-“,auth=”-“,client_ip=”10.10.10.10″,request=”/user/login”,http_version=1.1,resp_bytes=63i 1650309514001000000nginx_access_log,host=enicMBP,path=/Users/enic/github/telegraf/testdata/access.log,resp_code=400,verb=POST http_version=1.1,client_ip=”10.10.10.10″,request=”/user/login”,resp_bytes=63i,auth=”-“,ident=”-” 1650309514002000000nginx_access_log,host=enicMBP,path=/Users/enic/github/telegraf/testdata/access.log,resp_code=400,verb=POST resp_bytes=63i,ident=”-“,request=”/user/login”,http_version=1.1,client_ip=”10.10.10.10″,auth=”-” 1650309514003000000nginx_access_log,host=enicMBP,path=/Users/enic/github/telegraf/testdata/access.log,resp_code=400,verb=POST resp_bytes=63i,client_ip=”10.10.10.10″,ident=”-“,auth=”-“,request=”/user/login”,http_version=1.1 1650309574000000000nginx_access_log,host=enicMBP,path=/Users/enic/github/telegraf/testdata/access.log,resp_code=400,verb=POST client_ip=”10.10.10.10″,ident=”-“,auth=”-“,resp_bytes=63i,request=”/user/login”,http_version=1.1 1650309574001000000nginx_access_log,host=enicMBP,path=/Users/enic/github/telegraf/testdata/access.log,resp_code=400,verb=POST client_ip=”10.10.10.10″,resp_bytes=63i,ident=”-“,auth=”-“,request=”/user/login”,http_version=1.1 1650309574002000000nginx_access_log,host=enicMBP,path=/Users/enic/github/telegraf/testdata/access.log,resp_code=400,verb=POST resp_bytes=63i,client_ip=”10.10.10.10″,ident=”-“,auth=”-“,request=”/user/login”,http_version=1.1 1650309574003000000nginx_access_log,host=enicMBP,path=/Users/enic/github/telegraf/testdata/access.log,resp_code=400,verb=POST request=”/user/login”,resp_bytes=63i,client_ip=”10.10.10.10″,auth=”-“,http_version=1.1,ident=”-” 1650309514000000000nginx_access_log,host=enicMBP,path=/Users/enic/github/telegraf/testdata/access.log,resp_code=400,verb=POST auth=”-“,http_version=1.1,resp_bytes=63i,client_ip=”10.10.10.10″,request=”/user/login”,ident=”-” 1650309514001000000nginx_access_log,host=enicMBP,path=/Users/enic/github/telegraf/testdata/access.log,resp_code=400,verb=POST client_ip=”10.10.10.10″,auth=”-“,resp_bytes=63i,ident=”-“,request=”/user/login”,http_version=1.1 1650309514002000000nginx_access_log,host=enicMBP,path=/Users/enic/github/telegraf/testdata/access.log,resp_code=400,verb=POST auth=”-“,resp_bytes=63i,request=”/user/login”,http_version=1.1,client_ip=”10.10.10.10″,ident=”-” 1650309514003000000nginx_access_log,host=enicMBP,path=/Users/enic/github/telegraf/testdata/access.log,resp_code=400,verb=POST request=”/user/login”,resp_bytes=63i,client_ip=”10.10.10.10″,ident=”-“,http_version=1.1,auth=”-” 1650309514004000000nginx_access_log,host=enicMBP,path=/Users/enic/github/telegraf/testdata/access.log,resp_code=400,verb=POST request=”/user/login”,client_ip=”10.10.10.10″,http_version=1.1,resp_bytes=63i,ident=”-“,auth=”-” 1650309514005000000nginx_access_log,host=enicMBP,path=/Users/enic/github/telegraf/testdata/access.log,resp_code=400,verb=POST request=”/user/login”,http_version=1.1,resp_bytes=63i,client_ip=”10.10.10.10″,ident=”-“,auth=”-” 1650309514006000000nginx_access_log,host=enicMBP,path=/Users/enic/github/telegraf/testdata/access.log,resp_code=400,verb=POST ident=”-“,request=”/user/login”,resp_bytes=63i,client_ip=”10.10.10.10″,auth=”-“,http_version=1.1 1650309514007000000

郑重声明:本文内容及图片均整理自互联网,不代表本站立场,版权归原作者所有,如有侵权请联系管理员(admin#wlmqw.com)删除。
(0)
用户投稿
上一篇 2022年6月25日
下一篇 2022年6月25日

相关推荐

  • 如果当初的那个欧文再成熟一点,他的成就会不会比现在更高?

    说到欧文大家应该是非常了解的,身为状元的他,球风华丽打球观赏性特别的高,还没有进入联盟就已经收获了大批的粉丝。在被骑士选中之后,欧文就一直是球队的当家,但是球队的战绩一直很糟糕,一…

    2022年7月25日
  • 有游戏手机那味儿了,iQOO 10系列游戏体验

    由于工作的关系,已经很久没有好好体验过手机游戏了。于是趁着这次放年假的时间,重拾起了大学时期玩的几款手机游戏,顺便体验一下iQOO刚刚发布的iQOO10系列手机的游戏体验如何。 那…

    2022年7月22日
  • 香港瑜伽网红周慧贤被杀,被砍30多刀,凶手系其无业前男友

    就在7月1日凌晨,有港媒曝光了一则令人痛心的新闻,年仅23岁的香港女网红,瑜伽美女周慧贤被杀,此消息一出,震惊了不少人。 据悉,原本周慧贤还打算在30日凌晨飞往美国进修,可没想到,…

    2022年7月2日
  • 电脑键盘说明使用方法(键盘各个键的功能图解)

    【键盘操作方法大全】键盘可不仅仅能帮我们打字哦,还有很多快捷的操作你都知道吗?除了Ctrl+C、Ctrl+V以外,再多学几种吧,让你用起电脑来十指如飞~别再慢慢用鼠标点了,用开始键…

    2022年4月19日
  • 偶像派还是实力派?-荣耀70 Pro评测

    去年,荣耀凭借着荣耀50和60系列的热销,度过了销量低谷,让荣耀成功触底反弹。要知道,一直以来,荣耀的数字系列都肩负着在主流消费市场中抢占份额的任务。那么这回杀入阵中的荣耀70系列…

    2022年6月18日
  • 明星现实中真的很漂亮?

    许多年前在艺术中心工作,经常会有大牌时尚杂志借用展览场地拍摄明星。 她们确实跟普通人不一样。首先自身气场强大,即使一个人向你走过来,你都能感受到那股劲,自信十足。这是与她们日常工作…

    2022年6月24日
  • 蜻飞蝶舞,用佳能追踪夏日小精灵

    (此处已添加小程序,请到今日头条客户端查看) 图文/三蝶纪 炎炎夏日,万物生长,一年里最适合拍昆虫的季节悄然而至。在秋冬,很多昆虫会蛰伏起来,以各种形式越冬,春季陆续开始出来活动,…

    2022年6月19日
  • Kubernetes(k8s)安装以及搭建k8s-Dashboard详解

    一、前期准备 1)前提条件 三台兼容的 Linux 主机。Kubernetes 项目为基于 Debian 和 Red Hat 的 – – Linux 发行版…

    2022年6月13日
  • Oracle日常维护内容

    检查系统层面。 检查系统磁盘空间使用情况 root#df -h 检查内存使用情况 root#free -m 查看available值 检查CPU使用情况 root#top a)看第…

    2022年6月16日
  • 郭广昌的传奇人生26-携手三一和新希望讲述投资背后的思考

    [鼓掌][鼓掌]亲爱的朋友们大家好,这段时间蜗牛上班有点忙,没有和大家一起阅读。今天蜗牛继续和大家一起来阅读吧![小鼓掌] 复星创始人-郭广昌 [中国赞]郭广昌简介:郭广昌,男,汉…

    2022年8月19日

联系我们

联系邮箱:admin#wlmqw.com
工作时间:周一至周五,10:30-18:30,节假日休息