流程图

Nginx操作

添加Nginx日志格式json

yum安装的Nginx默认此配置文件
vim /etc/nginx/nginx.conf

    log_format  json '{'
                     '"timestamp":"$time_local",'
                     '"remote_addr":"$remote_addr",'
                     '"request":"$request",'
                     '"status":$status,'
                     '"body_bytes_sent":$body_bytes_sent,'
                     '"http_referer":"$http_referer",'
                     '"http_user_agent":"$http_user_agent",'
                     '"http_x_forwarded_for":"$http_x_forwarded_for",'
                     '"up_addr":"$upstream_addr",'
                     '"up_host":"$upstream_http_host",'
                     '"upstream_time":"$upstream_response_time",'
                     '"request_time":"$request_time"'
                     '}';

    access_log  /var/log/nginx/access.log  json;

重新加载

systemctl reload nginx

测试访问查看日志

curl - I 172.16.49.130

[root@002 nginx]# tail -1 /var/log/nginx/access.log
{"timestamp":"14/May/2024:15:22:48 +0800","remote_addr":"172.16.49.1","request":"GET / HTTP/1.1","status":304,"body_bytes_sent":0,"http_referer":"-","http_user_agent":"Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/124.0.0.0 Safari/537.36","http_x_forwarded_for":"-","up_addr":"-","up_host":"-","upstream_time":"-","request_time":"0.000"}

安装jq

jq可以格式化Json格式数据，方便查看

yum install jq -y

如果显示没有可用软件包 jq，请执行下边操作
yum install epel-release

[root@002 yum.repos.d]# tail -1 /var/log/nginx/access.log | jq
{
  "timestamp": "14/May/2024:15:22:48 +0800",
  "remote_addr": "172.16.49.1",
  "request": "GET / HTTP/1.1",
  "status": 304,
  "body_bytes_sent": 0,
  "http_referer": "-",
  "http_user_agent": "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/124.0.0.0 Safari/537.36",
  "http_x_forwarded_for": "-",
  "up_addr": "-",
  "up_host": "-",
  "upstream_time": "-",
  "request_time": "0.000"
}

环境清理

1、之前有一些不是json格式的日志数据，以免产生干扰全都清除，保证一个纯净的环境

>/var/log/nginx/access.log

2、Kibana清除之前索引

Filebeat操作

配置文件

cat > /etc/filebeat/filebeat.yml << 'EOF'
filebeat.inputs:
- type: log
  enabled: true
  paths:
    - /var/log/nginx/access.log
  json.keys_under_root: true
  json.overwrite_keys: true

output.elasticsearch:
  hosts: ["172.16.49.130:9200"]
  index: "nginx-%{[agent.version]}-%{+yyyy.MM}"

setup.ilm.enabled: false
setup.template.enabled: false
EOF

重启

systemctl restart filebeat
systemctl status filebeat

查看es数据

此时访问Nginx产生新日志信息，访问ES即是json格式数据，解决了上篇中的思考问题。

Kibana操作

再去Kibana添加新的索引，此时查看log已经细分化了