clickhouse导入原始nginx日志数据

举报
仙士可 发表于 2023/06/30 12:29:20 2023/06/30
【摘要】 原始日志数据139.224.56.94 - - [03/Dec/2022:16:01:01 +0800] "GET /output/zw/updateConsumeMoney?ad_num=1740763056491527&date=2022-12-03&cost=0&balance=0&sign=09a77e8f235f45b087a2d15a1b77f430 HTTP/1.0" 200 ...

原始日志数据

139.224.56.94 - - [03/Dec/2022:16:01:01 +0800] "GET /output/zw/updateConsumeMoney?ad_num=1740763056491527&date=2022-12-03&cost=0&balance=0&sign=09a77e8f235f45b087a2d15a1b77f430 HTTP/1.0" 200 20 "-" "-"
49.93.209.3 - - [03/Dec/2022:16:01:01 +0800] "GET /index/Quickapi1/read?ref=read&id=1029393&bookid=3109 HTTP/2.0" 200 14547 "https://quickapp.cn/com.hmydb.zzxd/29/page-frame.html" "Mozilla/5.0 (Linux; Android 12; ELS-AN00 Build/HUAWEIELS-AN00;)AppleWebKit/537.36 (KHTML, like Gecko) Version/4.0 Chrome/18.0.1025 Mobile Safari/537.36 hap/1102/huawei com.huawei.fastapp/12.6.1.304 com.hmydb.zzxd/2.0.9 ({\x22packageName\x22:\x22unknown\x22,\x22type\x22:\x22unknown\x22,\x22extra\x22:\x22{}\x22})"
139.224.56.94 - - [03/Dec/2022:16:01:02 +0800] "GET /output/zw/updateConsumeMoney?ad_num=1740763057082376&date=2022-12-03&cost=0&balance=0&sign=321a5fc2b7470896ca9e0991b3dcff7c HTTP/1.0" 200 0 "-" "-"
171.214.147.1 - - [03/Dec/2022:16:01:02 +0800] "GET /index/Quickapi1/getGender HTTP/2.0" 200 26 "https://quickapp.cn/com.hmydb.zzxd/29/page-frame.html" "Mozilla/5.0 (Linux; Android 12; NOH-AN00 Build/HUAWEINOH-AN00;)AppleWebKit/537.36 (KHTML, like Gecko) Version/4.0 Chrome/18.0.1025 Mobile Safari/537.36 hap/1102/huawei com.huawei.fastapp/12.6.1.304 com.hmydb.zzxd/2.0.9 ({\x22type\x22:\x22shortcut\x22,\x22packageName\x22:\x22com.huawei.android.launcher\x22,\x22extra\x22:{\x22isNative\x22:false,\x22scene\x22:\x22api\x22}})"
139.224.56.94 - - [03/Dec/2022:16:01:02 +0800] "GET /output/zw/updateConsumeMoney?ad_num=1740763057797133&date=2022-12-03&cost=0&balance=0&sign=fd36a029809d9d4ddf45da2e9544ad69 HTTP/1.0" 200 0 "-" "-"
复制

创建表:

 CREATE TABLE nginx_log (
               remote_addr String,
               time_local String,
               request String,
               status String,
               body_bytes_sent String,
               http_referer String,
               http_user_agent String
           ) ENGINE = Log;
复制

导入命令:

INSERT INTO nginx_log FROM INFILE 'nginx.log'
           SETTINGS
             format_regexp = '([0-9]+\\.[0-9]+\\.[0-9]+\\.[0-9]+) - - \\[([0-9]+\\/[a-z,A-Z]+\\/[0-9]+:[0-9]+:[0-9]+:[0-9]+ \\+[0-9]+)\\] "(.+?)" ([0-9]+) ([0-9]+) "(.+?)" "(.+?)"' , format_regexp_skip_unmatched = 1
           FORMAT Regexp
复制
【版权声明】本文为华为云社区用户原创内容,转载时必须标注文章的来源(华为云社区)、文章链接、文章作者等基本信息, 否则作者和本社区有权追究责任。如果您发现本社区中有涉嫌抄袭的内容,欢迎发送邮件进行举报,并提供相关证据,一经查实,本社区将立刻删除涉嫌侵权内容,举报邮箱: cloudbbs@huaweicloud.com
  • 点赞
  • 收藏
  • 关注作者

评论(0

0/1000
抱歉,系统识别当前为高风险访问,暂不支持该操作

全部回复

上滑加载中

设置昵称

在此一键设置昵称,即可参与社区互动!

*长度不超过10个汉字或20个英文字符,设置后3个月内不可修改。

*长度不超过10个汉字或20个英文字符,设置后3个月内不可修改。