Scenario Description
A user collects logs to CLS (Cloud Log Service) in a single-line text format. The user now wants to structure the logs by extracting information such as log time, log level, and requestid from the text to facilitate subsequent search and analysis.
Scenario Analysis
The user's processing requirements are analyzed, and the processing approach is as follows:
1. The contents in {...} are the details of operations, which can be extracted by the regular expression.
2. Use the regular expression to extract log time, log level, URL.
Raw Log
{
"content": "[2021-11-24 11:11:08,232][328495eb-b562-478f-9d5d-3bf7e][INFO] curl -H 'Host: ' http://abc.com:8080/pc/api -d {\\"version\\": \\"1.0\\",\\"user\\": \\"CGW\\",\\"password\\": \\"123\\"}"
}
Processing Result
{
"level":"INFO",
"password":"123",
"requestid":"328495eb-b562-478f-9d5d-3bf7e",
"time":"2021-11-24 11:11:08,232",
"user":"CGW",
"version":"1.0"
}
DSL Processing Function
ext_regex("content", regex="\\[(\\d{4}-\\d{2}-\\d{2} \\d{2}:\\d{2}:\\d{2},\\d{3})\\]\\[(\\w+-\\w+-\\w+-\\w+-\\w+)\\]\\[([A-Z]{4})\\].*(\\{[^\\}]+\\})", output="time,requestid,level,msg")
ext_json("msg")
fields_drop("content","msg")
DSL Processing Function Overview
1. Use the regular key-value extraction function ext_regex to extract information such as time, request, log level, and message, and save the relevant content to the time, requestid, level, and msg fields. ext_regex("content", regex="\\[(\\d{4}-\\d{2}-\\d{2} \\d{2}:\\d{2}:\\d{2},\\d{3})\\]\\[(\\w+-\\w+-\\w+-\\w+-\\w+)\\]\\[([A-Z]{4})\\].*(\\{[^\\}]+\\})", output="time,requestid,level,msg")
2. Use ext_json to flatten the msg field. The value of msg is {\\"version\\": \\"1.0\\",\\"user\\": \\"CGW\\",\\"password\\": \\"123\\"}. 3. Discard the content and msg fields.
fields_drop("content","msg")