tencent cloud

Cloud Log Service

Using Separators to Extract Specified Content from Logs

Download
Mode fokus
Ukuran font
Terakhir diperbarui: 2026-05-11 17:59:09

Use Case

Scenario 1: User logs are delimited using the vertical bar (|). You now need to extract structured fields from them.
Scenario 2: A user has collected the logs generated by Flink tasks to CLS in single-line text format. The logs contain a JSON segment, which is a package returned by Flink. The user wants to structuralize this JSON.

Scenario 1

Raw Log

{
"message": "2021-12-09 11:34:28.279|INFO|605c643e29e4|BIN--Python|192.168.1.1"
}

Processing Statement

//Use the vertical bar (|) as the delimiter (the vertical bar needs to be escaped) to extract the time, loglevel, taskId, ProcessName, and ip fields from the message.
ext_sepstr("message","time,loglevel,taskId,ProcessName,ip",sep="\\|")
// Discard the message.
fields_drop("message")

Processing Result

{
"ProcessName":"BIN--Python",
"ip":"192.168.1.1",
"loglevel":"INFO",
"taskId":"605c643e29e4",
"time":"2021-12-09 11:34:28.279"
}

Scenario 2

Raw Log

{
"regex": "2021-12-02 14:33:35.022 [1] INFO org.apache.Load - Response:status: 200, resp msg: OK, resp content: { \\"TxnId\\": 58322, \\"Label\\": \\"flink_connector_20211202_1de749d8c80015a8\\", \\"Status\\": \\"Success\\", \\"Message\\": \\"OK\\", \\"TotalRows\\": 1, \\"LoadedRows\\": 1, \\"FilteredRows\\": 0, \\"CommitAndPublishTimeMs\\": 16}"
}

DSL Processing Function

// Use a comma as the delimiter to split the log into three segments.
ext_sepstr("regex", "f1, f2, f3", sep=",")
// Retain only the f3 field: the characters starting from "resp content:" and beyond.
fields_drop("regex","f1","f2")
// Use a colon to split the f3 field value, retaining only the characters within the curly braces {} and parentheses. Store the result in the resp_content field, which is a json package returned by flink.
ext_sepstr("f3", "f1,resp_content", sep=":")
fields_drop("f1","f3")
// Structuralize the resp_content field.
ext_json("resp_content", prefix="")
// Discard the original field.
fields_drop("resp_content")

Processing Result

{
"CommitAndPublishTimeMs":"16",
"FilteredRows":"0",
"Label":"flink_connector_20211202_1de749d8c80015a8",
"LoadedRows":"1",
"Message":"OK",
"Status":"Success",
"TotalRows":"1",
"TxnId":"58322"
}


Bantuan dan Dukungan

Apakah halaman ini membantu?

masukan