tencent cloud

Cloud Log Service

DokumentasiCloud Log ServiceOperation GuideShipping and ConsumptionShipping to CKafkaCreating Shipping Task

Creating Shipping Task

Download
Mode fokus
Ukuran font
Terakhir diperbarui: 2026-05-21 19:55:02

Application Scenario

You can ship log topic data to CKafka for real-time stream computing and storage. If you haven't purchased a CKafka instance, you can consider using the Consumption over Kafka feature of CLS.

Prerequisites

You have activated the CKafka service.
Note:
If you enable ACL for CKafka, please note the version limitations: CKafka 1.X requires version 1.1.1 or higher; CKafka 2.X requires version 2.4.2 or higher. Enabling ACL on other versions will cause delivery failure. You can disable ACL to resolve the issue.
Currently only support real-time log delivery. Historical logs are not supported.
Delivery to CKafka elastic Topics is not supported.
CKafka support routing access method requires configuration as PLAINTEXT, otherwise delivery will fail.
Ensure that the current account has the permission to enable shipping to CKafka. For details, see CLS Access Policy Template.

Directions

1. Prepare and configure a CKafka Topic.
1.1 Create a CKafka instance in the same region as the log topic. For more information, see Creating Instance.
1.2 Create a topic. For details, see Creating Topic. You need to make the following changes to the topic configuration. Other parameters can be configured as needed.
cleanup.policy: Select delete; otherwise, shipping will fail.
max.message.bytes: Set this value to 12 MB or above. Otherwise, when the size of a single message in CLS exceeds the specified limit, the message cannot be written to the CKafka topic, and shipping will fail.

2. Go to the CLS console. There are two ways to enter the delivery task management.
Entry 1: In the left navigation bar, click Shipping Job Management, select the region, log set, and log topic, and then select the Ship to CKafka tab.
Entry 2: In the log topic list, select and enter the log topic that needs to deliver to CKafka. On the log topic details page, select the Ship to CKafka tab on the left side.
3. Click Edit on the right, enable the switch of delivering to CKafka, select the appropriate CKafka instance and corresponding Topic, then start shipping. You can choose to ship logs in original content or JSON format.
Deliver original content to CKafka. Configuration items are described:
Configuration Item
Description
Rule
Required
Target CKafka Topic Ownership
Current Root Account
Other Root Account
Deliver CLS logs to the current root account's CKafka Topic.
Deliver CLS logs to another root account's CKafka. For example, if account A ships logs to account B's CKafka Topic via CLS, account B must configure an access role in Cloud Access Management (CAM). After configuration, account A needs to enter the Role ARN and external ID in the CLS console to enable cross-account delivery. The steps to configure the role are as follows:
1. Create new role. Account B logged in to the CAM role management page.
1.1 Create an access policy, with a policy name such as cross_shipper. For policy syntax, see the following:
Note:
Note: The authorization in the example follows the minimum permission principle, with the resource configured as shipping to only the CKafka instance (ckafka-12abcde3) in the Guangzhou region. Please authorize according to the actual situation.
  {
    "statement": [
        {
            "action": [
                "cam:GetRole"
            ],
            "effect": "allow",
            "resource": [
                "*"
            ]
        },
        {
            "action": [
                "ckafka:ListRoute",
                "ckafka:AddRoute",
                "ckafka:DescribeInstanceAttributes",
                "ckafka:AuthorizeToken",
                "ckafka:CreateToken",
                "ckafka:DescribeTopicAttributes"
            ],
            "effect": "allow",
            "resource": [
                "qcs::ckafka:ap-guangzhou:uin/100001234567:ckafkaId/ckafka-12abcde3"
            ]//resource configured as shipping to CKafka: ckafka-12abcde3 in the Guangzhou region only, please authorize according to the actual situation
        }
    ],
    "version": "2.0"
}
1.2 Create a new role, select Tencent Cloud account as the role carrier, choose other root account for the cloud account type, then input Account A's ID, such as 100012345678, check enable verification and configure the external ID, for example: Hello123.
1.3 Configure role policy, configure access policy for the role, and select the pre-configured access policy cross_shipper (example).
1.4 Save the role, for example: uinA_writeCLS_to_CKafka.
2. Configure the carrier for the role. In the CAM role list, find uinA_writeCLS_to_CKafka (example), click the role, select role carrier > management carrier > add product service > CLS, then click refresh.
The current role's carriers are two: account A and cls.cloud.tencent.com (CLS log service).
3. Account A logs in to CLS and fills in Role ARN and external ID.
The two items of info need to be provided by account B:
Account B finds the role uinA_writeCLS_to_CKafka (example) in the CAM role list, clicks to view the RoleArn of the role, such as qcs::cam::uin/100001112345:roleName/uinA_writeCLS_to_CKafka.
The external ID, such as Hello123, is visible in the role carrier.
Note:
Fill in the Role ARN and external ID. Note: Do not enter extra spaces, as this will cause permission verification to fail.
Note: Cross-account delivery will generate read traffic fees for the log topic under Account A.
current root account other root account
No
CKafka instance
The CKafka Topic in the same region as the current log topic is used as the delivery target.
In the cross-account delivery scenario, the user manually fills in the CKafka instance ID and Topic name.
List selection
Required
Format of Data to Ship
Select Original content to deliver the user's raw logs.
List selection
Required
Data compression format
no compression\\SNAPPY\\LZ4.
List selection
Required
Shipping log preview
Preview your delivered log data.
-
-
Deliver in JSON format to CKafka. Configuration items are described:
Configuration Item
Description
Rule
Required
Target CKafka Topic Ownership
Current Root Account
Other Root Account
Deliver CLS logs to the current root account's CKafka Topic.
Ship CLS logs to another root account's CKafka. For example, if account A ships logs from CLS to account B's CKafka Topic, account B must configure a role in CAM (Access Management). After configuration, account A needs to enter the Role ARN and external ID in the CLS console to enable cross-account delivery. The steps to configure the role are as follows:
1. Create role. Account B logged in to the CAM role management page.
1.1 Create an access policy, with a policy name such as cross_shipper. For policy syntax, see the following:
Note:
Note: The authorization in the example follows the minimum permission principle, with the resource configured as shipping to only the CKafka instance (ckafka-12abcde3) in the Guangzhou region. Please authorize according to the actual situation.
  {
    "statement": [
        {
            "action": [
                "cam:GetRole"
            ],
            "effect": "allow",
            "resource": [
                "*"
            ]
        },
        {
            "action": [
                "ckafka:ListRoute",
                "ckafka:AddRoute",
                "ckafka:DescribeInstanceAttributes",
                "ckafka:AuthorizeToken",
                "ckafka:CreateToken",
                "ckafka:DescribeTopicAttributes"
            ],
            "effect": "allow",
            "resource": [
                "qcs::ckafka:ap-guangzhou:uin/100001234567:ckafkaId/ckafka-12abcde3"
            ]//resource configured as shipping to CKafka: ckafka-12abcde3 in the Guangzhou region only, please authorize according to the actual situation
        }
    ],
    "version": "2.0"
}
1.2 Create a new role, select Tencent Cloud account as the role carrier, choose other root account for the cloud account type, then input Account A's ID, such as 100012345678, check enable verification and configure the external ID, for example: Hello123.
1.3 Configure role policy, configure access policy for the role, and select the pre-configured access policy cross_shipper (example).
1.4 Save the role, for example: uinA_writeCLS_to_CKafka.
2. Configure a carrier for the role. Find uinA_writeCLS_to_CKafka (example) in the CAM role list, click the role, select role carrier > Entity management > add product service > CLS, then click update.
The current role's carriers are two: account A and cls.cloud.tencent.com (CLS log service).
3. Account A logs in to CLS and fills in Role ARN and external ID.
The two items of info need to be provided by account B:
Account B finds the role uinA_writeCLS_to_CKafka (example) in the CAM role list, clicks to view the RoleArn of the role, such as qcs::cam::uin/100001112345:roleName/uinA_writeCLS_to_CKafka.
The external ID, such as Hello123, is visible in the role carrier.
Note:
Fill in the Role ARN and external ID. Note: Do not enter extra spaces, as this will cause permission verification to fail.
Note: Cross-account delivery will generate read traffic fees for the log topic under Account A.
current root account other root account
No
CKafka instance
The CKafka Topic in the same region as the current log topic is used as the delivery target.
List selection
Required
Format of Data to Ship
Option JSON, deliver logs in JSON format.
List selection
Required
Log Fields to Ship
Selecting the log fields to ship
Escape/Do not escape in JSON format
Escape: Convert the value of the first-level nodes in the JSON to String. If the value of your first-level nodes is Struct and you need to convert the Struct to String in downstream storage or computation, you can select this option. Examples:
Non-escaping, do not modify your JSON structure and hierarchy, keep the log format consistent with that on the collection side. Example:
Note:
When the first-layer node of JSON contains a numeric value, it will automatically convert to int or float after delivery.
Original log: {"a":123, "b":"123", "c":"-123", "d":"123.45", "e":{"e1":123,"f1":"123"}}
Deliver to CKafka: {"a":123,"b":123,"c":-123,"d":123.45,"e":{"e1":123,"f1":"123"}}
Flatten or not flatten the __TAG__ metadata based on your actual business scenario.
__TAG__ meta information: {"__TAG__":{"fieldA":200,"fieldB":"text"}}
Flatten: {"__TAG__.fieldA":200,"__TAG__.fieldB":"text"}
Not Tiled: {"__TAG__":{"fieldA":200, "fieldB":"text"}}
Required or Not
Required
Data compression format
no compression\\SNAPPY\\LZ4.
Required or Not
Required
Shipping log preview
Preview your delivered log data.
-
-
Advanced Options
CKafka partition production rule: After the switch is turned on, a Hash value is calculated based on the field value you configured, and data with the same result is written to the same CKafka partition.
Note:
You can concatenate the values of up to 5 fields. The total character length of the concatenated values must not exceed 4096; any excess will be truncated.
Prefer to Hash fields with string or numeric values. When a field value is a complex type (such as JSON), the accuracy of partition delivery based on the Hash result is not guaranteed.
Required or Not
Required
4. Click OK to start shipping to CKafka.
Note:
To clean, process, and filter logs before shipping them to CKafka, see Data Processing.

FAQs

What should I do if the system prompts that I have no permissions to read/write the CKafka topic?

If you directly use an API to ship data to CKafka, you may not have the read/write permissions of the CKafka topic. If you ship data in the console, the system will guide you through the authorization process, but if you directly call an API for shipping, you need to authorize manually. For more information, see CLS Service Role Authorization.
Topik sebelumnya: Configure AlarmsTopik selanjutnya: Shipping Job Management

Bantuan dan Dukungan

Apakah halaman ini membantu?

masukan