Shipping to COS
CLS can ship data in a log topic to COS to meet the needs in the following scenarios:
Logs are shipped to and stored in COS in STANDARD storage class. If you need other storage classes, perform relevant operations in COS. For more information, see Overview. Log data is processed through offline computing or other computing programs. Such data is shipped to COS first and then loaded by Data Lake Compute (data lake) or EMR (big data platform) for further analysis. For more information, see Using Data Lake Compute (Hive) to Analyze CLS Log. We recommend you choose CSV or Parquet as the shipping format. Billing Description
Log shipping generates private network read traffic fees (cross-region shipping is not supported for now), and CLS will charge fees based on the compression format (Snappy/GZIP/lzop). If your raw log is 100 GB and you choose Snappy for compression, around 50 GB will be billable. As the read traffic price is 0.032 USD/GB, the fees will be 50 GB * 0.032 USD/GB = 1.6 USD.
Feature Limits
Historical data cannot be shipped.
Cross-region shipping is not supported. The log topic and COS bucket must be in the same region.
Cross-account shipping is not supported.
|
| Log data is shipped to COS based on the specified separator, such as space, tab, comma, semicolon, and vertical bar. | It can be used for computing in Data Lake Compute. It can be used to ship raw logs (logs collected in a single line, in multiple lines, and with separators). |
| Log data is shipped to COS in JSON format. | It is a common data format and can be selected as needed. |
| Log data is shipped to COS in Parquet format. | Log data needs to be structured data. The data type can be converted (data not collected in a single line or multiple lines). This format is mainly used for Hive batch processing. |
Note:
After log data is shipped to COS, COS storage fees will be incurred. For billing details, see Billing Overview.