tencent cloud

TencentDB for MySQL

Release Notes and Announcements
Release Notes
Product Announcements
User Tutorial
Product Introduction
Overview
Strengths
Use Cases
Database Architecture
Resource Isolation Policy
Economical Instance
Feature List
Database Instance
High Availability (Multi-AZ)
Regions and AZs
Service Regions and Service Providers
Kernel Features
Overview
Kernel Version Release Notes
Functionality Features
Performance Features
Security Features
Stability Features
TXRocks Engine
LibraDB Engine
Checking and Fixing Kernel Issues
Purchase Guide
Billing Overview
Selection Guide
Purchase Methods
Renewal
Payment Overdue
Refund
Pay-as-You-Go to Monthly Subscription
Instance Adjustment Fee
Backup Space Billing
Database Audit Billing Overview
Commercial Billing and Activity Description for Database Proxy
Description of the Database Proxy Billing Cycle
Viewing Bills
Getting Started
Overview
Creating MySQL Instance
Connecting to MySQL Instance
SQL Insight (Database Audit)
Overview
Viewing Audit Instance List
Enabling Audit Service
Viewing Audit Log
Log Shipping
Configuring Post-Event Alarms
Modifying Audit Rule
Modifying Audit Services
Disabling Audit Service
Audit Rule Template
SQL Audit Rule (Legacy)
Viewing Audit Task
Authorizing Sub-User to Use Database Audit
MySQL Cluster Edition
Introduction to TencentDB for MySQL Cluster Edition
Creating TencentDB for MySQL Cluster Edition Instance
Maintenance Management Instance
Viewing Instance Monitoring
Adjusting Instance Configuration
Operations for Other Features
Migrate or upgrade to TencentDB for MySQL Cluster Edition
Operation Guide
Use Limits
Operation Overview
Instance Management and Maintenance
Instance Upgrade
CPU Elastic Expansion
Read-Only/Disaster Recovery Instances
Database Proxy
Database Management Center (DMC)
Account Management
Parameter Configuration
Backup and Rollback
Data Migration
Network and Security
Monitoring and Alarms
Log Center
Read-Only Analysis Engine
Tag
Practical Tutorial
Using TencentDB for MySQL to Upgrade MySQL 5.7 to MySQL 8.0
Methods and Instructions for Upgrading from MySQL 5.6 to MySQL 5.7
Cybersecurity Classified Protection Practice for Database Audit of TencentDB for MySQL
Building All-Scenario High-Availability Architecture
Usage Specifications of TencentDB for MySQL
Configuring Automatic Application Reconnection
Impact of Modifying MySQL Source Instance Parameters
Limits on Automatic Conversion from MyISAM to InnoDB
Creating VPCs for TencentDB for MySQL
Enhancing Business Load Capacity with TencentDB for MySQL
Setting up 2-Region-3-DC Disaster Recovery Architecture
Improving TencentDB for MySQL Performance with Read/Write Separation
Migrating Data from InnoDB to RocksDB with DTS
Building LAMP Stack for Web Application
Building Drupal Website
Calling MySQL APIs in Python
The primary and secondary instances have inconsistent query data
White Paper
Performance White Paper
Security White Paper
Troubleshooting
Connections
Performance
Instance Data Sync Delay
Failure to Enable Case Insensitivity
Failure to Obtain slow_query_log_file via a Command
API Documentation
History
Introduction
API Category
Instance APIs
Making API Requests
Data Import APIs
Database Proxy APIs
Database Audit APIs
Security APIs
Task APIs
Backup APIs
Account APIs
Rollback APIs
Parameter APIs
Database APIs
Monitoring APIs
Log-related API
Data Types
Error Codes
FAQs
Related to Selection
Billing
Backup
Rollback
Connection and Login
Parameter Modifications
Instance Upgrade
Account Permissions
Performance and Memory
Ops
Data Migration
Features
Console Operations
Logs
Event
Database audit
Instance Switch Impact
API 2.0 to 3.0 Switch Guide
Service Agreement
Service Level Agreement
Terms of Service
Reference
Standards and Certifications
Contact Us
Glossary
DocumentationTencentDB for MySQLKernel FeaturesTXRocks EnginePractical Tutorial of TXRocks

Practical Tutorial of TXRocks

PDF
Focus Mode
Font Size
Last updated: 2024-07-31 10:00:32
This document describes how to accelerate the import of massive amounts of data to a database with the TXRocks practical tutorial.

Background

Scenario: The import of massive amounts of data to a database with the TXRocks engine needs to be accelerated.
Impact: The Rows inserted during bulk load must not overlap existing rows error may be reported when massive amounts of data are imported.

Option 1

1. Delete secondary indexes and retain only the primary key index.
2. Adjust memory parameters based on the specification and data volume.
Note:
Appropriately increase the values of rocksdb_merge_buf_size and rocksdb_merge_combine_read_size parameters based on the specification and data volume.
rocksdb_merge_buf_size indicates the data volume of each way in k-way merge during index creation. rocksdb_merge_combine_read_size indicates the total memory used in k-way merge.
rocksdb_block_cache_size indicates the size of rocksdb_block_cache. We recommend you decrease its value temporarily during k-way merge.
3. Use bulk load to import the data.
SET session rocksdb_bulk_load_allow_unsorted=1;
SET session rocksdb_bulk_load=1;
...
Import the data
...
SET session rocksdb_bulk_load=0;
SET session rocksdb_bulk_load_allow_unsorted=0;
Note:
If the imported data is sorted, you don't need to configure rocksdb_bulk_load_allow_unsorted.
4. Recreate secondary indexes one by one after all data is imported.
Note:
Secondary index creation involves k-way merge. rocksdb_merge_buf_size indicates the data volume of each way, and rocksdb_merge_combine_read_size indicates the total memory used in k-way merge.
For example, we recommend you set rocksdb_merge_buf_size to 64 MB or higher and set rocksdb_merge_combine_read_size to 1 GB or higher to avoid OOM. After all data is imported, you must modify the parameters to their original values.
As a lot of memory is used during the creation of each secondary index, we recommend you not create many of them at the same time.

Option 2

You can disable unique_check during data import to improve the import performance.
SET unique_checks=OFF;
...
Import the data.
...
SET unique_checks=ON;
Note:
After the operation is completed, you must set unique_checks back to ON; otherwise, the uniqueness of INSERT operations in subsequent normal transaction writes will not be checked.
Previous Topic: Cost PerformanceNext Topic: LibraDB Engine Feature Highlights

Help and Support

Was this page helpful?

Help us improve! Rate your documentation experience in 5 mins.

Feedback