In production environments, data loss or corruption is a critical risk that must be prevented in database Ops. Common triggering scenarios include:
Human error: Accidentally executing a drop command to delete a database or collection; or using an updateMany command with an empty condition to overwrite a large amount of business data.
Program logic flaw: Application code bugs cause data to be incorrectly written or overwritten.
Security incident: Unauthorized access causes data to be maliciously tampered with or deleted.
Version upgrade or architecture change: If an exception occurs during a major change, you need to roll back to the pre-change state.
For the scenarios described above, TencentDB for MongoDB provides rollback solutions at three granularities:
Clone Instance (instance-level): A complete, independent instance is restored from a backup file. This is suitable for scenarios with a large impact scope or where isolated verification is required.
Collection Rollback (database/collection-level): This restores a specified database or collection to the original instance or a new instance. It is suitable for scenarios where the scope of damage has been identified at the database or collection level.
Key-based Flashback (document-level): This precisely restores a single or a small number of documents to a target point in time based on their keys. It is suitable for scenarios involving document-level erroneous operations.
This document describes how to make appropriate decisions based on your business scenarios from three perspectives: backup selection before rollback, rollback solution selection, and verification and switchover after rollback.
1. Backup Selection
Any rollback solution depends on backup files generated beforehand. Before initiating a rollback, you must first confirm two things: the backup method used by the current instance is compatible with the rollback target, and usable backup files exist near the target point in time.
1.1 Data Backup
|
Logical backup | Back up using the mongodump tool. | Cross-version recovery: This restores data to an instance with a different version than the source instance (for example, from version 4.0 to version 5.0). Cross-instance-type migration: This refers to migration from a cloud environment to a self-built environment, or migration between different cloud providers. Small-data-volume instance: When the data volume is small, the time consumption difference is not significant. | Backup and recovery are relatively slow, and the time required increases linearly with the data volume and document quantity. The backup process has a certain impact on instance performance. It consumes a relatively large amount of storage space. |
Physical backup | Directly copy the underlying physical files of the storage engine (data files, index files, log files). | Preferred scenario for production environments: Use local disk instances of version 4.0 and later. They offer faster speeds and have a smaller impact on performance. Large-data-volume instance: The larger the data volume, the more pronounced the speed advantage becomes. Same-version recovery: This refers to rollback or cloning between instances running the same MongoDB version. | Cross-version recovery is not supported: It is bound to the MongoDB version. Supported only by version 4.0 and later. |
Key-based Flashback | Continuously records the document state (pre-image) before each Insert/Update/Delete operation on the specified collection and stores it in independent flashback storage. | Document-level protection against erroneous operations for core business tables Scenarios requiring precise recovery of individual documents | It must be enabled in advance. Collections that are not enabled cannot be used afterwards. Only MongoDB versions 5.0, 7.0, and 8.0 are supported. |
Cloud disk snapshot-based backup | Creates second-level snapshots for the storage layer based on the CBS snapshot capability. | Preferred for cloud disk edition (4.0+) instances: Faster backup speed with minimal impact on instance performance. | Supported only for cloud disk edition |
1.2 Full Backup and Incremental Backup
The backup system consists of two parts: full backup and incremental backup. Understanding their nature and how they work together helps you make more accurate decisions when you formulate a backup policy and estimate rollback duration.
Full backup: A complete data snapshot of the database at a specific point in time. It serves as the baseline starting point for rollback and can independently restore an instance to that point in time.
Incremental backup: Incremental change data based on the oplog. The kernel continuously generates oplog during instance operation for replica set synchronization. Based on this, the backup system continuously collects and saves incremental changes for the corresponding intervals, forming incremental backups that can be used for rollback.
|
Nature | A complete data snapshot at a specific point in time | Incremental change data based on oplog |
Generation Method | Triggered periodically by the backup policy or manually by users. | Automatically and continuously generated by the backup system, requiring no separate trigger. |
Storage Location | Tencent Cloud Object Storage (COS) | Tencent Cloud Object Storage (COS) |
Role in Rollback | Provides the baseline for rollback starting point | Supplements changes up to the target point in time on top of the baseline. |
Note:
Rollback collaboration: A full backup provides a "point-in-time baseline", while an incremental backup provides "fine-grained changes from that baseline to the target point in time". During rollback, the system's workflow is as follows: the most recent full backup + the incremental backups from that point to the target point in time → restoration to the target point in time. The entire process is automatically completed by the system during rollback. Users only need to select the target point in time and do not need to concern themselves with internal details such as the collection and replay of incremental data.
Backup policy recommendations: For most instances, we recommend using a combination of periodic automatic full backups + incremental backups. The retention period for incremental backups should be no shorter than that for full backups to avoid a rollback window gap where "the full backup is within its retention period, but the incremental backup has expired". Before and after critical changes (such as version upgrades or batch data corrections), you can add manual full backups to enhance disaster recovery granularity.
1.3 Performing a Full Manual Backup Before Major Changes
Significance of Manual Backup
The rollback duration depends on the time interval between the most recent full backup and the target point in time: the longer the interval, the more oplog that needs to be replayed, and the longer the rollback takes.
Core principle: Proactively triggering a manual full backup before a major change can fix the rollback starting point to the most recent time before the change. If the change fails and a rollback is required, this can significantly reduce the rollback duration.
Scenarios for Manual Backup
The common characteristic of the following scenarios is that the operations are irreversible or have a large impact scope. If they fail, a large amount of data or business continuity will be affected. Proactively triggering a manual backup before the operation can provide a backup baseline with a close time point for rollback.
|
Major Database Changes | • Database version upgrade (for example, 4.0 → 5.0) • Shard key adjustment, scaling in or out, and other architectural changes • Major database parameter adjustments and replica set member changes | The change involves the kernel or cluster topology level, and a failure affects the availability of the entire instance. |
Large-Scale Data Operations | • Before batch commands such as updateMany and deleteMany are executed • Before data migration scripts or data cleansing scripts are deployed • Before large-scale data import or merge operations are performed | A single operation affects a large number of documents, and logical errors are difficult to recover precisely through oplog. |
Application Version Release | • Application iterations involving data structure changes (such as adding fields and modifying field types) • Major version releases involving significant changes to business logic • A full data snapshot before a canary release | Application-layer bugs may continuously write erroneous data, and rollback requires a clear point-in-time baseline. |
Business-Critical Time Points | • Before business peaks such as major promotions, events, and year-end settlements • Data snapshots before quarterly or annual settlements | The rollback window is short during peak business periods, requiring advance preparation of a baseline that closely matches the current state. |
2. Rollback Solution Selection and Description
When selecting a rollback solution, match the solution granularity based on the scope of data corruption: use Key-based Flashback when the corruption scope is a small number of documents, use collection rollback when the corruption scope is a complete database or collection, and use a clone instance when the corruption scope covers the entire instance or multiple core databases. If Key-based Flashback is not enabled in advance, you can use collection rollback to generate a _bak collection, and then extract the required target documents from it and merge them back.
2.1 Rollback Solution Overview
|
Rollback Principle | Quickly searches for the document state at the target point in time based on the flashback Key (default is _id), and generates a rollback collection. | Restores the specified database or collection to the original instance (creating a new collection with the _bak suffix) or a new instance, based on existing backup files. | Clones data into a brand-new, independent database instance based on existing backup files. |
Rollback Granularity | Single document | Database or collection | The entire instance |
Typical Scenario | Document-level erroneous operations, with the affected Key range being clear and can be listed one by one. | Damage scope is clearly defined at the database or collection level. | The impact scope is extensive or its boundaries are unclear, requiring isolated verification. |
Rollback Output | Flashback collection + missing record collection | New collection with the _bak suffix | An independent new instance |
Switchover Method | Merge data and write it back to the original collection. | Batch rename tables (add the _ori suffix to the original table names) | IP address swap |
Prerequisites | Instance version 5.0, 7.0, or 8.0; the flashback feature for the target collection has been enabled in advance. | None. | None. |
2.2 Key-based Flashback
The following scenarios share a common characteristic: the number of affected documents is small (typically within 100), they involve sensitive data, have a significant impact on online business, and require rapid remediation. In these cases, each affected record can be precisely located via the _id or other index fields.
A program bug caused the status fields of a small number of orders to be incorrectly updated (for example, status changed from "paid" to "cancelled").
A user reported abnormal data for individual accounts. The specific userId list has been located.
In online games, a user may have accidentally deleted equipment, or equipment may have been damaged due to account theft, and the affected user ID or equipment ID is known.
Specific Operation
Submit a flashback task in the console based on the known affected record IDs (such as user IDs or product IDs) and the correct point in time before the failure. For detailed operations, see Key-based Flashback. Post-Rollback Processing
After you initiate a Key-based Flashback, the system only generates temporary collections containing historical snapshots (the flashback collection and the missing record collection). The original online business data is not automatically modified or overwritten. You need to manually compare the data and use commands to batch write the correct historical records back to the original table. For detailed steps, see Batch Data Update Example. 2.3 Collection Rollback
The following scenarios share a common characteristic: the scope of damage has been identified at the database or collection level, data in other databases or collections is normal, and a full instance rollback is not required.
The order collection was accidentally deleted by executing db.orders.drop(), but data in other databases and tables remains normal.
The configuration center table (for example, config_center) was incorrectly updated in a batch, but the user behavior table was not affected.
An exception in the data migration script caused data corruption in several business databases, while other databases remain normal.
The same configuration tables across multiple instances were corrupted by the same faulty script and require a batch rollback.
Specific Operation
Submit a rollback task in the console based on the known scope of damaged databases or collections and the point in time before the failure. The system supports performing a collection rollback within a single instance (with an upper limit of 2000 databases or collections per operation), or selecting multiple instances for a batch rollback. For detailed steps, see the official documentation: Database and Collection Rollback and Batch Rollback. Post-Rollback Processing
After you initiate a rollback, the system only generates temporary collections with the _bak suffix. The original online business data is not automatically modified. You need to complete the final data restoration by following the process below:
1. Data verification: In mongosh, spot-check the number of documents and key fields in the _bak collection to confirm that the data status meets expectations.
2. Full Table Switch (Batch Rename Tables in Console): To restore an entire table, directly use the "Batch Rename Tables" feature in the console. Rename the original table to keep it as a backup, and rename the _bak table to the original table name within seconds to complete the switch. For detailed operations, see Batch Rollback. 3. Precise Merge (Manual Command Line Processing): If you only need to restore some documents, you can use a script to iterate through the _bak collection and leverage updateOne with upsert: true to precisely write back or replace the specified records into the original collection.
2.4 Clone Instance
The following scenarios share a common characteristic: the scope of impact is large or its boundaries are unclear. Performing operations directly on the production instance risks amplifying the impact, requiring a rollback carrier that is completely isolated from the production environment.
After the new application version was launched, a critical Bug was discovered. Data in a large number of core business tables (orders, users, payments) was incorrectly overwritten, and the scope of impact is difficult to quickly define.
When an instance encounters a security incident (for example, data is maliciously encrypted, deleted, or tampered with), you need to restore the complete data without contaminating the current instance.
Compatibility issues arise after a version upgrade, requiring a rollback to the full instance state before the upgrade.
You need to build an independent test or staging environment based on historical data to pre-validate complex changes.
Specific Operation
Clone instances are billed separately at standard rates. For detailed operations, see Clone Instance. Post-Rollback Processing
1. Verify data integrity (data volume, key fields, indexes, timestamp accuracy) on the clone instance.
2. After verification is successful, complete the IP address switchover by using the Switch Network feature in the console.
3. In the console, set an IP retention period (24 hours by default) for the original instance's IP address. Release the original IP address only after the business runs stably on the new instance. During the retention period, if an exception occurs on the new instance, you can quickly roll back by redirecting the business to the original IP address, with a low rollback cost.
3. Data Verification and Service Switchover After Rollback
Completing the rollback task is only the first step in data recovery. Successful business recovery requires two additional phases: data verification and traffic switchover.
3.1 Key Points for Data Verification
Before switching, you should verify each item from at least the following dimensions:
|
Document quantity | The document quantity of key collections matches the data volume at the expected point in time. |
Key fields | Sample core business fields to verify that the content is restored to the state at the expected point in time. |
Index integrity | The list, types (such as single-key, compound, and TTL indexes), and fields of indexes match those at the target point in time. |
Point-in-time accuracy | Confirm that the data at the target rollback point in time is restored normally. |
Business logic | Core business status, document properties, and inter-collection association IDs comply with business rules, with no logical exceptions such as data state disorder or reference invalidation (such as orphaned documents and data link breaks). |
Application connectivity | Using the application account, normal read and write operations are possible, and the business flow of key APIs is normal. |
3.2 Principles for Service Switchover
Small-scale verification: After the switchover, first allow access for internal test accounts. Only after confirming that the core links are normal should you fully open access.
Retain the original instance: After the switchover is complete, retain the original instance for a period of time as a fallback option for exception rollback.
Execute During Off-Peak Hours: Prioritize performing the switchover during business off-peak hours to minimize the impact scope.
Prepare a Rollback Plan: Define how to quickly revert to the pre-switch state if new issues are discovered after the switchover.
4. Common Misconceptions
|
"Having auto-backup is sufficient, and manual backup is unnecessary." | Performing a manual backup before a major change allows the rollback starting point to be closer to the pre-change state, significantly reducing rollback time. |
"With the availability of key-based flashback, conventional backup capabilities are no longer necessary." | Key-based flashback is recommended only for rolling back a small amount of data. For large-scale data recovery, using the traditional rollback method is more efficient. |
"Collection rollback overwrites the original data." | Collection rollback creates a new collection with the _bak suffix, and the original data is retained. |
"The completion of a rollback task is equivalent to a successful recovery." | A complete recovery is only achieved after data verification, service switching, and an observation period. |