通过 mongoshake 进行 DDS 在线迁移

举报
HuaweiCloudDeveloper 发表于 2026/02/05 17:23:27 2026/02/05
【摘要】 通过 mongoshake 进行 DDS 在线迁移华为云迁移实践指南- 使用 mongo-shake 进行 DDS 数据库迁移1. 引言本实践指南将演示如何使用 mongo-shake 将数据从单节点 DDS 数据库迁移到华为云 ShardDDS。2. 开始使用在开始之前，您需要在华为云上设置一些资源，以便用于执行迁移。本节将介绍配置此DDS迁移演练所需的华为云资源的步骤。2.1配置环境2....

通过 mongoshake 进行 DDS 在线迁移

华为云迁移实践指南

- 使用 mongo-shake 进行 DDS 数据库迁移

1. 引言

本实践指南将演示如何使用 mongo-shake 将数据从单节点 DDS 数据库迁移到华为云 ShardDDS。

2. 开始使用

在开始之前，您需要在华为云上设置一些资源，以便用于执行迁移。本节将介绍配置此DDS迁移演练所需的华为云资源的步骤。

2.1

配置环境

2.1.1

使用Terraform配置sourceDDS

Terraform是一款开源的IT基础设施编排管理工具。Terraform支持使用配置文件来描述单个应用程序或整个数据中心。

借助Terraform，您可以轻松地创建、管理、删除和控制华为云资源的版本。

1. 访问Terraform官方网站https://www.terraform.io/intro/getting-started/install.html，并下载对应操作系统的安装包terraform_1.0.3_windows_amd64.zip。

2. 解压安装包，并将存储Terraform可执行文件的目录添加到系统环境变量PATH中。

3. 在工作目录中创建main.tf文件，以指定注册表源和HUAWEI CLOUD Provider。


content is as follows:

terraform {

terraform {

required_providers {

huaweicloud = {

source = "huaweicloud/huaweicloud"

version = ">= 1.20.0"

}

}

}

# Configure the HuaweiCloud Provider

provider "huaweicloud" {

region = var.myregion_master

access_key = " your access key "

secret_key = " your secret key "

}

variable "myregion_master" {

type = string

default = "ap-southeast-3"

description = "Define the region - Singapore"

}

variable "myaz_master" {

type = string

default = "ap-southeast-3a"

description = "Define the availability zone - Singapore AZ1"

}

variable "myeip_address" {

type = string

}

myeip_address = "your_eip_address"

variable "ecs_flavor_name" {

type = string

default = "c6.large.2"

description = "Define the flavor name that will be used to create instance"

}

variable "mytags" {

type = object({

name = string

staffno = number

})

description = "Define the tags that will be used to create resources"

}

mytags = {

name = "SMS-Lab"

staffno = 84191429

}

#Create a single DDS

resource "huaweicloud_dds_instance_v3" "single" {

name = "dds-single"

datastore {

type = "DDS-Community"

version = "3.4"

storage_engine = "wiredTiger"

}

enterprise_project_id = "a37b4a04-843a-40c4-90f7-2142ee2f32f5"

region = var.myregion_master

availability_zone = var.myaz_master

vpc_id = data.huaweicloud_vpc.myvpc.id

subnet_id = data.huaweicloud_vpc_subnet.subnet1.id

security_group_id = data.huaweicloud_networking_secgroup.mysecgroup.id

ssl = false

password = "R******4"

tags = var.mytags

mode = "Single"

flavor {

type = "single"

num = 1

storage = "ULTRAHIGH"

size = 20

spec_code = "dds.mongodb.c3.large.2.single"

}

}

You needs toreplace the access key and secret key in above main.tf with your own credentials

• Runterraform initcommand to initialize the system:

2.1.2

验证源DDS

• 登录华为云控制台，您将看到新的DDSinAvailable状态。

• Logintarget DDS to verify the database status

# mongo –host 192.168.1.78:8635 –u rwuser -p

# show dbs

• 节点类型 IP 用户名 密码 DDS-single 192.168.1.78:8635 rwuser R******4 记录迁移中将使用的源节点信息：


2.1.3

使用Terraform配置目标DDS

修改工作目录中的main.tf文件以添加DDS源。文件内容如下：

#Create a DDS

resource "huaweicloud_dds_instance_v3" "instance" {

name = "dds-84191429"

datastore {

type = "DDS-Community"

version = "4.0"

storage_engine = "wiredTiger"

}

enterprise_project_id = "a37b4a04-843a-40c4-90f7-2142ee2f32f5"

region = var.myregion_master

availability_zone = var.myaz_master

vpc_id = data.huaweicloud_vpc.myvpc.id

subnet_id = data.huaweicloud_vpc_subnet.subnet1.id

security_group_id = data.huaweicloud_networking_secgroup.mysecgroup.id

ssl = false

password = "R******4"

tags = var.mytags

mode = "Sharding"

flavor {

type = "mongos"

num = 2

spec_code = "dds.mongodb.c3.medium.4.mongos"

}

flavor {

type = "shard"

num = 2

storage = "ULTRAHIGH"

size = 20

spec_code = "dds.mongodb.c3.medium.4.shard"

}

flavor {

type = "config"

num = 1

storage = "ULTRAHIGH"

size = 20

spec_code = "dds.mongodb.c3.large.2.config"

}

}

• Runterraformapply, then terraform showto review the result.

2.1.4

验证目标DDS

• 登录华为云控制台，您将看到新的数据磁盘实例（DDSinstance）处于“可用”状态

• Logintarget DDS to verify the database status

# mongo –host 192.168.1.207:8635 –u rwuser -p

# show dbs

# sh.status()

•


3. 迁移实施

既然你已经完成了实验室环境的设置，那么你已经准备好迁移示例数据库了。

本分步指南将演示如何使用 mongo-shake 工具将数据从源华为云 DDS 迁移到目标 DDS。

3.1

Mongo-shake简介

Mongo-Shake是一个基于MongoDB操作日志（oplog）的通用数据复制平台。它读取MongoDB集群的操作日志，并复制MongoDB数据，随后通过操作日志实现特定需求。

我们仅列出了一些应用场景：

• 集群间MongoDB数据的异步复制消除了双重写入的成本。

• MongoDB集群数据的镜像备份。（此开源版本暂不支持）

• 日志离线分析。

• 日志订阅。

• 缓存同步。通过日志分析的结果，可以确定哪些缓存可以消除，哪些缓存可以预加载以促使缓存更新。

• 根据日志进行监控。

Mongo=Shake从源MongoDB数据库获取oplog（操作日志），并在目标MongoDB数据库中重放，或者通过不同的通道发送到其他端点。现有的通道类型如下：

• 直接写入。直接写入目标MongoDB数据库。

• RPC。在go中通过net/rpc进行连接。

• TCP。通过TCP连接。

• 文件。通过文件连接。

• Kafka。通过Kafka通道连接。

• 模拟。用于测试，丢弃所有数据，而不写入隧道。

3.2

Mongo-shake迁移流程

以下展示了使用 mongo-shake 迁移 MongoDB 数据库的步骤。

在以下链接下载 mongo-shake 工具：https://github.com/alibaba/MongoShake/releases/download/release-v2.6.5-20210723-1/mongo-shake-v2.6.5.tar.gz

2. Insert dump data in source DDS

#mongo --host 192.168.1.78:8635 -u rwuser –p

>show dbs

>use test2

> post={"title":"post","content":"My blog post"}

>db.blog.insert(post)

>db.blog.find()

3. Configure mongo-shake parameters

[root@ecs-84191429-test-01 mongo-shake]# grep -v ^# collector.conf

sync_mode = full

mongo_urls = mongodb://rwuser:R******4@192.168.1.78:8635 ##source DDS

tunnel = direct

tunnel.address = mongodb://rwuser:R******4@192.168.1.207:8635 #target DDS

mongo_connect_mode = primary

incr_sync.mongo_fetch_method = oplog

incr_sync.worker = 8

incr_sync.tunnel.write_thread = 8

4. Runmongo-shake to start the full data migration

# ./collector -conf=collector.conf –verbose 1

[root@ecs-84191429-test-01 mongo-shake]# more logs/collector.log

[2021/09/07 19:23:30 CST] [INFO] log init succ. log.dir[./logs/] log.name[collector.log] log.level[debug]

[2021/09/07 19:23:30 CST] [INFO] MongoDB Version Source[3.4.14.17] Target[4.0.3]

[2021/09/07 19:23:30 CST] [WARN]

if you have any problem, please visit https://github.com/alibaba/MongoShake/wiki/FAQ

[2021/09/07 19:23:30 CST] [INFO] New session to mongodb://rwuser:***@192.168.1.78:8635 successfully

[2021/09/07 19:23:30 CST] [INFO] Close session with mongodb://rwuser:***@192.168.1.78:8635

[2021/09/07 19:23:30 CST] [INFO] New session to mongodb://rwuser:***@192.168.1.78:8635 successfully

[2021/09/07 19:23:30 CST] [INFO] Close session with mongodb://rwuser:***@192.168.1.78:8635

[2021/09/07 19:23:30 CST] [INFO] Collector startup. shard_by[collection] gids[[]]

[2021/09/07 19:23:30 CST] [INFO] Collector configuration {"ConfVersion":10,"Id":"mongoshake","MasterQuorum":false,"FullSyncHTTPListenPort":9101,"IncrSyncHTTP

ListenPort":9100,"SystemProfilePort":9200,"LogLevel":"debug","LogDirectory":"./logs/","LogFileName":"collector.log","LogFlush":false,"SyncMode":"full","Mongo

Urls":["mongodb://rwuser:***@192.168.1.78:8635"],"MongoCsUrl":"","MongoSUrl":"","MongoSslRootCaFile":"","MongoSslClientCaFile":"","MongoConnectMode":"primary

","Tunnel":"direct","TunnelAddress":["mongodb://rwuser:***@192.168.1.207:8635"],"TunnelMessage":"raw","TunnelKafkaPartitionNumber":1,"TunnelJsonFormat":"","T

unnelMongoSslRootCaFile":"","FilterNamespaceBlack":[],"FilterNamespaceWhite":[],"FilterPassSpecialDb":[],"FilterDDLEnable":false,"FilterOplogGids":false,"Che

ckpointStorageUrl":"mongodb://rwuser:***@192.168.1.78:8635","CheckpointStorageDb":"mongoshake","CheckpointStorageCollection":"ckpt_default","CheckpointStorag

eUrlMongoSslRootCaFile":"","CheckpointStartPosition":1,"TransformNamespace":[],"SpecialSourceDBFlag":"","FullSyncReaderCollectionParallel":6,"FullSyncReaderW

riteDocumentParallel":8,"FullSyncReaderDocumentBatchSize":128,"FullSyncReaderParallelThread":1,"FullSyncReaderParallelIndex":"_id","FullSyncCollectionDrop":t

rue,"FullSyncCreateIndex":"none","FullSyncReaderOplogStoreDisk":false,"FullSyncReaderOplogStoreDiskMaxSize":256000,"FullSyncExecutorInsertOnDupUpdate":false,

"FullSyncExecutorFilterOrphanDocument":false,"FullSyncExecutorMajorityEnable":false,"IncrSyncMongoFetchMethod":"oplog","IncrSyncChangeStreamWatchFullDocument

":false,"IncrSyncOplogGIDS":[],"IncrSyncShardKey":"collection","IncrSyncShardByObjectIdWhiteList":[],"IncrSyncWorker":8,"IncrSyncTunnelWriteThread":8,"IncrSy

ncTargetDelay":0,"IncrSyncWorkerBatchQueueSize":64,"IncrSyncAdaptiveBatchingMaxSize":1024,"IncrSyncFetcherBufferCapacity":256,"IncrSyncExecutorUpsert":false,

"IncrSyncExecutorInsertOnDupUpdate":false,"IncrSyncConflictWriteTo":"none","IncrSyncExecutorMajorityEnable":false,"CheckpointStorage":"database","CheckpointI

nterval":5000,"FullSyncExecutorDebug":false,"IncrSyncDBRef":false,"IncrSyncExecutor":1,"IncrSyncExecutorDebug":false,"IncrSyncReaderDebug":"","IncrSyncCollis

ionEnable":false,"IncrSyncReaderBufferTime":1,"IncrSyncWorkerOplogCompressor":"none","IncrSyncTunnelKafkaDebug":"","Version":"$","SourceDBVersion":"3.4.14.17

","TargetDBVersion":"4.0.3","IncrSyncTunnel":"","IncrSyncTunnelAddress":null,"IncrSyncTunnelMessage":"","HTTPListenPort":0,"SystemProfile":0}

[2021/09/07 19:23:30 CST] [INFO] start running with mode[full], fullBeginTs[0[0, 0]]

[2021/09/07 19:23:30 CST] [INFO] source is replica or mongos, no need to fetching chunk map

[2021/09/07 19:23:30 CST] [INFO] New session to mongodb://rwuser:***@192.168.1.78:8635 successfully

[2021/09/07 19:23:30 CST] [INFO] Close client with mongodb://rwuser:***@192.168.1.78:8635

[2021/09/07 19:23:30 CST] [INFO] all namespace: map[{test2 blog}:{}]

[2021/09/07 19:23:30 CST] [INFO] New session to mongodb://rwuser:***@192.168.1.207:8635 successfully

[2021/09/07 19:23:30 CST] [INFO] replication from [replica] to [sharding]

[2021/09/07 19:23:30 CST] [INFO] document syncer-0 do replication for url=mongodb://rwuser:R******4@192.168.1.78:8635

[2021/09/07 19:23:30 CST] [INFO] New session to mongodb://rwuser:***@192.168.1.78:8635 successfully

[2021/09/07 19:23:30 CST] [INFO] Close client with mongodb://rwuser:***@192.168.1.78:8635

[2021/09/07 19:23:30 CST] [INFO] DBSyncer id[0] source[mongodb://rwuser:***@192.168.1.78:8635] target[mongodb://rwuser:***@192.168.1.207:8635] startTime[2021-09-07 19:23:30.850149394 +0800 CST m=+0.081746134] collExecutor-1 sync ns {test2 blog} to {test2 blog} begin

[2021/09/07 19:23:30 CST] [INFO] New session to mongodb://rwuser:***@192.168.1.207:8635 successfully

[2021/09/07 19:23:30 CST] [INFO] New session to mongodb://rwuser:***@192.168.1.78:8635 successfully

[2021/09/07 19:23:30 CST] [INFO] splitter[DocumentSplitter src[mongodb://rwuser:***@192.168.1.78:8635] ns[{test2 blog}] count[1] pieceByteSize[0 MB] pieceNumber[0]] disable split or no need

[2021/09/07 19:23:30 CST] [INFO] splitter[DocumentSplitter src[mongodb://rwuser:***@192.168.1.78:8635] ns[{test2 blog}] count[1] pieceByteSize[0 MB] pieceNumber[0]] exits

[2021/09/07 19:23:30 CST] [INFO] reader[DocumentReader id[0], src[mongodb://rwuser:***@192.168.1.78:8635] ns[{test2 blog}] query[map[]]] client is empty, create one

[2021/09/07 19:23:30 CST] [INFO] New session to mongodb://rwuser:***@192.168.1.78:8635 successfully

[2021/09/07 19:23:30 CST] [INFO] reader[DocumentReader id[0], src[mongodb://rwuser:***@192.168.1.78:8635] ns[{test2 blog}] query[map[]] docCursorId[0]] generates new cursor

[2021/09/07 19:23:30 CST] [INFO] reader[DocumentReader id[0], src[mongodb://rwuser:***@192.168.1.78:8635] ns[{test2 blog}] query[map[]] docCursorId[0]] finish

[2021/09/07 19:23:30 CST] [INFO] splitter reader finishes: DocumentReader id[0], src[mongodb://rwuser:***@192.168.1.78:8635] ns[{test2 blog}] query[map[]] docCursorId[0]

[2021/09/07 19:23:30 CST] [INFO] reader[DocumentReader id[0], src[mongodb://rwuser:***@192.168.1.78:8635] ns[{test2 blog}] query[map[]] docCursorId[0]] close

[2021/09/07 19:23:30 CST] [INFO] DBSyncer id[0] source[mongodb://rwuser:***@192.168.1.78:8635] target[mongodb://rwuser:***@192.168.1.207:8635] startTime[2021-09-07 19:23:30.850149394 +0800 CST m=+0.081746134] all readers finish, wait all writers finish

[2021/09/07 19:23:30 CST] [DEBG] DBSyncer id[0] doSync with table[{test2 blog}] batch _id interval [ObjectIdHex("61374746caec9e99a34cbeb1"), ObjectIdHex("61374746caec9e99a34cbeb1")]

[2021/09/07 19:23:30 CST] [INFO] Close session with mongodb://rwuser:***@192.168.1.207:8635

[2021/09/07 19:23:30 CST] [INFO] Close session with mongodb://rwuser:***@192.168.1.78:8635

[2021/09/07 19:23:30 CST] [INFO] DBSyncer id[0] source[mongodb://rwuser:***@192.168.1.78:8635] target[mongodb://rwuser:***@192.168.1.207:8635] startTime[2021-09-07 19:23:30.850149394 +0800 CST m=+0.081746134] collExecutor-1 sync ns {test2 blog} to {test2 blog} successful. db syncer-0 progress 100%

[2021/09/07 19:23:30 CST] [INFO] syncer[DBSyncer id[0] source[mongodb://rwuser:***@192.168.1.78:8635] target[mongodb://rwuser:***@192.168.1.207:8635] startTime[2021-09-07 19:23:30.850149394 +0800 CST m=+0.081746134]] closed

[2021/09/07 19:23:30 CST] [INFO] DBSyncer id[0] source[mongodb://rwuser:***@192.168.1.78:8635] target[mongodb://rwuser:***@192.168.1.207:8635] startTime[2021-09-07 19:23:30.850149394 +0800 CST m=+0.081746134] collExecutor-5 finish

[2021/09/07 19:23:30 CST] [INFO] DBSyncer id[0] source[mongodb://rwuser:***@192.168.1.78:8635] target[mongodb://rwuser:***@192.168.1.207:8635] startTime[2021-09-07 19:23:30.850149394 +0800 CST m=+0.081746134] collExecutor-0 finish

[2021/09/07 19:23:30 CST] [INFO] DBSyncer id[0] source[mongodb://rwuser:***@192.168.1.78:8635] target[mongodb://rwuser:***@192.168.1.207:8635] startTime[2021-09-07 19:23:30.850149394 +0800 CST m=+0.081746134] collExecutor-4 finish

[2021/09/07 19:23:30 CST] [INFO] DBSyncer id[0] source[mongodb://rwuser:***@192.168.1.78:8635] target[mongodb://rwuser:***@192.168.1.207:8635] startTime[2021-09-07 19:23:30.850149394 +0800 CST m=+0.081746134] collExecutor-3 finish

[2021/09/07 19:23:30 CST] [INFO] DBSyncer id[0] source[mongodb://rwuser:***@192.168.1.78:8635] target[mongodb://rwuser:***@192.168.1.207:8635] startTime[2021-09-07 19:23:30.850149394 +0800 CST m=+0.081746134] collExecutor-2 finish

[2021/09/07 19:23:30 CST] [INFO] DBSyncer id[0] source[mongodb://rwuser:***@192.168.1.78:8635] target[mongodb://rwuser:***@192.168.1.207:8635] startTime[2021-09-07 19:23:30.850149394 +0800 CST m=+0.081746134] collExecutor-1 finish

[2021/09/07 19:23:31 CST] [INFO] metric[name[replica] stage[full]] exit

[2021/09/07 19:23:31 CST] [INFO] document syncer sync end

[2021/09/07 19:23:31 CST] [INFO] Close session with mongodb://rwuser:***@192.168.1.207:8635

[2021/09/07 19:23:31 CST] [WARN]

5. Verify the target DDS data

# mongo –host 192.168.1.207:8635 –u rwuser -p

# show dbs

# use test2

# show collections

# db.blog.find()

6. Insertand update more dump data

>use test2

> post={"title":"post","content":"My blog post"}

>post.comment=[]

>db.blog.update({"title":"post"},post)

> db.blog.find()

>use test1

>for (i=1;i<=100;i++) db.students.insert({Name:"User"+i,Age:(i%150),Address:"#"+i+",beihuan Road,Langfang,China",PreferBooks:["book"+i,"hello world"]})

> db.student.find()

7. Runmongo-shake to start the incremental data synchronization

Modify collector.conf to set

sync_mode = incr

and rerun mongo-shake

# ./collector -conf=collector.conf –verbose 1

8. Verify the target DDS migration for the changes

# mongo –host 192.168.1.207:8635 –u rwuser -p

> show dbs

> use test2

> db.blog.find()



> use test1

> db.student.find()

> sh.status()

4. 总结

本实验演示了如何使用 mongo-shake 将数据从源分布式数据库系统（DDS）迁移到目标分布式数据库系统（DDS），并实时在线同步更改。

【声明】本内容来自华为云开发者社区博主，不代表华为云及华为云开发者社区的观点和立场。转载时必须标注文章的来源（华为云社区）、文章链接、文章作者等基本信息，否则作者和本社区有权追究责任。如果您发现本社区中有涉嫌抄袭的内容，欢迎发送邮件进行举报，并提供相关证据，一经查实，本社区将立刻删除涉嫌侵权内容，举报邮箱： cloudbbs@huaweicloud.com
云数据迁移 CDM 文档数据库服务 DDS
  • 点赞
  • 收藏
  • 关注作者

评论（0）

0/1000
抱歉，系统识别当前为高风险访问，暂不支持该操作

全部回复

上滑加载中

设置昵称

在此一键设置昵称，即可参与社区互动！

*长度不超过10个汉字或20个英文字符，设置后3个月内不可修改。

*长度不超过10个汉字或20个英文字符，设置后3个月内不可修改。

确认 取消

加入云驻计划，成为创作者

  • 华为云周边好礼
  • 免费体验产品
  • 特殊身份标识
  • 线下官方门票
  • 内部专家零距离
  • 与10000+优质创作者共同成长
立即加入