dtle 3.20.10.0

What's New

• Output job status: full dump or incremental process. 输出job状态: 全量或增量
• Add reschedule part for default job config to avoid unlimited retry. 样例job配置增加reschedule部分, 避免异常时无限重试
• Code refactoring. 部分代码重构

dtle 3.20.09.0

New

• Metrics for delay 复制延迟的监控项
• Metrics for memory 内存使用的监控项
  • See document 3.4 & 3.11 见文档3.4和3.11节

Fixed

• Use gencode instead of gob for incremental stage serialization 替换增量序列换库
  • Reduced CPU consumption and improved performance 降低CPU使用, 提升性能
• Fix #523: handle uppercase in 'where' column 处理含有大写字母的'where'列

dtle 3.20.08.0

Major changes have been made. Now dtle works as a plugin of nomad.

What's new

• Code optimization and refactoring -- 优化、重构代码

  • Adapting to latest nomad -- 适配最新的nomad

• Reduced memory consumption -- 降低内存消耗

• Improved Kafka ouput performance -- 提升Kafka输出性能

• Reorganized logs and error message -- 重整日志和报错信息

• Fixed some bugs -- 修复部分bug

  • NatsAddr might be empty sometime -- NatsAddr有时为空
  • Progress is updated before real execution -- 进度被提前更新
  • Kafka might report "unknown table structure" -- Kafka报错"unknown table structure"
  • Kafka might send the same data repeatedly -- Kafka重复发送相同消息

• Management -- 管理机制更新

  • Using consul for building up a cluster -- 使用consul来自动搭建集群
  • Using consul for saving meta data -- 使用consul保存
  • Jobs are identified by name instead of random UUID -- job标识符从随机UUID改为自定义名称
  • Use HCL for job specification -- 可使用HCL格式job配置
  • Json job (new format) and HTTP API is also supported -- json格式依然被支持(新内容格式)
  • Compatibility layer for old HTTP API -- 提供兼容层, 可使用老API和job json

• The support of Centos 6 has been dropped.

Getting started

• use rpm package, or
• docker pull consul:latest && docker pull actiontech/dtle:3.20.08.0

See updated documentation.

dtle 2.19.11.0

Features

• Binlog Relay (#475) (see doc)
  • dtle can download binlog from source mysql to local storage in case of PURGE BINARY LOGS on source mysql
• Allow starting a incremental job with binlog file & position (without GTID)
• Check the doc for new job.json options: BinlogRelay, BinlogFile, BinlogPos
• Preliminary support for MySQL 8.0 (#487)
• Change log system
  • support for dynamic updating of log levels
  • log format update
  • support log file rotation

Bugfixes

• Binlog entry size was wrongly incremented (#499)
• Job dead due to 'empty nats' (#433)
• Job dead in incremental copy of tables without a primary key (#488)
• Job dead when schema.table contains special characters (#476)
• Job stuck in incremental stage (1bee6d3)
• Job with schema.table mapping might fail (#481)

dtle 2.19.07.0

Release Note

Improvements/Features

• Reduce CPU usage in full-data copy process (#402)
• Add a tracing mechanism to locate performance problems (#477)

Bugfixes

• Fix a case of connection leakage after job removal (#41-2)
• Correctly use BindAddr and AdvertiseAddr for nats (#452 #367)
• Fix remaining error message after job resume (#479)
• Fix an immediate failure when using with MySQL 8.0
  • Notice: MySQL 8.0 is still not tested with us.
• When one transaction is too large, make the job dead instead of OOM crash (#462)
• Fix some cases on table renaming (mapping) (#461, #470)

dtle 2.19.05.0

Release Note

Features:

• improve: job API can give the reason after the job dead.

Bug-fixes:

• Table name case problem #441-1
• use mapping job.json failed when test increment , the job is 'complete' #447
• DTLE:after job create ,insert data to null schema in Src, the dtle 'panic' #450
• Mapping：After change name ,delete src table ,the Dest table can not be deleted high #453
• docker run failed to start #454
• kafka: unknown table structure #459

dtle 2.19.03.1

Release Note

• Fix that in some case table cannot be replicated. (#441)
• Different character_set in src/dst mysql causes inconsistency. (#350)

Pre-built Binary

rpm or tarball as below
docker image docker pull actiontech/dtle:2.19.03.1

dtle 2.19.03.0

Release Note

Features:

• Fine-grained DDL filtering (#392)
  • You can specify type of alter table (add|drop|modify|change|alter column)
• schema.table name conversion (alpha) (#231) (doc)

Bug-fixes:

• Consistent DDL meta-data (#321)
  • When doing an incremental copy (binlog streaming), upon a DDL,
    table structure info in dtle will be correctly updated. See demo
    test case in the issue.
• Job status will be set to dead if conection to dest mysqld is broken (#414).
• Closing a session with a temporary table now will not cause the job dead (#384).
• Detect binlog dump threads (source mysqld) aliveness (#416).
• Advoid binlog dump thread leakage (#220).
• Fix 'systemctl cannot start dtle' (#383).
• Choose PK for paging (in full copying) when there are multiple unique keys (#415).

Pre-built Binary

• rpm or tarball as below
• docker image docker pull actiontech/dtle:2.19.03.0

dtle 2.19.01.0

Release Note

• Keep connections alive (#376 )
• Refine Kafka output for various column types (#331 )
• (beta) Event filter (#360 #392 )
• (beta) Fix slow count row for large table (#362 )

Prebuilt Binary

Provided as rpm, tarball and docker image

docker pull actiontech/dtle:2.19.01.0

dtle 2.18.11.0

Release Note

• Fix an error when using TiDB as a target (#340).
• Fix gtid_executed overflow when handling intermittent GTID (#342).
• Eliminate the mechanism of lookup for AdvertiseAddr.
• Fix some racing conditions
• About where predicate
  • use inclusive between and
  • support text type
  • some bug fixes

Prebuilt Binary

Provided as rpm, tarball and docker image

docker pull actiontech/dtle:2.18.11.0