标题：*
140 字

TAG标签：（用空格隔开）
30 字

恢复历史版本：

HTML转MD HTML转MD2

<h2><a href="https://www.jianshu.com/p/926956e7c058">https://www.jianshu.com/p/926956e7c058</a></h2><h2>准备</h2><ul class=" list-paddingleft-2"><li>MySQL 5.6</li><li>Canal 1.0.24</li><li>Kafka 1.0.0</li><li>Zookeeper 3.4.11</li><li>Spring Boot</li><li>JDK 8</li><li>IntelliJ Idea 14.1.7</li></ul><h2>背景</h2>在实际工作中，多个系统使用相同的用户等业务数据是常有的事情。随着需求的迭代式开发，系统的增加。系统的业务数据变更通知逐渐成为痛点，将数据变更逐一通知到其他系统在系统少的时候勉强可以接受，但是系统变多，需要通知的业务数据增多再使用此类的方式无疑是得不偿失的，且极不易维护，耦合性太强，牵一发而动全身。此时，引入中间件无疑是一种明智的选择。将系统从繁杂的通知中解脱出来，把通知这件事交给中间件来做。而这是基于业务操作最终会落实到数据库中，这也是本文方案可行的前提。<h2>架构图</h2><img data-original-src="//upload-images.jianshu.io/upload_images/2393477-8eff25124078858a.png" data-original-width="733" data-original-height="495" data-original-format="image/png" data-original-filesize="58174" class="" data-image-index="0" style="cursor: zoom-in;" src="//upload-images.jianshu.io/upload_images/2393477-8eff25124078858a.png?imageMogr2/auto-orient/strip|imageView2/2/w/733/format/webp"/>image<h2>实现</h2><h3>Canal Server搭建</h3><h4>MySQL配置</h4><ul class=" list-paddingleft-2"><li>进入C:\Program Files\MySQL\MySQL Server 5.6目录</li><li>编辑my.ini文件（按需修改），打开mysql bin log功能，如下</li></ul><pre class="line-numbers language-ini">[mysqld]
#&nbsp;Remove&nbsp;leading&nbsp;#&nbsp;and&nbsp;set&nbsp;to&nbsp;the&nbsp;amount&nbsp;of&nbsp;RAM&nbsp;for&nbsp;the&nbsp;most&nbsp;important&nbsp;data
#&nbsp;cache&nbsp;in&nbsp;MySQL.&nbsp;Start&nbsp;at&nbsp;70%&nbsp;of&nbsp;total&nbsp;RAM&nbsp;for&nbsp;dedicated&nbsp;server,&nbsp;else&nbsp;10%.
#&nbsp;innodb_buffer_pool_size&nbsp;=&nbsp;128M

#&nbsp;Remove&nbsp;leading&nbsp;#&nbsp;to&nbsp;turn&nbsp;on&nbsp;a&nbsp;very&nbsp;important&nbsp;data&nbsp;integrity&nbsp;option:&nbsp;logging
#&nbsp;changes&nbsp;to&nbsp;the&nbsp;binary&nbsp;log&nbsp;between&nbsp;backups.
#&nbsp;log_bin

#&nbsp;These&nbsp;are&nbsp;commonly&nbsp;set,&nbsp;remove&nbsp;the&nbsp;#&nbsp;and&nbsp;set&nbsp;as&nbsp;required.
basedir&nbsp;=&nbsp;C:\Program&nbsp;Files\MySQL\MySQL&nbsp;Server&nbsp;5.6
datadir&nbsp;=&nbsp;C:\Program&nbsp;Files\MySQL\MySQL&nbsp;Server&nbsp;5.6\data
#&nbsp;port&nbsp;=&nbsp;.....
#&nbsp;server_id&nbsp;=&nbsp;.....

log-bin=mysql-bin&nbsp;&nbsp;#添加这一行即可
#选择row模式
binlog-format=ROW
#配置mysql&nbsp;replaction需要定义，不能和canal的slaveId重复
server_id=1&nbsp;
#&nbsp;Remove&nbsp;leading&nbsp;#&nbsp;to&nbsp;set&nbsp;options&nbsp;mainly&nbsp;useful&nbsp;for&nbsp;reporting&nbsp;servers.
#&nbsp;The&nbsp;server&nbsp;defaults&nbsp;are&nbsp;faster&nbsp;for&nbsp;transactions&nbsp;and&nbsp;fast&nbsp;SELECTs.
#&nbsp;Adjust&nbsp;sizes&nbsp;as&nbsp;needed,&nbsp;experiment&nbsp;to&nbsp;find&nbsp;the&nbsp;optimal&nbsp;values.
#&nbsp;join_buffer_size&nbsp;=&nbsp;128M
#&nbsp;sort_buffer_size&nbsp;=&nbsp;2M
#&nbsp;read_rnd_buffer_size&nbsp;=&nbsp;2M&nbsp;

sql_mode=NO_ENGINE_SUBSTITUTION,STRICT_TRANS_TABLES</pre><ul class=" list-paddingleft-2"><li>canal的原理是模拟自己为mysql slave，所以这里一定需要拥有做为mysql slave的相关权限（canal server配置需要用到此用户）</li></ul><pre class="line-numbers language-sql">CREATE&nbsp;USER&nbsp;canal&nbsp;IDENTIFIED&nbsp;BY&nbsp;&#39;canal&#39;;&nbsp;&nbsp;GRANT&nbsp;SELECT,&nbsp;REPLICATION&nbsp;SLAVE,&nbsp;REPLICATION&nbsp;CLIENT&nbsp;ON&nbsp;*.*&nbsp;TO&nbsp;&#39;canal&#39;@&#39;%&#39;;--&nbsp;GRANT&nbsp;ALL&nbsp;PRIVILEGES&nbsp;ON&nbsp;*.*&nbsp;TO&nbsp;&#39;canal&#39;@&#39;%&#39;&nbsp;;FLUSH&nbsp;PRIVILEGES;</pre><ul class=" list-paddingleft-2"><li>重启mysql server</li><li>查看bin log状态</li></ul><pre class="line-numbers language-sql">show&nbsp;binlog&nbsp;events;</pre><img data-original-src="//upload-images.jianshu.io/upload_images/2393477-4c0eecab96e54f18.png" data-original-width="891" data-original-height="48" data-original-format="image/png" data-original-filesize="4915" class="" data-image-index="1" style="cursor: zoom-in;" src="//upload-images.jianshu.io/upload_images/2393477-4c0eecab96e54f18.png?imageMogr2/auto-orient/strip|imageView2/2/w/891/format/webp"/>image<ul class=" list-paddingleft-2"><li>获取canalserver（<a href="https://link.jianshu.com?t=https://github.com/alibaba/canal/releases/download/canal-1.0.24/canal.deployer-1.0.24.tar.gz" target="_blank" rel="nofollow">https://github.com/alibaba/canal/releases/download/canal-1.0.24/canal.deployer-1.0.24.tar.gz</a>）</li><li>修改canal server配置</li><ul class=" list-paddingleft-2" style="list-style-type: square;"><li>进入D:\MySQL_Sync_Component\canal.deployer-1.0.24\conf\example目录</li><li>修改canal.properties文件</li><li>修改instance.properties文件</li></ul></ul><pre class="line-numbers language-properties">#canal.properties
##配置zookeeper地址
canal.zkServers=127.0.0.1:2181</pre><pre class="line-numbers language-properties">#################################################
#instance.properties
##&nbsp;mysql&nbsp;serverId
canal.instance.mysql.slaveId&nbsp;=&nbsp;1234

#&nbsp;position&nbsp;info,需要改成自己的数据库信息
canal.instance.master.address&nbsp;=&nbsp;127.0.0.1:3306
canal.instance.master.journal.name&nbsp;=&nbsp;
canal.instance.master.position&nbsp;=&nbsp;
canal.instance.master.timestamp&nbsp;=&nbsp;

#canal.instance.standby.address&nbsp;=&nbsp;
#canal.instance.standby.journal.name&nbsp;=
#canal.instance.standby.position&nbsp;=&nbsp;
#canal.instance.standby.timestamp&nbsp;=&nbsp;

#&nbsp;username/password,需要改成自己的数据库信息
canal.instance.dbUsername&nbsp;=&nbsp;canal
canal.instance.dbPassword&nbsp;=&nbsp;canal
canal.instance.defaultDatabaseName&nbsp;=
canal.instance.connectionCharset&nbsp;=&nbsp;UTF-8

#&nbsp;table&nbsp;regex
canal.instance.filter.regex&nbsp;=&nbsp;.*\\..*
#&nbsp;table&nbsp;black&nbsp;regex
canal.instance.filter.black.regex&nbsp;=&nbsp;&nbsp;

#################################################</pre><ul class=" list-paddingleft-2"><li>启动canal server</li><ul class=" list-paddingleft-2" style="list-style-type: square;"><li>进入D:\MySQL_Sync_Component\canal.deployer-1.0.24\bin目录</li><li>运行startup.bat</li></ul><li>查看canal server日志</li><ul class=" list-paddingleft-2" style="list-style-type: square;"><li>D:\MySQL_Sync_Component\canal.deployer-1.0.24\logs\canal\canal.log</li><li>D:\MySQL_Sync_Component\canal.deployer-1.0.24\logs\example\example.log</li></ul></ul><pre class="line-numbers language-log">canal.log

2017-11-24&nbsp;17:55:32.550&nbsp;[main]&nbsp;INFO&nbsp;&nbsp;com.alibaba.otter.canal.deployer.CanalLauncher&nbsp;-&nbsp;##&nbsp;start&nbsp;the&nbsp;canal&nbsp;server.
2017-11-24&nbsp;17:55:32.770&nbsp;[main]&nbsp;INFO&nbsp;&nbsp;com.alibaba.otter.canal.deployer.CanalController&nbsp;-&nbsp;##&nbsp;start&nbsp;the&nbsp;canal&nbsp;server[192.168.191.1:11111]
2017-11-24&nbsp;17:55:34.503&nbsp;[main]&nbsp;INFO&nbsp;&nbsp;com.alibaba.otter.canal.deployer.CanalLauncher&nbsp;-&nbsp;##&nbsp;the&nbsp;canal&nbsp;server&nbsp;is&nbsp;running&nbsp;now&nbsp;......</pre><pre class="line-numbers language-log">example.log

2017-11-24&nbsp;17:55:33.202&nbsp;[main]&nbsp;INFO&nbsp;&nbsp;c.a.o.c.i.spring.support.PropertyPlaceholderConfigurer&nbsp;-&nbsp;Loading&nbsp;properties&nbsp;file&nbsp;from&nbsp;class&nbsp;path&nbsp;resource&nbsp;[canal.properties]
2017-11-24&nbsp;17:55:33.209&nbsp;[main]&nbsp;INFO&nbsp;&nbsp;c.a.o.c.i.spring.support.PropertyPlaceholderConfigurer&nbsp;-&nbsp;Loading&nbsp;properties&nbsp;file&nbsp;from&nbsp;class&nbsp;path&nbsp;resource&nbsp;[example/instance.properties]
2017-11-24&nbsp;17:55:33.343&nbsp;[main]&nbsp;WARN&nbsp;&nbsp;org.springframework.beans.TypeConverterDelegate&nbsp;-&nbsp;PropertyEditor&nbsp;[com.sun.beans.editors.EnumEditor]&nbsp;found&nbsp;through&nbsp;deprecated&nbsp;global&nbsp;PropertyEditorManager&nbsp;fallback&nbsp;-&nbsp;consider&nbsp;using&nbsp;a&nbsp;more&nbsp;isolated&nbsp;form&nbsp;of&nbsp;registration,&nbsp;e.g.&nbsp;on&nbsp;the&nbsp;BeanWrapper/BeanFactory!
2017-11-24&nbsp;17:55:33.471&nbsp;[main]&nbsp;INFO&nbsp;&nbsp;c.a.otter.canal.instance.spring.CanalInstanceWithSpring&nbsp;-&nbsp;start&nbsp;CannalInstance&nbsp;for&nbsp;1-example&nbsp;
2017-11-24&nbsp;17:55:33.647&nbsp;[main]&nbsp;INFO&nbsp;&nbsp;c.a.otter.canal.instance.core.AbstractCanalInstance&nbsp;-&nbsp;start&nbsp;successful....
2017-11-24&nbsp;17:55:33.896&nbsp;[destination&nbsp;=&nbsp;example&nbsp;,&nbsp;address&nbsp;=&nbsp;/127.0.0.1:3306&nbsp;,&nbsp;EventParser]&nbsp;WARN&nbsp;&nbsp;c.a.otter.canal.parse.inbound.mysql.MysqlEventParser&nbsp;-&nbsp;prepare&nbsp;to&nbsp;find&nbsp;start&nbsp;position&nbsp;just&nbsp;show&nbsp;master&nbsp;status</pre><h3>Zookeeper Server搭建</h3><ul class=" list-paddingleft-2"><li>获取并配置zookeeper（<a href="https://link.jianshu.com?t=http://www.apache.org/dyn/closer.cgi/zookeeper/" target="_blank" rel="nofollow">http://www.apache.org/dyn/closer.cgi/zookeeper/</a>）</li><li>复制D:\MySQL_Sync_Component\zookeeper\zookeeper-3.4.11\conf\zoo_sample.cfg并重命名为zoo.cfg,zookeeper读取的是zoo.cfg这个配置文件</li><li>启动zookeeper，进入D:\MySQL_Sync_Component\zookeeper\zookeeper-3.4.11\bin，执行</li></ul><pre class="line-numbers language-css">zkServer.cmd</pre><h3>Kafka Server搭建</h3><ul class=" list-paddingleft-2"><li>获取Kafaka Server（<a href="https://link.jianshu.com?t=https://www.apache.org/dyn/closer.cgi?path=/kafka/1.0.0/kafka_2.11-1.0.0.tgz" target="_blank" rel="nofollow">https://www.apache.org/dyn/closer.cgi?path=/kafka/1.0.0/kafka_2.11-1.0.0.tgz</a>）</li><li>配置Kafka Server</li><ul class=" list-paddingleft-2" style="list-style-type: square;"><li>修改D:\MySQL_Sync_Component\kafka\kafka_2.12-1.0.0\config\server.properties</li></ul></ul><pre class="line-numbers language-properties">#server.properteis#
#配置zookeeper连接地址
zookeeper.connect=localhost:2181</pre><ul class=" list-paddingleft-2"><li>启动Kafka服务，进入D:\MySQL_Sync_Component\kafka\kafka_2.12-1.0.0\bin\windows，执行</li></ul><pre class="line-numbers language-css">kafka-server-start.bat&nbsp;D:\MySQL_Sync_Component\kafka\kafka_2.12-1.0.0\bin\conf\server.properties</pre><ul class=" list-paddingleft-2"><li>创建topic，进入D:\MySQL_Sync_Component\kafka\kafka_2.12-1.0.0\bin\windows，执行</li></ul><pre class="line-numbers language-css">kafka-topics.bat&nbsp;--create&nbsp;--zookeeper&nbsp;localhost:2181&nbsp;--replication-factor&nbsp;1&nbsp;--partitions&nbsp;1&nbsp;--topic&nbsp;testbinlog</pre><ul class=" list-paddingleft-2"><li>启动一个kafka生产者，用于测试kafka是否启动成功。执行</li></ul><pre class="line-numbers language-css">kafka-console-producer.bat&nbsp;--broker-list&nbsp;localhost:9092&nbsp;--topic&nbsp;testbinlog</pre><ul class=" list-paddingleft-2"><li>启动一个kafka消费者，接收生产者发送的消息。测试通过，consumer可以收到producer发送的消息。执行</li></ul><pre class="line-numbers language-css">kafka-console-consumer.bat&nbsp;--bootstrap-server&nbsp;localhost:9092&nbsp;--topic&nbsp;testbinlog&nbsp;--from-beginning</pre><h3>创建Spring Boot应用</h3><ul class=" list-paddingleft-2"><li>新建一个Spring Boot应用（<a href="https://link.jianshu.com?t=http://projects.spring.io/spring-boot/#quick-start" target="_blank" rel="nofollow">http://projects.spring.io/spring-boot/#quick-start</a>）</li><li>pom文件增加kafka依赖</li></ul><pre class="line-numbers language-xml">&lt;dependency&gt;
&nbsp;&nbsp;&nbsp;&nbsp;&lt;groupId&gt;org.springframework.kafka&lt;/groupId&gt;
&nbsp;&nbsp;&nbsp;&nbsp;&lt;artifactId&gt;spring-kafka&lt;/artifactId&gt;&lt;/dependency&gt;</pre><ul class=" list-paddingleft-2"><li>编写kafka生产者和消费者代码，代码在（<a href="https://link.jianshu.com?t=https://github.com/hxysea/mynote/tree/master/%E9%A1%B9%E7%9B%AE%E7%BB%8F%E9%AA%8C/Mysql%E6%95%B0%E6%8D%AE%E5%90%8C%E6%AD%A5%E8%A7%A3%E5%86%B3%E6%96%B9%E6%A1%88/DataSyncApplication" target="_blank" rel="nofollow">https://github.com/hxysea/mynote/tree/master/%E9%A1%B9%E7%9B%AE%E7%BB%8F%E9%AA%8C/Mysql%E6%95%B0%E6%8D%AE%E5%90%8C%E6%AD%A5%E8%A7%A3%E5%86%B3%E6%96%B9%E6%A1%88/DataSyncApplication</a>）</li><li>启动应用</li></ul><h3>总结</h3>如上，已经介绍了整个解决方案所需组件的配置及启动方法。文中的解决方案用于业务数据通知或者缓存刷新等场景。本文除了提供一种业务场景的解决方案以外，更多的是希望读者可以从中领会到各组件的设计思想及其优劣势。除了可以熟练配置及使用各组件，这些组件的编码思想及设计模式更是我辈学习的典范。对于各组件的使用，最好先去阅读官方文档，然后参考相关博客，再编码实践，最后阅读下源码，达到融会贯通的境界。