1. 文章
  2. 全部

Greenplum扩容节点步骤

Stephev · 2019-03-29 14:22:30发表 · 5529 次点击
   0    0   

概述

greenplum扩容segment可以分为2类：

  • 单机上的seg增加，我们称之为“纵向扩展”
  • 单机不变，通过增加机器，我们称之为“横向扩展”

下面详细介绍下这两种扩容方式的具体步骤和方法

gpexpand命令

首先我们来学习一下 gpexpand 命令，基本使用方法如下：

  1. gpexpand 
  2.       [-f <hosts_file>]
  3.       | -i <input_file> [-B <batch_size>] [-V] 
  4.       | {-d <hh:mm:ss> | -e '<YYYY-MM-DD hh:mm:ss>'} 
  5.         [-analyze] [-n <parallel_processes>]
  6.       | --rollback
  7.       | --clean
  8. [-D <database_name>][--verbose] [--silent]

命令参数：

  2. -B <batch_size>
  3. 并发数可以指定同时运行ssh命令的数量，默认时16，参数范围是1-128
  4. 一般的默认值即可
  6. -D <database_name>
  7. 指定要在其中创建扩展模式的数据库和表。如果未提供此选项，则为使用环境变量PGDATABASE。数据库模板不能使用template1和template0。
  10. -i | --input <input_file>
  11. 指定扩展配置文件的名称
  12.   <hostname>:<address>:<port>:<fselocation>:<dbid>:<content>:<preferred_role>:<replication_port>
  13. 主机名:地址:端口:segment data目录,注意是全路径:dbid号:角色
  16. -n <parallel_processes>
  17. 同时进行重分布的表的个数，有效的参数为1-16，每一个表重分布必须要有两个数据库连接，设置之前，要确保最大连接数max_connections这个参数可以满足这个值
  19. -S | --simple_progress
  20.  简单模式，不会显示提示
  22. -t | --tardir <directory>
  23. 指定要放置tar文件的段主机上的临时目录。
  25. -v | --verbose
  26. 调试模式，打印出详细的执行计划
  28. -V | --novacuum
  29. 创建schema副本时，不对数据字典进行vacuum

纵向拓展

原Greenplum集群状态

原集群的一个master节点和2个数据节点，每一个数据节点上有4个primary计算节点（为了显示清晰，就没有增加mirror节点了）具体信息如下：

  1. postgres=# SELECT * from gp_segment_configuration ;
  2.  dbid | content | role | preferred_role | mode | status | port  | hostname | address | replication_port | san_mounts 
  3. ------+---------+------+----------------+------+--------+-------+----------+---------+------------------+------------
  4.     1 |      -1 | p    | p              | s    | u      |  5432 | gw_mdw1  | gw_mdw1 |                  | 
  5.     2 |       0 | p    | p              | s    | u      | 40000 | gw_sdw1  | gw_sdw1 |                  | 
  6.     6 |       4 | p    | p              | s    | u      | 40000 | gw_sdw2  | gw_sdw2 |                  | 
  7.     3 |       1 | p    | p              | s    | u      | 40001 | gw_sdw1  | gw_sdw1 |                  | 
  8.     7 |       5 | p    | p              | s    | u      | 40001 | gw_sdw2  | gw_sdw2 |                  | 
  9.     4 |       2 | p    | p              | s    | u      | 40002 | gw_sdw1  | gw_sdw1 |                  | 
  10.     8 |       6 | p    | p              | s    | u      | 40002 | gw_sdw2  | gw_sdw2 |                  | 
  11.     5 |       3 | p    | p              | s    | u      | 40003 | gw_sdw1  | gw_sdw1 |                  | 
  12.     9 |       7 | p    | p              | s    | u      | 40003 | gw_sdw2  | gw_sdw2 |                  | 
  13. (9 rows)

现在准备通过在每台机器上再增加1个节点，来扩容segment，简单的来说就是纵向扩容

生成参数文件

指定一个数据库，在此数据库下生成schema

再创建一个seg_hosts的文件，即所有数据节点的主机名

  1. [gpadmin@gw_mdw1 ~]$ cat seg_hosts 
  2. gw_sdw1
  3. gw_sdw2

执行命令，生成参数文件

  1. [gpadmin@gw_mdw1 ~]$ gpexpand -f seg_hosts -D test
  2. 20190327:23:18:01:007122 gpexpand:gw_mdw1:gpadmin-[INFO]:-local Greenplum Version: 'postgres (Greenplum Database) 4.3.1.0 build 6'
  3. 20190327:23:18:01:007122 gpexpand:gw_mdw1:gpadmin-[INFO]:-master Greenplum Version: 'PostgreSQL 8.2.15 (Greenplum Database 4.3.1.0 build 6) on x86_64-unknown-linux-gnu, compiled by GCC gcc (GCC) 4.4.2 compiled on Jun 11 2014 17:23:40'
  4. 20190327:23:18:01:007122 gpexpand:gw_mdw1:gpadmin-[INFO]:-Querying gpexpand schema for current expansion state
  6. System Expansion is used to add segments to an existing GPDB array.
  7. gpexpand did not detect a System Expansion that is in progress.
  9. Before initiating a System Expansion, you need to provision and burn-in
  10. the new hardware.  Please be sure to run gpcheckperf/gpcheckos to make
  11. sure the new hardware is working properly.
  13. Please refer to the Admin Guide for more information.
  15. Would you like to initiate a new System Expansion Yy|Nn (default=N):
  16. > y            <------确认添加
  18. How many new primary segments per host do you want to add? (default=0):
  19. > 1            <------每台机器上添加1个计算节点
  20. Enter new primary data directory 1:
  21. > /data/primary    <------增加的计算节点存放的目录
  23. Generating configuration file...
  25. 20190327:23:19:03:007122 gpexpand:gw_mdw1:gpadmin-[INFO]:-Generating input file...
  27. Input configuration files were written to 'gpexpand_inputfile_20190327_231903' and 'None'.
  28. Please review the file and make sure that it is correct then re-run
  29. with: gpexpand -i gpexpand_inputfile_20190327_231903 -D test
  31. 20190327:23:19:03:007122 gpexpand:gw_mdw1:gpadmin-[INFO]:-Exiting...

参数文件的内容如下：

  1. [gpadmin@gw_mdw1 ~]$ cat gpexpand_inputfile_20190327_231903 
  2. gw_sdw1:gw_sdw1:40004:/data/primary/gpseg8:10:8:p
  3. gw_sdw2:gw_sdw2:40004:/data/primary/gpseg9:11:9:p

如果存放的目录有所改变，可以手动去修改此文件，将该计算节点存放在自己想要放在的位置

利用参数文件执行拓展命令

  1. [gpadmin@gw_mdw1 ~]$ gpexpand -i gpexpand_inputfile_20190327_231903 -D test
  2. 20190327:23:20:19:007227 gpexpand:gw_mdw1:gpadmin-[INFO]:-local Greenplum Version: 'postgres (Greenplum Database) 4.3.1.0 build 6'
  3. 20190327:23:20:19:007227 gpexpand:gw_mdw1:gpadmin-[INFO]:-master Greenplum Version: 'PostgreSQL 8.2.15 (Greenplum Database 4.3.1.0 build 6) on x86_64-unknown-linux-gnu, compiled by GCC gcc (GCC) 4.4.2 compiled on Jun 11 2014 17:23:40'
  4. .......（省略信息）
  5. 20190327:23:22:19:007227 gpexpand:gw_mdw1:gpadmin-[INFO]:-************************************************
  6. 20190327:23:22:19:007227 gpexpand:gw_mdw1:gpadmin-[INFO]:-Initialization of the system expansion complete.
  7. 20190327:23:22:19:007227 gpexpand:gw_mdw1:gpadmin-[INFO]:-To begin table expansion onto the new segments
  8. 20190327:23:22:19:007227 gpexpand:gw_mdw1:gpadmin-[INFO]:-rerun gpexpand
  9. 20190327:23:22:19:007227 gpexpand:gw_mdw1:gpadmin-[INFO]:-************************************************
  10. 20190327:23:22:19:007227 gpexpand:gw_mdw1:gpadmin-[INFO]:-Exiting...

查看新添加的状态

此时去数据库里查看相关节点信息，可以看到已经增加了两个节点了。

  1. postgres=# SELECT * from gp_segment_configuration ;
  2.  dbid | content | role | preferred_role | mode | status | port  | hostname | address | replication_port | san_mounts 
  3. ------+---------+------+----------------+------+--------+-------+----------+---------+------------------+------------
  4.     1 |      -1 | p    | p              | s    | u      |  5432 | gw_mdw1  | gw_mdw1 |                  | 
  5.     2 |       0 | p    | p              | s    | u      | 40000 | gw_sdw1  | gw_sdw1 |                  | 
  6.     6 |       4 | p    | p              | s    | u      | 40000 | gw_sdw2  | gw_sdw2 |                  | 
  7.     3 |       1 | p    | p              | s    | u      | 40001 | gw_sdw1  | gw_sdw1 |                  | 
  8.     7 |       5 | p    | p              | s    | u      | 40001 | gw_sdw2  | gw_sdw2 |                  | 
  9.     4 |       2 | p    | p              | s    | u      | 40002 | gw_sdw1  | gw_sdw1 |                  | 
  10.     8 |       6 | p    | p              | s    | u      | 40002 | gw_sdw2  | gw_sdw2 |                  | 
  11.     5 |       3 | p    | p              | s    | u      | 40003 | gw_sdw1  | gw_sdw1 |                  | 
  12.     9 |       7 | p    | p              | s    | u      | 40003 | gw_sdw2  | gw_sdw2 |                  | 
  13.    10 |       8 | p    | p              | s    | u      | 40004 | gw_sdw1  | gw_sdw1 |                  | 
  14.    11 |       9 | p    | p              | s    | u      | 40004 | gw_sdw2  | gw_sdw2 |                  | 
  15. (11 rows)

数据重分布

执行重分布命令将数据重分布

  1. [gpadmin@gw_mdw1 ~]$ gpexpand -a -d 1:00:00 -D test

此时还没有完成，去指定的数据库下面查看重分布状态

  1. test=# select * from gpexpand.status;
  2.    status   |          updated           
  3. ------------+----------------------------
  4.  SETUP      | 2019-03-27 23:21:49.299565
  5.  SETUP DONE | 2019-03-27 23:21:53.737479
  6. (2 rows)

还有多少个表需要重分布，这里我的已经同步完成，因为是测试环境没有什么数据量

  1. test=# select * from gpexpand.expansion_progress ;
  2.              name             | value 
  3. ------------------------------+-------
  4.  Estimated Expansion Rate     | 
  5.  Estimated Time to Completion | 
  6. (2 rows)

完成重分布之后，将schema删除

  1. [gpadmin@gw_mdw1 ~]$ gpexpand -c -D test

横向拓展

这次我们不仅在每台机器上添加1个计算节点，还添加一个数据节点（也就是一个新机器）

生成配置文件

这次我们在主机的名单中再增加一个新的机器gw_sdw3，此机器是新添加进来的一个节点。只是安装了软件，没有任何数据。

  1. [gpadmin@gw_mdw1 ~]$ cat seg_hosts 
  2. gw_sdw1
  3. gw_sdw2
  4. gw_sdw3

使用同样的方法执行，每一节点也选择增加1个计算节点

  1. [gpadmin@gw_mdw1 ~]$ gpexpand -f seg_hosts -D test

生成文件如下：
可以看到，虽然只是添加一个，greenplum查看到gw_sdw3上并没有一个节点，就自动为其一次添加6个，是它与其他两个机器数量一致 

  1. [gpadmin@gw_mdw1 ~]$ cat gpexpand_inputfile_20190328_014748 
  2. gw_sdw3:gw_sdw3:40000:/data/primary/gpseg10:12:10:p
  3. gw_sdw3:gw_sdw3:40001:/data/primary/gpseg11:13:11:p
  4. gw_sdw3:gw_sdw3:40002:/data/primary/gpseg12:14:12:p
  5. gw_sdw3:gw_sdw3:40003:/data/primary/gpseg13:15:13:p
  6. gw_sdw3:gw_sdw3:40004:/data/primary/gpseg14:16:14:p
  7. gw_sdw1:gw_sdw1:40005:/data/primary/gpseg15:17:15:p
  8. gw_sdw2:gw_sdw2:40005:/data/primary/gpseg16:18:16:p
  9. gw_sdw3:gw_sdw3:40005:/data/primary/gpseg17:19:17:p

执行扩容命令

执行扩容命令，和上面一样

  1. [gpadmin@gw_mdw1 ~]$ gpexpand -i gpexpand_inputfile_20190328_014748 -D test
  2. 20190328:23:50:44:001498 gpexpand:gw_mdw1:gpadmin-[INFO]:-local Greenplum Version: 'postgres (Greenplum Database) 4.3.1.0 build 6'
  3. 20190328:23:50:44:001498 gpexpand:gw_mdw1:gpadmin-[INFO]:-master Greenplum Version: 'PostgreSQL 8.2.15 (Greenplum Database 4.3.1.0 build 6) on x86_64-unknown-linux-gnu, compiled by GCC gcc (GCC) 4.4.2 compiled on Jun 11 2014 17:23:40'
  4. ....(忽略信息)
  5. 20190328:23:54:59:001498 gpexpand:gw_mdw1:gpadmin-[INFO]:-Starting Greenplum Database
  6. 20190328:23:55:32:001498 gpexpand:gw_mdw1:gpadmin-[INFO]:-************************************************
  7. 20190328:23:55:32:001498 gpexpand:gw_mdw1:gpadmin-[INFO]:-Initialization of the system expansion complete.
  8. 20190328:23:55:32:001498 gpexpand:gw_mdw1:gpadmin-[INFO]:-To begin table expansion onto the new segments
  9. 20190328:23:55:32:001498 gpexpand:gw_mdw1:gpadmin-[INFO]:-rerun gpexpand
  10. 20190328:23:55:32:001498 gpexpand:gw_mdw1:gpadmin-[INFO]:-************************************************
  11. 20190328:23:55:32:001498 gpexpand:gw_mdw1:gpadmin-[INFO]:-Exiting...

查看数据库的节点状态，发现已经增加了8个节点，其中2个是纵向添加，6个是横向添加

  1. postgres=# SELECT * from gp_segment_configuration ;
  2.  dbid | content | role | preferred_role | mode | status | port  | hostname | address | replication_port | san_mounts 
  3. ------+---------+------+----------------+------+--------+-------+----------+---------+------------------+------------
  4.     1 |      -1 | p    | p              | s    | u      |  5432 | gw_mdw1  | gw_mdw1 |                  | 
  5.     2 |       0 | p    | p              | s    | u      | 40000 | gw_sdw1  | gw_sdw1 |                  | 
  6.     6 |       4 | p    | p              | s    | u      | 40000 | gw_sdw2  | gw_sdw2 |                  | 
  7.     3 |       1 | p    | p              | s    | u      | 40001 | gw_sdw1  | gw_sdw1 |                  | 
  8.     7 |       5 | p    | p              | s    | u      | 40001 | gw_sdw2  | gw_sdw2 |                  | 
  9.     4 |       2 | p    | p              | s    | u      | 40002 | gw_sdw1  | gw_sdw1 |                  | 
  10.     8 |       6 | p    | p              | s    | u      | 40002 | gw_sdw2  | gw_sdw2 |                  | 
  11.     5 |       3 | p    | p              | s    | u      | 40003 | gw_sdw1  | gw_sdw1 |                  | 
  12.     9 |       7 | p    | p              | s    | u      | 40003 | gw_sdw2  | gw_sdw2 |                  | 
  13.    10 |       8 | p    | p              | s    | u      | 40004 | gw_sdw1  | gw_sdw1 |                  | 
  14.    12 |      10 | p    | p              | s    | u      | 40000 | gw_sdw3  | gw_sdw3 |                  | 
  15.    11 |       9 | p    | p              | s    | u      | 40004 | gw_sdw2  | gw_sdw2 |                  | 
  16.    13 |      11 | p    | p              | s    | u      | 40001 | gw_sdw3  | gw_sdw3 |                  | 
  17.    14 |      12 | p    | p              | s    | u      | 40002 | gw_sdw3  | gw_sdw3 |                  | 
  18.    15 |      13 | p    | p              | s    | u      | 40003 | gw_sdw3  | gw_sdw3 |                  | 
  19.    16 |      14 | p    | p              | s    | u      | 40004 | gw_sdw3  | gw_sdw3 |                  | 
  20.    17 |      15 | p    | p              | s    | u      | 40005 | gw_sdw1  | gw_sdw1 |                  | 
  21.    18 |      16 | p    | p              | s    | u      | 40005 | gw_sdw2  | gw_sdw2 |                  | 
  22.    19 |      17 | p    | p              | s    | u      | 40005 | gw_sdw3  | gw_sdw3 |                  | 
  23. (19 rows)

重分布表

执行表的重分布命令

  1. [gpadmin@gw_mdw1 ~]$ gpexpand -a -d 1:00:00 -D test 
  2. 20190329:00:00:56:002723 gpexpand:gw_mdw1:gpadmin-[INFO]:-local Greenplum Version: 'postgres (Greenplum Database) 4.3.1.0 build 6'
  3. 20190329:00:00:56:002723 gpexpand:gw_mdw1:gpadmin-[INFO]:-master Greenplum Version: 'PostgreSQL 8.2.15 (Greenplum Database 4.3.1.0 build 6) on x86_64-unknown-linux-gnu, compiled by GCC gcc (GCC) 4.4.2 compiled on Jun 11 2014 17:23:40'
  4. 20190329:00:00:56:002723 gpexpand:gw_mdw1:gpadmin-[INFO]:-Querying gpexpand schema for current expansion state
  5. 20190329:00:01:03:002723 gpexpand:gw_mdw1:gpadmin-[INFO]:-EXPANSION COMPLETED SUCCESSFULLY
  6. 20190329:00:01:03:002723 gpexpand:gw_mdw1:gpadmin-[INFO]:-Exiting...

去指定数据库下查看，重分布状态

  1. test=# select * from gpexpand.status;
  2.        status       |          updated           
  3. --------------------+----------------------------
  4.  SETUP              | 2019-03-28 23:54:39.325544
  5.  SETUP DONE         | 2019-03-28 23:54:48.213527
  6.  EXPANSION STARTED  | 2019-03-29 00:00:58.467156
  7.  EXPANSION COMPLETE | 2019-03-29 00:00:58.979106
  8. (4 rows)

完成重分布之后，将schema删除

  1. [gpadmin@gw_mdw1 ~]$ gpexpand -c -D test

至此，两种方式的扩容测试都演示完毕

本站文章，未经作者同意，请勿转载，如需转载，请邮件customer@csudata.com.
0 评论  
添加一条新评论

用户登录

个人首页
立即登录