Openstack Kolla

List of Hosts:

10.240.169.3: “MAAS, docker, openstack-kolla, kolla” all installed on here

Local Docker Registry

To install multi-node docker openstack, we need to have local registry service, Nexus3 is a GUI visible easy to use registry server.
install it via docker,
create ./nexus3/data/docker-compose.yml

=====================================
nexus:
image: sonatype/nexus3:latest
ports:
– “8081:8081”
– “5000:5000”
volumes:
– ./data:/nexus-data
=======================================

and then “docker-compose up -d” to create docker container. May need to pip install docker-compose.

Launch web browser to 10.240.169.3(docker host):8081, default account:admin/admin123, then create a new repo type hosted/docker, use port 5000 and enable docker v1.

verify on docker hosts they can login this private registry: docker login -p admin123 -u admin 10.240.169.3:5000

To pull images from internet repo to local registry

on 10.240.169.3

pip install kolla
kolla-build –base ubuntu –type source –registry 10.240.169.3:5000 –push

This will pull all available docker images from internet, and stored at local.

Prepare hosts for ceph osd

part disk label on each host:
parted /dev/sdb -s — mklabel gpt mkpart KOLLA_CEPH_OSD_BOOTSTRAP 1 -1
parted /dev/sdc -s — mklabel gpt mkpart KOLLA_CEPH_OSD_BOOTSTRAP 1 -1
parted /dev/sdd -s — mklabel gpt mkpart KOLLA_CEPH_OSD_BOOTSTRAP 1 -1
parted /dev/sde -s — mklabel gpt mkpart KOLLA_CEPH_OSD_BOOTSTRAP 1 -1
parted /dev/sdf -s — mklabel gpt mkpart KOLLA_CEPH_OSD_BOOTSTRAP 1 -1
parted /dev/sdg -s — mklabel gpt mkpart KOLLA_CEPH_OSD_BOOTSTRAP 1 -1
parted /dev/sdh -s — mklabel gpt mkpart KOLLA_CEPH_OSD_BOOTSTRAP_1 1 -1
parted /dev/sdi -s — mklabel gpt mkpart KOLLA_CEPH_OSD_BOOTSTRAP_2 1 -1
parted /dev/sdj -s — mklabel gpt mkpart KOLLA_CEPH_OSD_BOOTSTRAP_1_J 1 -1
parted /dev/sdk -s — mklabel gpt mkpart KOLLA_CEPH_OSD_BOOTSTRAP_2_J 1 -1

each host needs to install following:

apt install python-pip -y
pip install -U docker-py

apt-get install bridge-utils debootstrap ifenslave ifenslave-2.6 lsof lvm2 ntp ntpdate openssh-server sudo tcpdump python-dev vlan -y

no need to install docker.io manually, as there’s a bootstrap cmd doing this job under kolla-ansible: kolla-ansible -i multinode bootstrap-servers

if any deployment failure, copy /usr/local/share/kolla-ansible/tools/cleanup-containers to each host and run it to clean up containers and redo deploy again.

“kolla-ansible -i multinode destroy” can remove all deployed containor on all nodes, but ceph partitions will be kept. so to erase partitioned disks, run following on each host:

umount /dev/sdb1
umount /dev/sdc1
umount /dev/sdd1
umount /dev/sde1
umount /dev/sdf1
umount /dev/sdg1
umount /dev/sdh1
umount /dev/sdi1
dd if=/dev/zero of=/dev/sdb bs=512 count=1
dd if=/dev/zero of=/dev/sdc bs=512 count=1
dd if=/dev/zero of=/dev/sdd bs=512 count=1
dd if=/dev/zero of=/dev/sde bs=512 count=1
dd if=/dev/zero of=/dev/sdf bs=512 count=1
dd if=/dev/zero of=/dev/sdg bs=512 count=1
dd if=/dev/zero of=/dev/sdh bs=512 count=1
dd if=/dev/zero of=/dev/sdi bs=512 count=1
dd if=/dev/zero of=/dev/sdj bs=512 count=1
dd if=/dev/zero of=/dev/sdk bs=512 count=1

iSCSI on MDS

一篇很棒的BLOG。

CISCO关于iSCSI的指导

MDS在iSCSI中扮演一个中介角色,它一端通过FC连接STORAGE,一端通过GI口连接SERVER。在FC看来,它在和SERVER的PWWN相连(其实是MDS给的假PWWN),从SERVER端看来,它在连一个ISCSI TARGET IP或IQN。

iSCSI和ISLB的很大区别是,iSCSI initiator命令只用作修改VSAN和CHAP,而TARGET需要另外用iscsi virtual-target命令定义。islb是把这两个合并到islb initiator命令中使用。同时islb virtual-target提供更多的详细control access。

在MDS没有配置任何iSCSI initiator的情况下,如果有SERVER前来连接MDS,MDS会根据iSCSI口上是否启用动态分配PWWN功能做成反应。如果开启,就会自动分配地址。

MDS1(config)# show iscsi initiator 
iSCSI Node name is iqn.1998-01.com.vmware:53de1d20-106c-8c14-070d-0025b500010d-612838b7 
    Initiator ip addr (s): 10.150.150.10 
    iSCSI alias name:  
    Auto-created node (iSCSI)
    Node WWN is 21:09:00:0d:ec:54:63:82 (dynamic) 
    Member of vsans: 101
    Number of Virtual n_ports: 1
    Virtual Port WWN is 21:0a:00:0d:ec:54:63:82 (dynamic)
      Interface iSCSI 1/2, Portal group tag: 0x3001 
      VSAN ID 101, FCID 0x010104

iSCSI配置举例:

feature iscsi
iscsi enable module 1
开启MODULE1上的GI口的iSCSI功能,开启后自动生成GI口对应的iSCSI口,但处于SHUT状态

int iscsi 1/2
no shut

switchport proxy-initiator 这句可有可无,PROXY的作用是将多个FLOGI和FCID合并为一个

vsan database
vsan 101 interface iscsi 1/2
将新iSCSI口分配给VSAN101,后面才能和其他VSAN101中的FC上的PWWN归为一个ZONE

iscsi import target fc
将FC上连的所有PWWN对象都引入为自动创建的iSCSI TARGET

zoneset name VSAN101 vsan 101
zone name ESXi-JBOD1-D2
member pwwn 21:00:00:1d:38:1c:6f:24
FC上的STORAGE PWWN
member ip-address 10.150.150.10
SERVER端IP地址
member pwwn 21:0a:00:0d:ec:54:63:82
MDS为iSCSI 1/2自动生成的PWWN,在STORAGE看来它就是SERVER端的PWWN
member symbolic-nodename iqn.1998-01.com.vmware:53de1d20-106c-8c14-070d-0025b500010d-612838b7
SERVER端IQN
zoneset activate name VSAN101 vsan 101

iscsi save-initiator
将系统自动分配的动态SERVER PWWN存为固态PWWN,防止重启后PWWN改变

======================================================================================
以上是动态分配的配置方法,下面介绍固态
iscsi import target fc会自动将所有连入MDS的IQN请求关联,如果想针对某一TARGET IQN的连接进行限制,就要针对TARGET建立可以访问的表格
UCS BOOT from iSCSI是需要写TARGET IQN,就会用到这个

iscsi virtual-target name iqn.2014-08.lab.mds1:jbod1-d3
pwwn 21:00:00:1d:38:1c:78:fa
通过FC连接的STORAGE的PWWN
initiator ip address 10.150.150.10 255.255.255.255 permit
可以和此STORAGE连接的SERVER IP
initiator iqn.1998-01.com.vmware:53de1d20-106c-8c14-070d-0025b500010d-612838b7 permit
可以和此STORAGE连接的SERVER IQN
advertise interface g1/2
限定只能SERVER只能从G1/2连入(可以不限定)
=======================================================================================
CHAP的配置
username iscsiuser password abc123 iscsi
创建一个属于ISCSI的用户
iscsi initiator name iqn.1998-01.com.vmware:53de1d20-106c-8c14-070d-0025b500010d-612838b7
username iscsiuser
限制可以访问IQN的用户
vsan 101
可以对个别initiator指定可其所属VSAN,对initiator进行限制除了CHAP以外意义不大。

iSLB

iSCSI Server Load Balancing,作为iSCSI的高级应用,依靠VRRP支持负载均衡。设计中,系统为每个VRRP机器配置一个负荷值(METRIC,从0开始计数),负荷小的机器会在下一次分配中被分配到任务。默认VRRP MASTER为(0+),所以第一次分配永远不会分给MASTER。

Configuration Sample:

islb distribute
开启全局ISLB
int iscsi 1/2
no shut
开启G1/2 iSCSI
interface g1/2
ip add 10.150.150.5 255.255.255.0
no shut
vrrp 150
ip 10.150.150.254
no shut
配置G1/2 VRRP
islb vrrp 150 load-balance
启用VRRP 150 LB
islb commit
配置完ISLB就要COMMIT
islb initiator name iqn.1998-01.com.vmware:53de1d20-106c-8c14-070d-0025b500010d-612838b7
vsan 101
static nwwn system-assign
static pwwn system-assign 1
以ISLB替换ISCSI关键字,功能是一样的
target pwwn 22:00:00:1d:38:1c:76:db iqn-name iqn.2014-08.lab.mds.jbod1-d8-b
ISLB中的TARGET是在INIT下面定义的,和ISCSI不同

CCIE DC TIPs

FI到NEXUS的UPLINK口,在NEXUS上要spanning-tree port type edge trunk,实现快速收敛。

Gen1不支持UCS Chassis到FI之间Port Channel。只有两边都是Gen2才行。

从一个FI UPLINK口收到的流量不会从另一个UPLINK口出去。

同一CHASSIS上的同一IO只连同一个FI,不可以IO A连FI A+B。

HOST上的VIC card分1280和1240,1280从HOST到IO A/B分别有4条PATH,每条10G,一共80G带宽。1240从HOST到IO A/B分别有2条PATH,每条10G,一共40G带宽。IO分为2208和2204两种,2208有8口共80G带宽,2204有4口共40G带宽。所以用2208对外一个有160G带宽可用,而如果CHASSIS使用1280卡满负荷运行会产生8×80G=640G流量。

HOST上的STORAGE流量通过VSAN PIN GROUP从指定FI的FC口流出,从HOST到FI的路径是自动内部FCOE。FI的HOST模式自动将FI设成NPV MODE,所以要求上家FC交换机有NPIV功能。所以就是MDS要开NPIV。

要令Nexus上的Unified Port在ethernet和fc间转换需要使用命令slot然后进入各个口进行type定义,之后需要重启。开启FCOE之后就可以启用新的FC口了。

VDC ha-policy:

Default VDC不可以修改,RESET=RELOAD。

reload:重启机器,保持VDC配置不变, copy start run。

restart:不重启机器,no vdc, copy start run。

bringdown:什么都不做,只将坏的sup关停。

switchover:SUP间切换。

NPV-NPIV

NPV mode SW 没有E PORT,只有F,NP,SD。

NPIV-NPIV之间应该是E port????

qos/mtu

You can only apply input to a qos policy; you can apply both input and output to a queuing policy.

qos是给input traffic分组打TAG,queuing负责bandwidth,network-qos负责对qos分组的数据进行再塑(qos class default没有cos标,所以如果是改MTU的话应该应用在class default上)。

1000v needs mtu to be applied under port-profile and qos needs to be in service policy under port-profile; 5k needs it to be applied in network-qos under “system qos – service policy”.