Untitled: White board for stacking my life

Ref

https://help.ubuntu.com/community/HighlyAvailableNFS

Add hosts and install packages on each nodes

# vi /etc/hosts

[IPADDR1]    nfs1
[IPADDR2]    nfs2

# sudo apt-get install ntp drbd8-utils heartbeat

Create drbd config named 'nfs' on each nodes

# vi /etc/drbd.d/nfs.res
resource nfs {
        protocol C;

        handlers {
                pri-on-incon-degr "echo o > /proc/sysrq-trigger ; halt -f";
                pri-lost-after-sb "echo o > /proc/sysrq-trigger ; halt -f";
                local-io-error "echo o > /proc/sysrq-trigger ; halt -f";
                outdate-peer "/usr/lib/heartbeat/drbd-peer-outdater -t 5";
        }

        startup {
                degr-wfc-timeout 120;
        }

        disk {
                on-io-error detach;
                no-disk-flushes ;
                no-disk-barrier;
                c-plan-ahead 0;
                c-fill-target 1M;
                c-min-rate 180M;
                c-max-rate 720M;
        }

        net {
                cram-hmac-alg sha1;
                shared-secret "PASSWORD";
                after-sb-0pri disconnect;
                after-sb-1pri disconnect;
                after-sb-2pri disconnect;
                rr-conflict disconnect;
                max-buffers 40k;
                sndbuf-size 0;
                rcvbuf-size 0;
        }

        syncer {
                rate 210M;
                verify-alg sha1;
                al-extents 3389;
        }

        on nfs1 {
                device  /dev/drbd0;
                disk    /dev/sdb1;
                address IP:7788;
                meta-disk internal;
        }

        on nfs2 {
                device  /dev/drbd0;
                disk    /dev/sdb1;
                address IP:7788;
                meta-disk internal;
        }
}

Setup DRBD on each nodes

# sudo chgrp haclient /sbin/drbdsetup
# sudo chmod o-x /sbin/drbdsetup
# sudo chmod u+s /sbin/drbdsetup
# sudo chgrp haclient /sbin/drbdmeta
# sudo chmod o-x /sbin/drbdmeta
# sudo chmod u+s /sbin/drbdmeta

# sudo drbdadm create-md nfs

Master node

# sudo drbdadm -- --overwrite-data-of-peer primary nfs

Check Primary/Secondary state and sync progress

# cat /proc/drbd

After sync completed on both node, configure NFS

Test data sync

# sudo mkfs.ext4 /dev/drdb0
# mkdir -p /srv/data
# sudo apt-get install nfs-kernel-server

Master node again

# sudo mount /dev/drbd0 /srv/data
# sudo mv /var/lib/nfs/ /srv/data/
# sudo ln -s /srv/data/nfs/ /var/lib/nfs
# sudo mv /etc/exports /srv/data
# sudo ln -s /srv/data/exports /etc/exports

Slave node

# sudo rm -rf /var/lib/nfs
# sudo ln -s /srv/data/nfs/ /var/lib/nfs
# sudo rm /etc/exports
# sudo ln -s /srv/data/exports /etc/exports

Configure HEARTBEAT on both node

# vi /etc/heartbeat/ha.cf
logfacility local0
keepalive 2
deadtime 30
warntime 10
initdead 120
bcast enp1s0f0np0
node nfs1
node nfs2


# sudo vi /etc/heartbeat/authkeys

auth 3
3 md5 PASSWORD



# sudo chmod 600 /etc/heartbeat/authkeys
# vi /etc/heartbeat/haresources

nfs1 IPaddr::NFS_MASTER_IP/17/IF_NAME drbddisk::nfs Filesystem::/dev/drbd0::/srv/data::ext4 nfs-kernel-server



# sudo systemctl enable heartbeat

# sudo systemctl enable drbd

# sudo reboot now

리눅스 커널 이미지 업데이트 시 systemd에서 Network Service 재시작이 발생할 수 있음

리눅스 Heartbeat 서비스를 사용한 Active / Stand-by 구성에서 해당 작업 시 순차적으로 진행해야 하며, Ansible 등의 병렬 작업을 할 경우, 동시에 네트워크 서비스가 재시작 되며 fail over가 정상적으로 동작하지 않을 수 있음.

When updating linux kernel image with 'apt-get upgrade' on ubuntu system, you should aware of that the network service on the host could be restarted. This could be disaster to some network topology. For example, doing some task something like Ansible which execute jobs paratactically on Active-Standby system with Heartbeat service. That may cause restarting network service at subjects at the same time.