Update CubeOS
Update CubeOS
USB drive
copy CUBE_2.0.0_20201125-2132_a4d7645.pkg into usb drive (format: fat or exfat)
Plug the usb to the server
SCP
upload CUBE_2.0.0_20201125-2132_a4d7645.pkg to your server over scp, directory: /var/update
$ scp CUBE_2.0.0_20201125-2132_a4d7645.pkg [email protected]:/var/update/
Connect to console
$ ssh [email protected]
Warning: Permanently added '10.32.2.220' (ECDSA) to the list of known hosts.
Password:
Run Health Check
CLI: cluster> health
sky1> cluster
sky1:cluster> health
[ ClusterLink ]
link:
Ping 10.32.2.220 ... OK
[ ClusterSettings ]
etcd:
http://10.32.2.220:12379 is healthy: successfully committed proposal: took = 1.020481ms
[ HaCluster ]
hacluster:
Cluster name: cube-9aid0G8SFk65gm93
Cluster Summary:
* Stack: corosync
* Current DC: sky1 (version 2.0.3-5.el8_2.1-4b1f869f0f) - partition with quorum
* Last updated: Mon Dec 7 16:19:11 2020
* Last change: Sun Nov 8 23:33:16 2020 by root via cibadmin on sky1
* 1 node configured
* 0 resource instances configured
Node List:
* Online: [ sky1 ]
Full List of Resources:
* No resources
Daemon Status:
corosync: active/disabled
pacemaker: active/disabled
pcsd: active/disabled
[ MsgQueue ]
rabbitmq:
Cluster status of node rabbit@sky1 ...
Basics
Cluster name: rabbit@localhost
Disk Nodes
rabbit@sky1
Running Nodes
rabbit@sky1
Versions
rabbit@sky1: RabbitMQ 3.8.3 on Erlang 22.3.4.1
Alarms
(none)
Network Partitions
(none)
Listeners
Node: rabbit@sky1, interface: [::], port: 25672, protocol: clustering, purpose: inter-node and CLI tool communication
Node: rabbit@sky1, interface: 10.32.2.220, port: 5672, protocol: amqp, purpose: AMQP 0-9-1 and AMQP 1.0
Feature flags
Flag: implicit_default_bindings, state: enabled
Flag: quorum_queue, state: enabled
Flag: virtual_host_metadata, state: enabled
Timeout: 60.0 seconds ...
Listing queues for vhost / ...
name pid synchronised_slave_pids
compute_fanout_953e54e3b2544147805392b69a61028d <[email protected]>
engine_worker.afe7a13c-d807-4ceb-9526-2dc9263c9828 <[email protected]>
scheduler_fanout_1d604aeb42f14b2daad6e4b7c90da789 <[email protected]>
[ IaasDb ]
mysql:
wsrep_cluster_status Disconnected
wsrep_cluster_size 0
[ VirtualIp ]
vip:
non-HA
haproxy_ha:
non-HA
[ Storage ]
ceph:
cluster:
id: c6e64c49-09cf-463b-9d1c-b6645b4b3b85
health: HEALTH_OK
services:
mon: 1 daemons, quorum sky1 (age 4w)
mgr: sky1(active, since 4w)
mds: cephfs:1 {0=sky1=up:active}
osd: 20 osds: 20 up (since 4w), 20 in (since 4w)
rgw: 1 daemon active (sky1)
task status:
scrub status:
mds.sky1: idle
data:
pools: 23 pools, 720 pgs
objects: 93.53k objects, 387 GiB
usage: 379 GiB used, 30 TiB / 30 TiB avail
pgs: 720 active+clean
io:
client: 121 KiB/s rd, 696 KiB/s wr, 214 op/s rd, 111 op/s wr
[ ApiService ]
haproxy:
(sky1)
# pxname svname check_status
openstack_horizon localhost L4OK
grafana_backend localhost L4OK
kibana_backend localhost L4OK
cube_lmi localhost L4OK
swift_radosgw localhost L4OK
httpd:
+------------+--------------+------------+
| Status | API Endpoint | Error Code |
+------------+--------------+------------+
| healthy | available | 0 |
+------------+--------------+------------+
lmi:
+------------+--------------+------------+
| Status | API Endpoint | Error Code |
+------------+--------------+------------+
| healthy | available | 0 |
+------------+--------------+------------+
memcache:
+------------+------------+
| Status | Error Code |
+------------+------------+
| healthy | 0 |
+------------+------------+
[ Compute ]
nova:
+----+----------------+------+----------+---------+-------+----------------------------+
| ID | Binary | Host | Zone | Status | State | Updated At |
+----+----------------+------+----------+---------+-------+----------------------------+
| 10 | nova-scheduler | sky1 | internal | enabled | up | 2020-12-07T08:19:23.000000 |
| 16 | nova-conductor | sky1 | internal | enabled | up | 2020-12-07T08:19:18.000000 |
| 20 | nova-compute | sky1 | nova | enabled | up | 2020-12-07T08:19:16.000000 |
+----+----------------+------+----------+---------+-------+----------------------------+
[ Network ]
neutron:
+--------------------------------------+------------------------------+------+-------------------+-------+-------+-------------------------------+
| ID | Agent Type | Host | Availability Zone | Alive | State | Binary |
+--------------------------------------+------------------------------+------+-------------------+-------+-------+-------------------------------+
| aca5c90d-d014-4435-9f5b-e59822e046b3 | OVN Controller Gateway agent | sky1 | n/a | :-) | UP | ovn-controller |
| a1bdab2a-630e-4b8e-95b2-7cec8498544d | OVN Metadata agent | sky1 | n/a | :-) | UP | networking-ovn-metadata-agent |
+--------------------------------------+------------------------------+------+-------------------+-------+-------+-------------------------------+
[ Image ]
glance:
+------------+--------------+------------+
| Status | API Endpoint | Images |
+------------+--------------+------------+
| healthy | available | 10 |
+------------+--------------+------------+
[ BlockStor ]
cinder:
+------------------+-----------+------+---------+-------+----------------------------+
| Binary | Host | Zone | Status | State | Updated At |
+------------------+-----------+------+---------+-------+----------------------------+
| cinder-backup | sky1 | nova | enabled | up | 2020-12-07T08:19:30.000000 |
| cinder-volume | cube@ceph | nova | enabled | up | 2020-12-07T08:19:33.000000 |
| cinder-scheduler | sky1 | nova | enabled | up | 2020-12-07T08:19:29.000000 |
+------------------+-----------+------+---------+-------+----------------------------+
[ FileStor ]
manila:
+----+------------------+--------------+------+---------+-------+----------------------------+
| Id | Binary | Host | Zone | Status | State | Updated_at |
+----+------------------+--------------+------+---------+-------+----------------------------+
| 1 | manila-scheduler | sky1 | nova | enabled | up | 2020-12-07T08:19:31.000000 |
| 2 | manila-share | sky1@generic | nova | enabled | up | 2020-12-07T08:19:38.000000 |
+----+------------------+--------------+------+---------+-------+----------------------------+
[ ObjectStor ]
swift:
+------------+--------------+--------+------------+------------+
| Status | API Endpoint | Tenant | Containers | Objects |
+------------+--------------+--------+------------+------------+
| healthy | available | admin | 0 | 0 |
+------------+--------------+--------+------------+------------+
[ Orchestration ]
heat:
+------------+--------------+------------------+
| Status | API Endpoint | Engine (Up/Down) |
+------------+--------------+------------------+
| healthy | available | 5/0 |
+------------+--------------+------------------+
[ LBaaS ]
octavia:
+------------+--------------+------------+
| Status | API Endpoint | Error Code |
+------------+--------------+------------+
| healthy | available | 0 |
+------------+--------------+------------+
[ DNSaaS ]
designate:
+------------+--------------+------------+
| Status | API Endpoint | Error Code |
+------------+--------------+------------+
| healthy | available | 0 |
+------------+--------------+------------+
+--------------------------------------+----------+--------------+--------+-------+--------------+
| id | hostname | service_name | status | stats | capabilities |
+--------------------------------------+----------+--------------+--------+-------+--------------+
| 3b3d3977-df4d-4842-8072-c03c5949706d | sky1 | api | UP | - | - |
| 69a20406-a2f7-4c7e-a5a7-b9dbc99142d4 | sky1 | central | UP | - | - |
| 82b68fc4-e7ff-4c50-bb5c-3fd0ffaa381d | sky1 | worker | UP | - | - |
| 9c7113c9-fec9-4ad5-87fa-9772569934ba | sky1 | mdns | UP | - | - |
| dbb4cf77-7a1b-4c43-96d7-3132050393bc | sky1 | producer | UP | - | - |
+--------------------------------------+----------+--------------+--------+-------+--------------+
[ K8SaaS ]
k3s:
NAME STATUS ROLES AGE VERSION INTERNAL-IP EXTERNAL-IP OS-IMAGE KERNEL-VERSION CONTAINER-RUNTIME
sky1 Ready master 28d v1.18.8+k3s1 10.32.2.220 <none> CentOS Linux 8 (Core) 4.18.0-193.19.1.el8_2.x86_64 containerd://1.3.3-k3s2
rancher:
NAMESPACE NAME READY STATUS RESTARTS AGE
kube-system pod/local-path-provisioner-6d59f47c7-8wsz6 1/1 Running 0 28d
kube-system pod/metrics-server-7566d596c8-zpddc 1/1 Running 0 28d
kube-system pod/coredns-7944c66d8d-nrdq7 1/1 Running 0 28d
cattle-system pod/rancher-5c5fb59466-rmmr7 1/1 Running 1 28d
cattle-system pod/rancher-5c5fb59466-hh8nl 1/1 Running 0 28d
cattle-system pod/rancher-5c5fb59466-4wks6 1/1 Running 0 28d
NAMESPACE NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
default service/kubernetes ClusterIP 10.43.0.1 <none> 443/TCP 28d
kube-system service/kube-dns ClusterIP 10.43.0.10 <none> 53/UDP,53/TCP,9153/TCP 28d
kube-system service/metrics-server ClusterIP 10.43.177.106 <none> 443/TCP 28d
cattle-system service/rancher ClusterIP 10.43.236.92 <none> 80/TCP 28d
cattle-system service/rancher-nodeport NodePort 10.43.92.136 <none> 443:30443/TCP 28d
NAMESPACE NAME READY UP-TO-DATE AVAILABLE AGE
kube-system deployment.apps/local-path-provisioner 1/1 1 1 28d
kube-system deployment.apps/metrics-server 1/1 1 1 28d
kube-system deployment.apps/coredns 1/1 1 1 28d
cattle-system deployment.apps/rancher 3/3 3 3 28d
NAMESPACE NAME DESIRED CURRENT READY AGE
kube-system replicaset.apps/local-path-provisioner-6d59f47c7 1 1 1 28d
kube-system replicaset.apps/metrics-server-7566d596c8 1 1 1 28d
kube-system replicaset.apps/coredns-7944c66d8d 1 1 1 28d
cattle-system replicaset.apps/rancher-5c5fb59466 3 3 3 28d
[ InstanceHa ]
masakari:
+------------+--------------+------------+
| Status | API Endpoint | Error Code |
+------------+--------------+------------+
| healthy | available | 0 |
+------------+--------------+------------+
[ DisasterRecovery ]
freezer:
+------------+--------------+------------+
| Status | API Endpoint | Error Code |
+------------+--------------+------------+
| healthy | available | 0 |
+------------+--------------+------------+
[ DataPipe ]
zookeeper:
+------------+------------+
| Status | Error Code |
+------------+------------+
| healthy | 0 |
+------------+------------+
kafka:
+------------+------------+
| Status | Error Code |
+------------+------------+
| healthy | 0 |
+------------+------------+
[ Metrics ]
monasca:
+------------+------------+
| Status | Error Code |
+------------+------------+
| healthy | 0 |
+------------+------------+
telegraf:
+------------+------------+
| Status | Error Code |
+------------+------------+
| healthy | 0 |
+------------+------------+
grafana:
+------------+--------------+------------+
| Status | API Endpoint | Error Code |
+------------+--------------+------------+
| healthy | available | 0 |
+------------+--------------+------------+
[ LogAnalytics ]
filebeat:
+------------+------------+
| Status | Error Code |
+------------+------------+
| healthy | 0 |
+------------+------------+
auditbeat:
+------------+------------+
| Status | Error Code |
+------------+------------+
| healthy | 0 |
+------------+------------+
logstash:
+------------+------------+
| Status | Error Code |
+------------+------------+
| healthy | 0 |
+------------+------------+
es:
+------------+------------+
| Status | Error Code |
+------------+------------+
| healthy | 0 |
+------------+------------+
kibana:
+------------+--------------+------------+
| Status | API Endpoint | Error Code |
+------------+--------------+------------+
| healthy | available | 0 |
+------------+--------------+------------+
[ Notifications ]
influxdb:
+------------+--------------+------------+
| Status | API Endpoint | Error Code |
+------------+--------------+------------+
| healthy | available | 0 |
+------------+--------------+------------+
kapacitor:
+------------+--------------+------------+
| Status | API Endpoint | Error Code |
+------------+--------------+------------+
| healthy | available | 0 |
+------------+--------------+------------+
[ Node ]
node:
+------------+-------+---------------+---------------+
| | CPU | Disk | Memory |
+------------+-------+-------+-------+-------+-------+
| Host | Usage | Usage | Avail | Usage | Avail |
+------------+-------+-------+-------+-------+-------+
| sky1 | 27.8% | 44% | 99G | 85% | 14Gi |
+------------+-------+-------+-------+-------+-------+
Update CubeOS
sky1> update
sky1:update> update
1: local
2: usb
3: server
Enter index: 1
1: CUBE_2.0.0_20201125-2132_a4d7645.pkg
Enter index: 1
Firmware update will require an appliance reboot.
Enter 'YES' to confirm: YES
Formatting partition 2
Installing CUBE_2.0.0_20201125-2132_a4d7645
Running install script
Installing postinstall script
Finished updating. Please reboot appliance.
Reboot
sky1:update> reboot
Enter 'YES' to confirm: YES
Connection to 10.32.2.220 closed by remote host.
Connection to 10.32.2.220 closed.
Run bootstrap_cube
Welcome to the Cube Appliance
Enter "help" for a list of available commands
sky1> boot
sky1:boot> bootstrap_cube
bootstraping cube...
bootstrap successfully
if this is a single node start, just run "boot> cluster_sync"
if this is a cluster start, wait until bootstrap_done is done in all nodes and run "cluster> start" to sync cluster data for all nodes
Run cluster_sync
sky1:boot> cluster_sync
cluster_sync successfully
Check CubeOS version
sky1:boot> back
sky1> update
sky1:update> show
Current: 2.0.0
Rollback: 1.3.5
sky1:update> back
sky1> firmware
sky1:firmware> get_info
1: CUBE_1.3.5_20200426-0942_6c7ef1e
2: CUBE_2.0.0_20201125-2132_a4d7645
Enter index: 2
Firmware Version: Cube Appliance 2.0.0
Installation Date: Nov 27, 2020 08:35:34 PM
Installation Type: Upgrade
Last Boot: Nov 27, 2020 08:48:54 PM
sky1:firmware>
Run Service check & repair
sky1> cluster
sky1:cluster> check_repair
Service Status Report
ClusterLink ok [ link(v) ]
ClusterSettings ok [ etcd(v) ]
HaCluster FIXING [ hacluster(3) ]
ok [ hacluster(f) ]
warning: the VM is running with native name encoding of latin1 which may cause Elixir to malfunction as it expects utf8. Please ensure your locale is set to UTF-8 (which can be verified by running "locale" in your shell)
MsgQueue ok [ rabbitmq(v) ]
IaasDb ok [ mysql(v) ]
VirtualIp ok [ vip(v) haproxy_ha(v) ]
Storage ok [ ceph(v) ceph_mon(v) ceph_mgr(v) ceph_mds(v) ceph_osd(v) ceph_rgw(v) ]
ApiService ok [ haproxy(v) httpd(v) lmi(v) memcache(v) ]
Compute ok [ nova(v) ]
Network ok [ neutron(v) ]
Image ok [ glance(v) ]
BlockStor ok [ cinder(v) ]
FileStor ok [ manila(v) ]
ObjectStor ok [ swift(v) ]
Orchestration ok [ heat(v) ]
LBaaS ok [ octavia(v) ]
DNSaaS ok [ designate(v) ]
K8SaaS ok [ k3s(v) rancher(v) ]
InstanceHa ok [ sky1akari(v) ]
DisasterRecovery ok [ freezer(v) ]
DataPipe ok [ zookeeper(v) kafka(v) ]
Metrics ok [ monasca(v) telegraf(v) grafana(v) ]
LogAnalytics ok [ filebeat(v) auditbeat(v) logstash(v) es(v) kibana(v) ]
Notifications ok [ influxdb(v) kapacitor(v) ]
sky1:cluster>