[PowerVM] Ubuntu 16.04 does not install bootloader on multiple PReP partitions in Software RAID1 configuration

== Comment: #0 - Heitor Ricardo Alves de Siqueira - 2016-02-19 11:37:33 ==
---Problem Description---
When installing Ubuntu 16.04 to a RAID1 powerVM configuration, grub only installs to the first disk. The second disk will not have a valid PReP partition, rendering the system unbootable if the first disk fails in any way.

We are currently seeing this on a RAID1 install to two NVMe adapters.

---uname output---
Linux tul131p1 4.4.0-6-generic #21-Ubuntu SMP Tue Feb 16 20:31:37 UTC 2016 ppc64le GNU/Linux

Machine Type = 8286-42A lpar

---boot type---
kexec vmlinuz/initrd

---Kernel cmdline used to launch install---
default cmdline

---Install repository type---
Internet repository

---Install repository Location---
ports.ubuntu.com

---Point of failure---
Problem during post-install (stage 2) configuration or other problem seen after reboot

== Comment: #1 - Mauricio Faria De Oliveira <email address hidden> - 2016-02-26 05:31:43 ==
Currently:
1) grub-install lists all PReP partitions in the system
2) and prefers a PReP partition on the same disk as the root/boot filesystem.
3) wipe it and install the bootloader code in it.

Idea:
0) keep that as-is / backward compatible, as it works OK so far, but slightly change a particular case
1) grub-install lists all PReP partitions in the system
2) and prefers a PReP partition on the same disk as the root/boot filesystem.
3) but if the root/boot filesystem is a MD device,
4) find its component disks
5) check which ones have a PReP partition
6) wipe it and install bootloader code in it.

== Comment: #4 - Mauricio Faria De Oliveira <email address hidden> - 2016-02-26 08:54:04 ==
Hi Canonical,

This bug is a follow on to LP bug 1487365 (LTC BZ bug 129040) in the area of PReP partitions and Software RAID installations.

@taco-screen-team
I'd suggest for the bug assignee/owner to be @mathieu-tl on this bug too.

== Comment: #5 - Mauricio Faria De Oliveira <email address hidden> - 2016-02-26 09:00:23 ==
This patch for Xenial adds support in grub-installer for handling multiple PReP partitions on component devices of MD/dmadm devices.
It's backward compatible w/ the existing handling of other device types.

== Comment: #6 - Mauricio Faria De Oliveira <email address hidden> - 2016-02-26 09:01:01 ==
Debugging logs from the installation:

Syslog:
 ~ # grep grub-install /var/log/syslog
 Feb 26 11:09:15 anna[7389]: DEBUG: retrieving grub-installer 1.128ubuntu2
 Feb 26 11:57:48 main-menu[1776]: INFO: Menu item 'grub-installer' selected
 Feb 26 11:57:48 grub-installer: info: architecture: ppc64el/chrp_ibm
 Feb 26 11:57:52 grub-installer: info: Identified partition label for /dev/md0p2: gpt
 Feb 26 11:57:52 grub-installer: info: Wiping PReP partition /dev/sda1
 Feb 26 11:57:52 grub-installer: info: Wiping PReP partition /dev/sdb1
 Feb 26 11:57:52 grub-installer: info: Wiping PReP partition /dev/sdc1
 Feb 26 11:57:53 grub-installer: info: Installing grub on '/dev/sdc1'
 Feb 26 11:57:53 grub-installer: info: grub-install does not support --no-floppy
 Feb 26 11:57:53 grub-installer: info: Running chroot /target grub-install --force "/dev/sda1"
 Feb 26 11:57:54 grub-installer: Installing for powerpc-ieee1275 platform.
 Feb 26 11:58:02 grub-installer: Installation finished. No error reported.
 Feb 26 11:58:02 grub-installer: info: grub-install ran successfully
 Feb 26 11:58:02 grub-installer: info: grub-install does not support --no-floppy
 Feb 26 11:58:02 grub-installer: info: Running chroot /target grub-install --force "/dev/sdb1"
 Feb 26 11:58:02 grub-installer: Installing for powerpc-ieee1275 platform.
 Feb 26 11:58:11 grub-installer: Installation finished. No error reported.
 Feb 26 11:58:11 grub-installer: info: grub-install ran successfully
 Feb 26 11:58:11 grub-installer: info: grub-install does not support --no-floppy
 Feb 26 11:58:11 grub-installer: info: Running chroot /target grub-install --force "/dev/sdc1"
 Feb 26 11:58:11 grub-installer: Installing for powerpc-ieee1275 platform.
 Feb 26 11:58:20 grub-installer: Installation finished. No error reported.
 Feb 26 11:58:20 grub-installer: info: grub-install ran successfully

Shell script tracing:

For the shell tracing, add the following to /usr/bin/grub-install:
  exec 2>/grub.debug
  set -x

 ~ # cat /grub.debug
 <...>
 + wipe_bootdevs=
 + /usr/lib/grub-installer/prep-bootdev
 + wipe_bootdev=/dev/sda1
 + /usr/lib/grub-installer/prep-bootdev -l
 + [ /dev/sda = /dev/md ]
 + grep -q /dev/md[0-9]\+
 + echo /dev/md0
 + grep /dev/sda
 + grep -o /dev/[sv]d[a-z]\+
 + mdadm --detail --verbose /dev/md0
 + wipe_bootdevs= /dev/sda1
 + [ /dev/sdb = /dev/md ]
 + grep -q /dev/md[0-9]\+
 + echo /dev/md0
 + grep /dev/sdb
 + grep -o /dev/[sv]d[a-z]\+
 + mdadm --detail --verbose /dev/md0
 + wipe_bootdevs= /dev/sda1 /dev/sdb1
 + [ /dev/sdc = /dev/md ]
 + grep -q /dev/md[0-9]\+
 + echo /dev/md0
 + grep /dev/sdc
 + grep -o /dev/[sv]d[a-z]\+
 + mdadm --detail --verbose /dev/md0
 + wipe_bootdevs= /dev/sda1 /dev/sdb1 /dev/sdc1
 + unset prep_p
 + [ -z /dev/sda1 /dev/sdb1 /dev/sdc1 ]
 + [ -n /dev/sda1 ]
 + info Wiping PReP partition /dev/sda1
 + log info: Wiping PReP partition /dev/sda1
 + logger -t grub-installer info: Wiping PReP partition /dev/sda1
 + blockdev --getsz /dev/sda1
 + log-output -t grub-installer dd if=/dev/zero of=/dev/sda1 bs=512 count=14336
 + [ -n /dev/sdb1 ]
 + info Wiping PReP partition /dev/sdb1
 + log info: Wiping PReP partition /dev/sdb1
 + logger -t grub-installer info: Wiping PReP partition /dev/sdb1
 + blockdev --getsz /dev/sdb1
 + log-output -t grub-installer dd if=/dev/zero of=/dev/sdb1 bs=512 count=14336
 + [ -n /dev/sdc1 ]
 + info Wiping PReP partition /dev/sdc1
 + log info: Wiping PReP partition /dev/sdc1
 + logger -t grub-installer info: Wiping PReP partition /dev/sdc1
 + blockdev --getsz /dev/sdc1
 + log-output -t grub-installer dd if=/dev/zero of=/dev/sdc1 bs=512 count=14336
 <...>
 + bootdevs=/dev/sdc1
 + bootdevs= /dev/sda1 /dev/sdb1 /dev/sdc1
 <...>
 + info Running chroot /target grub-install --force "/dev/sda1"
 + log info: Running chroot /target grub-install --force "/dev/sda1"
 + logger -t grub-installer info: Running chroot /target grub-install --force "/dev/sda1"
 + log-output -t grub-installer chroot /target grub-install --force /dev/sda1
 + [ 0 = 0 ]
 + info grub-install ran successfully
 + log info: grub-install ran successfully
 + logger -t grub-installer info: grub-install ran successfully
 <...>
 + info Running chroot /target grub-install --force "/dev/sdb1"
 + log info: Running chroot /target grub-install --force "/dev/sdb1"
 + logger -t grub-installer info: Running chroot /target grub-install --force "/dev/sdb1"
 + log-output -t grub-installer chroot /target grub-install --force /dev/sdb1
 + [ 0 = 0 ]
 + info grub-install ran successfully
 + log info: grub-install ran successfully
 + logger -t grub-installer info: grub-install ran successfully
 <...>
 + info Running chroot /target grub-install --force "/dev/sdc1"
 + log info: Running chroot /target grub-install --force "/dev/sdc1"
 + logger -t grub-installer info: Running chroot /target grub-install --force "/dev/sdc1"
 + log-output -t grub-installer chroot /target grub-install --force /dev/sdc1
 + [ 0 = 0 ]
 + info grub-install ran successfully
 + log info: grub-install ran successfully
 + logger -t grub-installer info: grub-install ran successfully
 <...>

== Comment: #7 - Mauricio Faria De Oliveira <email address hidden> - 2016-02-26 09:01:29 ==
Test scenario:

3 individual disks (2 active, 1 spare)

$ for i in 1 2 3; do qemu-img create -f qcow2 disk$i.qcow2 8G; done
Formatting 'disk1.qcow2', fmt=qcow2 size=8589934592 encryption=off cluster_size=65536 lazy_refcounts=off refcount_bits=16
Formatting 'disk2.qcow2', fmt=qcow2 size=8589934592 encryption=off cluster_size=65536 lazy_refcounts=off refcount_bits=16
Formatting 'disk3.qcow2', fmt=qcow2 size=8589934592 encryption=off cluster_size=65536 lazy_refcounts=off refcount_bits=16

$ qemu-system-ppc64 -enable-kvm \
    -nographic -nodefaults \
    -monitor stdio -serial pty \
    -M pseries -smp 1,cores=1,threads=1 -m 8G \
    \
    -device spapr-vscsi \
    -drive file=disk1.qcow2 \
    -drive file=disk2.qcow2 \
    -drive file=disk3.qcow2 \
    -drive media=cdrom,file=xenial-server-ppc64el.iso \
    \
    -device spapr-vlan,netdev=net0,mac=4C:45:42:45:02:03 \
    -netdev bridge,id=net0,br=br0 \
    -S

1) On the 'Partition disks' dialog, select 'Go back', and 'Execute a shell'

2) Check the 'grub-installer' script is present

 ~ # ls -l /usr/bin/grub-installer
 -rwxr-xr-x 1 root root 44452 Jan 4 14:05 /usr/bin/grub-installer

3) Download and install the test package

 ~ # wget http://ausgsa.ibm.com/~mauricfo/public/bugs/bz137572/v1/grub-installer_1.128ubuntu2multiprep1_ppc64el.udeb
 ~ # udpkg -i *.udeb
 <... ignore the messages ...>

4) Verify the patch is present

 ~ # grep MD/mdadm /usr/bin/grub-installer
  # On MD/mdadm devices, each component device may have a PReP partition.
          # On MD/mdadm devices, each component device may have a PReP partition.

5) Return to the installation.

 ~ # exit

6) Proceed normally with a Software RAID based partitioning
   *with a PReP partition per component device.*

7) On the 'Installation finished' prompt (no errors expected), select 'Go Back', and 'Execute a shell'

8) Verify all the PReP partitions on the component devices
   were covered by grub-installer.
   (e.g., notice the 'Wiping PReP partition' and 'chroot /target grub-install' messages)

 ~ # grep -i grub-install /var/log/syslog
 Feb 26 13:19:45 anna[7383]: DEBUG: retrieving grub-installer 1.128ubuntu2
 Feb 26 13:22:22 grub-installer: info: architecture: ppc64el/chrp_ibm
 Feb 26 13:34:36 main-menu[1769]: INFO: Menu item 'grub-installer' selected
 Feb 26 13:34:36 grub-installer: info: architecture: ppc64el/chrp_ibm
 Feb 26 13:34:38 grub-installer: info: Identified partition label for /dev/md0p2: gpt
 Feb 26 13:34:39 grub-installer: info: Wiping PReP partition /dev/sda1
 Feb 26 13:34:39 grub-installer: info: Wiping PReP partition /dev/sdb1
 Feb 26 13:34:39 grub-installer: info: Wiping PReP partition /dev/sdc1
 Feb 26 13:34:55 grub-installer: info: Installing grub on '/dev/sdc1'
 Feb 26 13:34:55 grub-installer: info: grub-install does not support --no-floppy
 Feb 26 13:34:55 grub-installer: info: Running chroot /target grub-install --force "/dev/sda1"
 Feb 26 13:34:55 grub-installer: Installing for powerpc-ieee1275 platform.
 Feb 26 13:35:04 grub-installer: Installation finished. No error reported.
 Feb 26 13:35:04 grub-installer: info: grub-install ran successfully
 Feb 26 13:35:04 grub-installer: info: grub-install does not support --no-floppy
 Feb 26 13:35:04 grub-installer: info: Running chroot /target grub-install --force "/dev/sdb1"
 Feb 26 13:35:04 grub-installer: Installing for powerpc-ieee1275 platform.
 Feb 26 13:35:13 grub-installer: Installation finished. No error reported.
 Feb 26 13:35:13 grub-installer: info: grub-install ran successfully
 Feb 26 13:35:13 grub-installer: info: grub-install does not support --no-floppy
 Feb 26 13:35:13 grub-installer: info: Running chroot /target grub-install --force "/dev/sdc1"
 Feb 26 13:35:13 grub-installer: Installing for powerpc-ieee1275 platform.
 Feb 26 13:35:23 grub-installer: Installation finished. No error reported.
 Feb 26 13:35:23 grub-installer: info: grub-install ran successfully

9) Return to the installation, and select 'Finish the installation'.

 ~ # exit

Boot tests:

During boot, the SLOF firmware could successfully detect and load the GRUB2 code from
the PReP partition of *each individual disk* (test: run guest with either disk1, disk2, or disk3.qcow2).

 Trying to load: from: disk ...
 No DOS disk-label found.
   Successfully loaded
 error: no suitable video mode found.
 <...>
                  GNU GRUB version 2.02~beta2-36
 <...>

The only boot failure case is with only the spare disk (disk3.qcow2), since GRUB2 can't load
the remaining of its code from one of the active devices.
(notice it's a GRUB2 error message, so PReP was correctly detected and GRUB2 loaded from it.)

 Trying to load: from: disk ...
 No DOS disk-label found.
   Successfully loaded
 error: disk `mduuid/b3069040df508c48c7c8ddc6a19a6bc0,2' not found.
 Entering rescue mode...
 grub rescue>

== Comment: #8 - Mauricio Faria De Oliveira <email address hidden> - 2016-02-26 09:03:00 ==
Hi Heitor,

Can you please check it works correctly on your PowerVM environment?

I verified this works correctly on a qemu-kvm guest.

Thanks

== Comment: #9 - Mauricio Faria De Oliveira <email address hidden> - 2016-02-26 10:41:01 ==
Hi Canonical,

The patch is verified for 14.04 too. Please apply on 14.04 as well.

Thanks

== Comment: #10 - Mauricio Faria De Oliveira <email address hidden> - 2016-02-26 12:30:05 ==
(In reply to comment #9)
> The patch is verified for 14.04 too. Please apply on 14.04 as well.

Feb 26 14:17:48 anna[6955]: DEBUG: retrieving grub-installer 1.78ubuntu20.3
Feb 26 15:31:21 main-menu[1787]: INFO: Menu item 'grub-installer' selected
Feb 26 15:31:21 grub-installer: info: architecture: ppc64el/chrp_ibm
Feb 26 15:31:34 grub-installer: info: Identified partition label for /dev/md0p2: loop
Feb 26 15:31:34 grub-installer: info: Wiping PReP partition /dev/sda1
Feb 26 15:31:34 grub-installer: info: Wiping PReP partition /dev/sdb1
Feb 26 15:31:34 grub-installer: info: Wiping PReP partition /dev/sdc1
Feb 26 15:31:50 grub-installer: info: Installing grub on '/dev/sdc1'
Feb 26 15:31:50 grub-installer: info: grub-install does not support --no-floppy
Feb 26 15:31:50 grub-installer: info: Wiping PReP partition /dev/sda1
Feb 26 15:31:50 grub-installer: info: Running chroot /target grub-install --force "/dev/sda1"
Feb 26 15:31:50 grub-installer: Installing for powerpc-ieee1275 platform.
Feb 26 15:31:59 grub-installer: Installation finished. No error reported.
Feb 26 15:31:59 grub-installer: info: grub-install ran successfully
Feb 26 15:31:59 grub-installer: info: grub-install does not support --no-floppy
Feb 26 15:31:59 grub-installer: info: Wiping PReP partition /dev/sdb1
Feb 26 15:31:59 grub-installer: info: Running chroot /target grub-install --force "/dev/sdb1"
Feb 26 15:31:59 grub-installer: Installing for powerpc-ieee1275 platform.
Feb 26 15:32:08 grub-installer: Installation finished. No error reported.
Feb 26 15:32:08 grub-installer: info: grub-install ran successfully
Feb 26 15:32:08 grub-installer: info: grub-install does not support --no-floppy
Feb 26 15:32:08 grub-installer: info: Wiping PReP partition /dev/sdc1
Feb 26 15:32:08 grub-installer: info: Running chroot /target grub-install --force "/dev/sdc1"
Feb 26 15:32:08 grub-installer: Installing for powerpc-ieee1275 platform.
Feb 26 15:32:16 grub-installer: Installation finished. No error reported.
Feb 26 15:32:16 grub-installer: info: grub-install ran successfully