Temporary installer password and key fingerprints are not printed with latest impish daily ISOs

Status changed to 'Confirmed' because the bug affects multiple users.

Hi @Frank,

So I looked into this a bit, and with some feedback help from mwhudson, I see that we attempt to print the ssh info. Presumably this isn't going in the place you're looking, however.

Something that changed relatively recently was that we no longer have cloud-init handle setting the temp password of the installer user.

Is that how you get the ssh key / installer user password historically? From the cloud-init output? Some message like
'Above you will find SSH host keys and a random password set for the `installer` user'

Hmm looking into our livecd-rootfs stuff a bit, I'm a bit confused about tty names. We have special handling for sclp_line0 and (on s390x only) ttyS0 but from your logs it seems the tty in z/vm is called ttysclp0, is that correct? Which terminal are you looking at and which can run the subiquity tui?

Regarding the (main) s390x ttys:
SCLP line-mode terminal device driver /dev/sclp_line0 ('Operating Systems Messages')
SCLP full-screen mode VT220 terminal device driver /dev/ttysclp0 ('Integrated ASCII console', or z/VM)

Subiquity tui can only run in full-screen, not in line-mode.

There are two "consoles" in case of an LPAR - the "Operating Systems Messages" (line-mode) where only low-level console messages are displayed, including the installer boot messages that contain the temporary installer password,
and the "Integrated ASCII Console" (the emulated VT220 full-screen-mode) where the subiquity tui can run - and is in fact started there automatically (this still works and allows to complete an installation).

The z/VM console (by default) uses the 3215 line-mode, for the console messages that include temporary installer password, but the Subiquity tui can not run there (since it's line-mode), hence a remote ssh installation is needed here.

Just notice that all this worked fine in the past - and even with the early Impish images (about 06-14-2021 and before) and I do not think that things changed that much.

And since paride tried a current impish amd64 ISO image, where the temp installer password is also missing,things seems to be platform agnostic and should not be cause by any of the special s390x consoles - I think.

What's missing is an output like this (see attached example.txt taken from a 20.04 installation), that provides lines like:
"
"Set the following 'random' passwords"
installer:x3Z9hgeXCPNDQC8VFeqK
"
and
"
It is possible to connect to the installer over the network, which
might allow the use of a more capable terminal and can offer more languages
than can be rendered in the Linux console.

To connect, SSH to installer@10.245.236.14.

You should use the preconfigured password passed to cloud-init.

The host key fingerprints are:

RSA SHA256:YTmNejNSf1OlzQbOcNzkcYrDzqfy9aoT1laAs+oca5k
ECDSA SHA256:MKGkFuNzdQ3aNBL33bhqapn8clpDD9ZIL2SLr24IDAI
ED25519 SHA256:NitkBIaBVl97j1E3AXOrQRNrfR89UChEFxbJWnYhtVE
"
and yes, I believe the output came from cloud-init.

And even earlier the random / temp. password was not only given "somewhere" in the boot log, but also at the end (close to "To connect, SSH to installer@10.245.236.14."), so that one didn't needed to scroll through the messages to find it.

On Tue, 29 Jun 2021 at 19:50, Frank Heimes <email address hidden>
wrote:

> Regarding the (main) s390x ttys:
> SCLP line-mode terminal device driver /dev/sclp_line0
> ('Operating Systems Messages')
> SCLP full-screen mode VT220 terminal device driver /dev/ttysclp0
> ('Integrated ASCII console', or z/VM)
>
> Subiquity tui can only run in full-screen, not in line-mode.
>
> There are two "consoles" in case of an LPAR - the "Operating Systems
> Messages" (line-mode) where only low-level console messages are displayed,
> including the installer boot messages that contain the temporary installer
> password,
> and the "Integrated ASCII Console" (the emulated VT220 full-screen-mode)
> where the subiquity tui can run - and is in fact started there
> automatically (this still works and allows to complete an installation).
>
> The z/VM console (by default) uses the 3215 line-mode, for the console
> messages that include temporary installer password, but the Subiquity
> tui can not run there (since it's line-mode), hence a remote ssh
> installation is needed here.
>
> Just notice that all this worked fine in the past - and even with the
> early Impish images (about 06-14-2021 and before) and I do not think
> that things changed that much.
>
> And since paride tried a current impish amd64 ISO image, where the temp
> installer password is also missing,things seems to be platform agnostic
> and should not be cause by any of the special s390x consoles - I think.
>
>
> What's missing is an output like this (see attached example.txt taken from
> a 20.04 installation), that provides lines like:
> "
> "Set the following 'random' passwords"
> installer:x3Z9hgeXCPNDQC8VFeqK
> "
>

So I expect this to no longer be present; basically cloud-init has changed
to make it much harder / impossible to find this password later.

> and
> "
> It is possible to connect to the installer over the network, which
> might allow the use of a more capable terminal and can offer more languages
> than can be rendered in the Linux console.
>
>
> To connect, SSH to installer@10.245.236.14.
>
> You should use the preconfigured password passed to cloud-init.
>
> The host key fingerprints are:
>
> RSA SHA256:YTmNejNSf1OlzQbOcNzkcYrDzqfy9aoT1laAs+oca5k
> ECDSA SHA256:MKGkFuNzdQ3aNBL33bhqapn8clpDD9ZIL2SLr24IDAI
> ED25519 SHA256:NitkBIaBVl97j1E3AXOrQRNrfR89UChEFxbJWnYhtVE
> "
>

This output should still be there though, and that's what we should focus
on figuring out. The serial-getty service running on sclp_line0 is what is
_supposed_ to be producing this output.

I guess you should try this on an LPAR install where you can actually get
into the system: can you post the output of systemctl
status serial-getty@sclp_line0.service? And maybe systemctl
show serial-getty@sclp_line0.service? TIA

To be able to better compare between focal-daily and impish I now started LPAR installations:
1) on focal daily (from pending) - where things work, means where I get the temp/randon pw
2) and on impish (daily) (again from pending) - where I do not get a temp/randon pw

I used LPAR installs to be able to get into the system via the Integrated ASCII console.

The attached file "lpar_OS_messages.zip" contains the following corresponding two files:
lpar_OS_messages_console_focal_daily.txt
lpar_OS_messages_console_impish.txt

At the beginning of each file one can find the console installer boot-up messages
and after a separator "__________" as desired the output of:
"systemctl status serial-getty@sclp_line0.service"
and
"systemctl show serial-getty@sclp_line0.service"

@Michael: the installer:<password> output should still be present, but only as a printout in the console output (/dev/console), cloud-init doesn't log it anymore in cloud-init-output.log. This is the relevant change:

https://github.com/canonical/cloud-init/commit/b794d426b9ab43ea9d6371477466070d86e10668

just did the following quick test:
from the "Integrated ASCII console", where impish/subiquity ran I opened the live shell and did:
root@ubuntu-server:/# echo "x" > /dev/console
and
root@ubuntu-server:/# echo "x" > /dev/sclp_line0
and I can see two "x" coming through to the "Operating Systems messages" which is sclp_line0
that lets me think that /dev/console is properly 'mapped' to /dev/sclp_line0

and fwiw:
- Operating System Messages:
   $ tty
   /dev/sclp_line0
- Integrated ASCII Console:
   $ tty
   /dev/ttysclp0

Oh, I see the problem: the process that prints the network connection information is running, but does not have its stdout connected to the right place so the output is going nowhere. Easy fix fortunately (easier than figuring out what is going on!)

Ok, sounds promising.
Please give me a heads-up when it lands in Impish daily, so that I can update to it and retry.

Marking invalid on Subiquity as I believe that no change is required there. We can of course reverse that position if we find that Subiquity changes are indeed desired.

This bug was fixed in the package livecd-rootfs - 2.728

---------------
livecd-rootfs (2.728) impish; urgency=medium

  [ Dan Bungert ]
  * Remove the customized final message from the live server cloud-init config
    as it is no longer displayed.

  [ Michael Hudson-Doyle ]
  * Make sure the process that prints SSH connection info on s390x has
    its output connected to the console. (LP: #1933523)

  [ Brian Murray ]
  * Add a subarch of intel-iotg which is used for creating images with a
    kernel optimized for Intel IOT devices.

 -- Michael Hudson-Doyle <email address hidden> Thu, 01 Jul 2021 11:00:08 +1200

The fix should be in the next images that get built.

@mwhudson @dbungert
Looks like the line in subiquity-serial.conf did the trick!
With this in place and as part of the Impish daily ISO (I used the image from today, July 2nd) I can now see the temp./random password again - and even more nicely at the end of the installer boot-up process - great !

I've attached the last lines from an LPAR as well as from a z/VM installation console log.

I'm now happily closing this ticket as Fix Released - thx!

Yay thanks. Now to get back to backporting this whole mess to focal.

That sounds like 'fun'...
If there is once a candidate available that can be used by focal (or an appropriate focal daily) - I'm happy to do some more testing on focal, too.

Hi, now as [1] landed if I don't go wrong we'll have to:

- Wait for livecd-rootfs 2.664.25 to be accepted and installed
  in the Focal archive.
- Wait for new ISOs to be built and check from the cd-build-log [2]
  which livecd build was used.
- Check that livecd build log at [3] actually uses livecd-rootfs 2.664.25.
  (The livecd-rootfs version is not included in the cd-build-log [2]).
- If everything is as expected => Test the ISO.

[1] https://code.launchpad.net/~mwhudson/livecd-rootfs/+git/livecd-rootfs/+merge/406127
[2] https://people.canonical.com/~ubuntu-archive/cd-build-logs/ubuntu-server/focal/
[3] https://launchpad.net/~ubuntu-cdimage/+livefs/ubuntu/focal/ubuntu-server-live/

On Fri, 30 Jul 2021 at 21:11, Paride Legovini <email address hidden>
wrote:

> Hi, now as [1] landed if I don't go wrong we'll have to:
>
> - Wait for livecd-rootfs 2.664.25 to be accepted and installed
> in the Focal archive.
> - Wait for new ISOs to be built and check from the cd-build-log [2]
> which livecd build was used.
> - Check that livecd build log at [3] actually uses livecd-rootfs 2.664.25.
> (The livecd-rootfs version is not included in the cd-build-log [2]).
> - If everything is as expected => Test the ISO.
>

More or less yes. But no ISO will be built using the livecd-rootfs from
proposed automatically, we'll have to ask someone to do that.

Cheers,
mwh

Hello Frank, or anyone else affected,

Accepted livecd-rootfs into focal-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/livecd-rootfs/2.664.26 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-focal to verification-done-focal. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-focal. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

Hello Frank, or anyone else affected,

Accepted livecd-rootfs into focal-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/livecd-rootfs/2.664.27 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-focal to verification-done-focal. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-focal. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

As requested, I ran a server-live build with proposed enabled:

http://cdimage.ubuntu.com/ubuntu-server/focal/daily-live/20210805.1/

I tested the updated livecd-rootfs that is part of the new live-server image that Łukasz created:
http://cdimage.ubuntu.com/ubuntu-server/focal/daily-live/20210805.1/
and did a couple of installations (mainly LPAR, but DASD and FCP) and so far the boot-up of the installer always succeeded, and the temporary installer password and key fingerprints were always present and I was (so far) always able to login to the installer via ssh.
So I consider this now as successfully verified and think that we can close this ticket.

+1 on the verification done by Frank: I did a couple of non-s390x installs using the -proposed ISO and everything looks good.

Thanks! We will be probably re-spinning livecd-rootfs with some additional fixes on top, but this verification still stands.

This bug was fixed in the package livecd-rootfs - 2.664.27

---------------
livecd-rootfs (2.664.27) focal; urgency=medium

  * And whoops, we missed adding ARCH in the SUBARCH ubuntu-image handling for
    intel-iot.

livecd-rootfs (2.664.26) focal; urgency=medium

  * Revert previous change of fixing /dev sharing - this causes weird
    autopkgtest issues.

livecd-rootfs (2.664.25) focal; urgency=medium

  [ Brian Murray ]
  * Add support for creating images (ubuntu-core and classic) with a kernel
    optimized for Intel IoT devices. (LP: #1938338)

  [ Michael Hudson-Doyle ]
  * Simplify how the subiquity client is run on the serial console in the live
    server environment, breaking a unit cycle that sometimes prevents
    subiquity from starting up at all. (LP: #1888497)
  * Do not set the password for the installer user via cloud-init as subiquity
    can now do this itself. (LP: #1933523)

  [ Łukasz 'sil2100' Zemczak ]
  * Fix sharing of the /dev tree to make sure we can safely umount the chroot
    when needed. This fixes local non-livefs-builder image builds.
    (LP: #1938414)

 -- Łukasz 'sil2100' Zemczak <email address hidden> Wed, 04 Aug 2021 17:32:37 +0200

The verification of the Stable Release Update for livecd-rootfs has completed successfully and the package is now being released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.