Severe regression with latest kernel update: 3.0.0-14.23 takes an unreasonable amount of time to boot due to udev

Trying to investigate a bit further

Issue is most likely with

# Stop udevd, we'll miss a few events while we run init, but we catch up
udevadm control --timeout=61 --exit

in /usr/share/initramfs-tools/scripts/init-bottom/udev

For some reason with the -14 kernel this command exits when udevadm times out, while previous kernels let udevadm exit much earlier.

As a proof, notice that trying to reduce the timeout makes -14 completely unbootable (machine comes up without /dev/sda, in spite of this device getting mentioned in dmesg), while previous kernels could still boot properly.

Cannot investigate any further as udev debugging is completely out of my reach. Cannot even find why the udevadm timeout does not get reported anywhere.

This is happening on a Dell E6500 with intel graphics which should be a rather widely spread machine, in case this information can help testing.

I have also tried the kernel on an eeepc 1000H, which does not seem to be affected by the bug.

Same problem here. Had to revert to old kernel for now.

Tried to catch some error info by booting in recovery mode

I think I managed watching for an instant something about

watershed -c sh /sbin/lvm vgscan; /sbin/lvm vgchange -a y

Can this issue be related to lvm?

To Jason: do you have any lvm partition?

Ok, I think I am probably running into bug Bug #802626 or Bug #818177

What happens is that if I add --noudevsync to the lvm2 udev rule in lib/udev and then I update the initramfs the delay disappears.

Thus adding lvm2 and udev to the bug.

However, I still wonder why I am running into this.

1) My other machines that are on LVM do not have this issue
2) It is completely unclear to me what changed in the -14 kernel so that this kernel shows the issue and -13 does not
3) My situation is not exactly the one in VG #802626 because my root filesystem _is_ on a logical volume

This issue appears to be an upstream bug, since you tested the latest upstream kernel. Would it be possible for you to open an upstream bug report at bugzilla.kernel.org [1]? That will allow the upstream Developers to examine the issue, and may provide a quicker resolution to the bug.

If you are comfortable with opening a bug upstream, It would be great if you can report back the upstream bug number in this bug report. That will allow us to link this bug to the upstream report.

The bugzilla.kernel.org site may still be unavailable due to the recent break-in. If that is the case, please add the tag: kernel-needs-upstream-bug-report

[1] https://wiki.ubuntu.com/Bugs/Upstream/kernel

Also, you mention this is a regression. Does the 3.0.0-13-generic kernel boot in a reasonable amount of time? If not, can you post the last kernel that was booting properly?

3.0.0-13 is the latest kernel that is booting fine.

3.0.0-14 does not, nor does 3.2RC4

However I am not extremely comfortable at opening an upstream kernel bug. The following points are unclear to me:

1) I have a strong suspicion that the issue I am running into is caused by a subtle timing issue. This suspicion is supported by the fact that other machines that I have, with a very similar configuration boot just fine. Maybe it is not that -14 is buggy, but only that it does something differently enough from -13 to alter timings, so that two events that are very closely spaced swap their order

2) It is unclear to me whether the bug is in the kernel, in udev or in lvm2. It looks like the timing relationships between the 3 is still object of investigation in Bug #802626. Specifically, it appears to me that debian derivatives run udev rules regarding lvm through watershed which is possibly a situation different from other distros.

3) It is unclear to me which upstream kernel version to report the bug against. Which upstream kernel does the ubuntu -14 correspond to?

Maybe some ubuntu kernel developer could have sounder grounds to opening an upstream bug than me.

In any case, if after thinking about the above points you still think that I should open an upstram bug, I'll be glad to to so.

Status changed to 'Confirmed' because the bug affects multiple users.

It looks like I hit this problem too, although perhaps I didn't leave it long enough to see if it would boot. Just assumed with no disk activity it wasn't going to boot.
In recovery mode it complained about a problem with root. Note root is on lvm2 and I'm running 64 Bit

3.0.0-13 works fine.

Can those who hit the bug please check if the patch in #802626 fixes the issue?

(basicaly it consists in adding a --noudevsync parameter to the vgchange command in
/lib/udev/rules.d/85-lvm2.rules. Then you need to regenerate the initramfs with
update-initramfs -u)

If it does we can probably mark this bug as a duplicate of that one.

@Sergio, does the patch in 802626 resolve your issue?

Yes, as mentioned in #6, adding the --noudevsync parameter fixes the delay for me.

Now, I wonder if it is just me, or also others have the same experience.

IMHO, this is important to know because the bug title in 802626 is about those who have the root filesystem outside LVM and all the rest in LVM, i.e. a rather restrictive case, but the proposed fix (at least for me) corrects the boot delay also in a more general cases (I have the root fs in LVM).

If it fixes the bug for everybody, we can mark this bug as a duplicate and widen the title of 802626.

In the meantime, note that while the --noudevsync fixes the delay, I do not know if it may have nasty side effects on some configurations. I do not have a good enough knowledge of udev, but for sure the man page of lvm says that --noudevsync should be used with care.

Another little note.

I made a lot of testing in the last days.

I have a feeling that in extremely rare cirumstances also 3.0.0-13 hangs at boot (happened once in about 100 test boots)
Only, on my machine 3.0.0-13 hangs 1% of the boots (which goes unnoticed), while -14 does systematically.

I have all my partitions on lvm except boot. I can confirm --noudevsync does work around the problem.

I've also run into this bug.

3.0.0-14 introduced a - precisely - 60 second delay into my boot-time. Tried 3.0.0-15 from proposed, with the same result. The only way to alleviate it was to roll-back to 3.0.0-13 (last kernel to not exhibit this behaviour). I've also tried a 3.2 kernel from Pangolin, and this too displays the delay.

I have my entire root file-system on LVM Including /boot), on GTP on an SSD.

I've now tried the "--noudevsync" fix and can confirm it removes the 60-second boot-delay when running 3.0.0-14 (and presumably all later kernels).

I'm interested in comment #2, which notes that this issue may be t related to:

# Stop udevd, we'll miss a few events while we run init, but we catch up
udevadm control --timeout=61 --exit

...as this duration exactly matches the delay I experience.

I had registered a bug about this (909805), but I've now marked it as a duplicate of this bug.

I have a testing USB flash drive with bootable Ubuntu Oneiric on it.
I boot it with debug=y in kernel command line which causes init scripts to output execution debug.info right to the console
I boot with either 3.0.0.-13 and 3.0.0-14 and find that noth of them hang exactly on run-init executing real init on the root file system.
This run-init thing is absolutely undebuggable from the command line. I could run strace or ltrace over it but they are dynamic executables which require ld-linux.so to be mounted and active before their run.
What else should I do in this situation, to go without excessive complications in startup troubleshooting ?

And yes, I tried to remove lvm2 & mdadm and regenerate initramfs,
this did not change anything.

Yes, I have exactly the same problem on my Thinkpad X61.

I have this bug too with 3.2 kernels.
Acer Aspire TimelineX 5820TG
Ubuntu 11.10
All my partitions on lvm .

I can confirm --noudevsync does work around the problem. Just tested it on latest 3.2.1 kernel

Given that the "--noudevsync" work-around from #802626 also appears to reliably fix this bug, I'm marking this bug as a duplicate of #802626.

Reopened the bug by removing the duplicate status, following the comment on bug 802626:

James,

I don't believe that's true. If the rootfs is in the same VG, there should be no possibility of udev being stopped in the initramfs prior to the dependent event making its way through the system, because without those events the rootfs can't be mounted at all. And certainly, root-on-LVM has been 100% reliable for me here. I think you should file a separate bug for the issue you're seeing.

Possible dups: bug 906358 and bug 631795

I caught the same bug, but I don't use any LVM. System hangs on boot when I use -15 kernel (generic and manually recompiled). -13 kernel works fine. I can't add --noudevsync to any line because there are no appropriate.

Laptop: Acer Aspire 5739G.

removing as a dup only temorarily.