apt segfaults on armel in oneiric

Excerpt from build log:

Processing triggers for libc-bin ...
ldconfig deferred processing now taking place
RUN: /usr/share/launchpad-buildd/slavebin/sbuild-package ['sbuild-package', '328ae026dfba17aa1f63d6a2ef5dd75df88ced94', 'armel', 'oneiric', '--nolog', '--batch', '--archive=ubuntu', '--dist=oneiric', '--purpose=PRIMARY', '--architecture=armel', '--comp=universe', 'boost-mpi-source1.46_1.46.1-4ubuntu3.dsc']
Initiating build 328ae026dfba17aa1f63d6a2ef5dd75df88ced94 with 0 processor cores.
Automatic build of boost-mpi-source1.46_1.46.1-4ubuntu3 on anacardiaceae by sbuild/armel 1.170.5
Build started at 20110430-0051
******************************************************************************
boost-mpi-source1.46_1.46.1-4ubuntu3.dsc exists in cwd
** Using build dependencies supplied by package:
Build-Depends: debhelper (>= 8), zlib1g-dev, libbz2-dev, libicu-dev, mpi-default-dev, bison, flex, docbook-to-man, help2man, xsltproc, doxygen, python, python-all-dev, python3, python3-all-dev (>= 3.1), libboost-serialization1.46-dev, libboost-python1.46-dev
Build-Conflicts: libopenmpi-dev (= 1.3.2-2)
/usr/bin/apt-cache exit status 11

ogra's backtrace:

root@isis:/root# strace apt-get update
execve("/usr/bin/apt-get", ["apt-get", "update"], [/* 24 vars */]) = 0
brk(0) = 0x5e9000
uname({sys="Linux", node="isis", ...}) = 0
access("/etc/ld.so.nohwcap", F_OK) = -1 ENOENT (No such file or directory)
mmap2(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x40044000
access("/etc/ld.so.preload", R_OK) = -1 ENOENT (No such file or directory)
open("/etc/ld.so.cache", O_RDONLY) = 3
fstat64(3, {st_mode=S_IFREG|0644, st_size=8252, ...}) = 0
mmap2(NULL, 8252, PROT_READ, MAP_PRIVATE, 3, 0) = 0x4007b000
close(3) = 0
access("/etc/ld.so.nohwcap", F_OK) = -1 ENOENT (No such file or directory)
open("/usr/lib/libapt-pkg.so.4.10", O_RDONLY) = 3
read(3, "\177ELF\1\1\1\0\0\0\0\0\0\0\0\0\3\0(\0\1\0\0\0\200\32\2\0004\0\0\0"..., 512) = 512
mmap2(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x40085000
fstat64(3, {st_mode=S_IFREG|0644, st_size=761456, ...}) = 0
mmap2(NULL, 793620, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_DENYWRITE, 3, 0) = 0x40098000
mprotect(0x40150000, 28672, PROT_NONE) = 0
mmap2(0x40157000, 12288, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0xb7) = 0x40157000
close(3) = 0
access("/etc/ld.so.nohwcap", F_OK) = -1 ENOENT (No such file or directory)
open("/usr/lib/arm-linux-gnueabi/libstdc++.so.6", O_RDONLY) = 3
read(3, "\177ELF\1\1\1\0\0\0\0\0\0\0\0\0\3\0(\0\1\0\0\0000_\4\0004\0\0\0"..., 512) = 512
fstat64(3, {st_mode=S_IFREG|0644, st_size=641012, ...}) = 0
mmap2(NULL, 699912, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_DENYWRITE, 3, 0) = 0x4015a000
mprotect(0x401f2000, 28672, PROT_NONE) = 0
mmap2(0x401f9000, 24576, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x97) = 0x401f9000
mmap2(0x401ff000, 24072, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_ANONYMOUS, -1, 0) = 0x401ff000
close(3) = 0
access("/etc/ld.so.nohwcap", F_OK) = -1 ENOENT (No such file or directory)
open("/lib/arm-linux-gnueabi/libgcc_s.so.1", O_RDONLY) = 3
read(3, "\177ELF\1\1\1\0\0\0\0\0\0\0\0\0\3\0(\0\1\0\0\0\340%\0\0004\0\0\0"..., 512) = 512
fstat64(3, {st_mode=S_IFREG|0644, st_size=34440, ...}) = 0
mmap2(NULL, 65812, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_DENYWRITE, 3, 0) = 0x40205000
mprotect(0x4020d000, 28672, PROT_NONE) = 0
mmap2(0x40214000, 8192, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x7) = 0x40214000
close(3) = 0
access("/etc/ld.so.nohwcap", F_OK) = -1 ENOENT (No such file or directory)
open("/lib/arm-linux-gnueabi/libc.so.6", O_RDONLY) = 3
read(3, "\177ELF\1\1\1\0\0\0\0\0\0\0\0\0\3\0(\0\1\0\0\0\375V\1\0004\0\0\0"..., 512) = 512
fstat64(3, {st_mode=S_IFREG|0755, st_size=885952, ...}) = 0
mmap2(NULL, 922920, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_DENYWRITE, 3, 0) = 0x40216000
mprotect(0x402eb000, 28672, PROT_NONE) = 0
mmap2(0x402f2000, 12288, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0xd4) = 0x402f2000
mmap2(0x402f5000, 9512, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_ANONYMOUS, -1, 0) = 0x402f5000
clo...

We may need a gdb backtrace with debugging symbols. All I can tell from the strace backend is that we try to allocate >1GB of memory via mmap2() and that fails. I don't know where this is happening though.

I was probably doing it wrong, but this all I got:

(gdb) run
Starting program: /usr/bin/apt-cache

Program received signal SIGSEGV, Segmentation fault.
0x40239c02 in __vsnprintf_chk () from /lib/arm-linux-gnueabi/libc.so.6

Scott, please run the 'bt' command after the segfault to get a backtrace.

0x40239c02 in __vsnprintf_chk () from /lib/arm-linux-gnueabi/libc.so.6
(gdb) (gdb) bt
#0 0x40239c02 in __vsnprintf_chk () from /lib/arm-linux-gnueabi/libc.so.6
#1 0x40042480 in vsnprintf (this=0x400e08f8, type=GlobalError::WARNING, Description=0x40314008 "(null) - (0: Success)", args=..., msgSize=@0
xbef78afc) at /usr/include/bits/stdio2.h:79
#2 GlobalError::Insert (this=0x400e08f8, type=GlobalError::WARNING, Description=0x40314008 "(null) - (0: Success)", args=..., msgSize=@0xbef
78afc) at contrib/error.cc:156
#3 0x40042746 in GlobalError::InsertErrno (this=0x400e08f8, type=GlobalError::WARNING, Function=0xd358 "", Description=0x0, args=..., errsv=0
, msgSize=@0xbef78afc) at contrib/error.cc:113
#4 0x40040ce0 in GlobalError::WarningE (this=0x400e08f8, Function=0xd358 "", Description=0x0) at contrib/error.cc:77
#5 0x00000000 in ?? ()
(gdb)

Could yoi maybe run bt full in gdb?

(gdb) (gdb) bt full
#0 0x40239c02 in __vsnprintf_chk () from /lib/arm-linux-gnueabi/libc.so.6
No symbol table info available.
#1 0x40042480 in vsnprintf (this=0x400e08f8, type=GlobalError::WARNING,
Description=0x40314008 "(null) - (0: Success)", args=...,
msgSize=@0xbef78afc) at /usr/include/bits/stdio2.h:79
No locals.
#2 GlobalError::Insert (this=0x400e08f8, type=GlobalError::WARNING,
Description=0x40314008 "(null) - (0: Success)", args=...,
msgSize=@0xbef78afc) at contrib/error.cc:156
        S = 0x0
        m = {Text = {static npos = 4294967295, _M_dataplus =
{<std::allocator<char>> = {<__gnu_cxx::new_allocator<char>> = {<No data
fields>}, <No data fields>},
              _M_p = 0x1 <Address 0x1 out of bounds>}}, Type = 1}
#3 0x40042746 in GlobalError::InsertErrno (this=0x400e08f8,
type=GlobalError::WARNING, Function=0xd358 "", Description=0x0, args=...,
errsv=0, msgSize=@0xbef78afc) at contrib/error.cc:113
        S = 0x40314008 "(null) - (0: Success)"
        geins = <value optimized out>
#4 0x40040ce0 in GlobalError::WarningE (this=0x400e08f8, Function=0xd358 "",
Description=0x0) at contrib/error.cc:77
        args = {__ap = 0xbef78b24}
        msgSize = 0
        errsv = 0
#5 0x00000000 in ?? ()
No symbol table info available.

does an apt built using GCC 4.5 show the same behaviour?

I built the oneiric apt on natty (with gcc4.5) and it built and worked fine on natty. Then I upgraded the chroot I used for this to oneiric and got the segmentation fault. I suspect that exonerates gcc4.6. I'm open for suggestions what to try next.

(Notes for other oneiric newbies)

To reproduce:
 * Start with a ARM natty system
 * % apt-get install ubuntu-dev-tools
 * Change to /usr/share/debootstrap/scripts
 * % ln -s gutsy oneiric to let debootstrap know how to make oneiric
 * % cd ~/linaro/roots
 * % export http_proxy=...
 * % debootstrap --include=gdb oneiric oneiric-1
 * % chroot oneiric-1
 * % apt-cache
 * See the segault

The natty debootstrap already knows about oneiric, so manually adding the symlink should not be needed.

I'm a bit confused. After installing libc6-dbg and also the ddebs, apt-get seems to segfault before hitting main:

root@ursa1:/# gdb /usr/bin/apt-get
GNU gdb (Ubuntu/Linaro 7.2-1ubuntu11) 7.2
Copyright (C) 2010 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law. Type "show copying"
and "show warranty" for details.
This GDB was configured as "arm-linux-gnueabi".
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>...
Reading symbols from /usr/bin/apt-get...Reading symbols from /usr/lib/debug/usr/bin/apt-get...done.
done.
(gdb) b main
Breakpoint 1 at 0xef0e: file apt-get.cc, line 3264.
(gdb) r
Starting program: /usr/bin/apt-get

Program received signal SIGSEGV, Segmentation fault.
0x2ae45c02 in ?? ()
(gdb) back
#0 0x2ae45c02 in ?? ()
#1 0x2adf6858 in ?? ()
#2 0x2adf6858 in ?? ()
Backtrace stopped: previous frame identical to this frame (corrupt stack?)

According to /proc/<pid>/maps, the 0x2adxxxxx address is in libc6.

Upgraded a panda to Oneiric with apt pinned to natty.
Rebuilt apt without optimization and the resulting deb works - libapt-pkg-4.10 does not crash anymore (it is this lib that the APT utils depend link against and the one with the origin of the backtrace)

Can be reproduced by getting the source and
DEB_BUILD_OPTIONS=noopt debuild -us -uc

the eglibc 2.13-0ubuntu15 build shows some regressions in the testsuite, compared to 2.13-0ubuntu13. Built using g++-4.5 (apt is built using the default g++). Before uploading a apt without optimization, I'd like to investigate these. Help is appreciated.

A rebuild with gcc 4.5.2 with optimization left at -O2 also results in correct binaries.

I put the two correctly working debs here, suffixed the file with the gcc version and opt level
http://people.ubuntu.com/~jani/

Building the oneiric apt source package in a debian unstable chroot using these gcc options suggested by doko

 -march=armv7-a -mfloat-abi=softfp -mfpu=vfpv3-fp16

also results in a functional package.

tracking the eglibc testsuite regressions in bug #775849

jani, could you combine the object of the failing and a successful (the debian?) build to find the problematice object files?

The segault doesn't show with a gcc-4.6 build with the Linaro changes disabled, the test build can be found at:

deb http://people.canonical.com/~doko/tmp/gcc-4.6-armel/ ./

The miscompiled file is apt-pkg/contrib/error.cc which , the one that contains the varargs using functions in the backtrace.
Compiling that object with gcc 4.5 -O2, gcc 4.6 at -O0 or -O1 results in good code generated.
Using gcc 4.6 at -O2 causes a segfault.
I'll provide more info shortly.

Observation (which may or may not be relevant):
It doesn't seem to get anywhere near main or libc startup (gdb break on main or __start or anything like that just doesn't stop).
Break'ing on _GLOBAL__sub_I_error.cc does stop and is an initialisation routine from the error.cc which is known to be broken.
objdump of the error.opic misdisasembles the first word of this function as:

00000000 <_GLOBAL__sub_I_error.cc>:
   0: 4d08b538 .word 0x4d08b538 <<<<<<<<<<<!!!
   4: 4c08 ldr r4, [pc, #32] ; (28 <_GLOBAL__sub_I_error.cc+0x28>)
   6: 447d add r5, pc
   8: 3508 adds r5, #8
   a: 447c add r4, pc
   c: 4628 mov r0, r5
   e: f7ff fffe bl 0 <_ZNSt8ios_base4InitC1Ev>
  12: 4b06 ldr r3, [pc, #24] ; (2c <_GLOBAL__sub_I_error.cc+0x2c>)
  14: 4628 mov r0, r5
  16: 58e1 ldr r1, [r4, r3]
  18: 4b05 ldr r3, [pc, #20] ; (30 <_GLOBAL__sub_I_error.cc+0x30>)
  1a: 58e2 ldr r2, [r4, r3]
  1c: e8bd 4038 ldmia.w sp!, {r3, r4, r5, lr}
  20: f7ff bffe b.w 0 <__aeabi_atexit>
  24: 0000001a .word 0x0000001a
  28: 0000001a .word 0x0000001a

That's actually a push and ldr; in gdb the instruction actually seems to be there, but when you break on _GLOBAL__sub_I_error you seem to be left at the 1st instruction after the push.

Dave

I put the error.o generated by Linaro Gcc 4.6 and Debian Unstable Gcc 4.6 here , along with a diff of the two objdumps.

http://people.ubuntu.com/~jani/

Putting the Debian generated object in the build tree results in a correctly functioning lib in Ubuntu.

Ah my bad, I forgot to add -mthumb to the Debian build. Added now and updated the files (.sid.thumb)
Now the diff is better.

Interesting. I just built apt-0.8.14.1ubuntu2 on Maverick using the plain gcc-linaro-4.6-2011.04 compiler and build/bin/apt-cache runs OK. The plain compiler doesn't have the stack protector or fortify turned on by default.

Changing the source slightly to use a constant that fits in 1 byte, leads to correct code. (In the function not in the macro, see below diff, so the code is only different in one place).

A movs instruction is generated for 200, whereas for 400 it is a mov.w and then a nop at the end of the function for padding.

This may affect other offset calculations but I could not determine why and which one contributes to the fault.

diff --git a/apt-pkg/contrib/error.cc b/apt-pkg/contrib/error.cc
index 6f6b7a4..17e6f13 100644
--- a/apt-pkg/contrib/error.cc
+++ b/apt-pkg/contrib/error.cc
@@ -84,7 +84,7 @@ GEMessage(DebugE, DEBUG)
 bool GlobalError::InsertErrno(MsgType const &type, const char *Function,
                                const char *Description,...) {
        va_list args;
- size_t msgSize = 400;
+ size_t msgSize = 200;
        int const errsv = errno;
        while (true) {
                va_start(args,Description);

Building with

 -O1 works
 -O1 -finline-small-functions -fpeephole2 -freorder-functions -fschedule-insns fails Removing any of these options and it works.
 -O2 fails

Compiling with -O1 -finline-small-functions -fpeephole2 -freorder-functions -fschedule-insns fails, but removing the -freorder-functions works.

The only difference in the assembler files produces is a set of section directives:

diff error.S.noreorder error-bad.S
503a504
> .section .text.unlikely,"ax",%progbits
732a734
> .text
806a809
> .section .text.unlikely
1065a1069
> .text
2476a2481
> .section .text.startup,"ax",%progbits

Removing that first .text.unlikely section mark makes it work.

Dave

OK, I think I've got it - it certainly smells like a linkerism but I didn't want to blame it until I actually found the bad code, but I believe it's another instance of bug 745843 - a problem with the fixup for the cortex-a8 branch erratum:

In the bad build we have:

00021fe0 <_GLOBAL__sub_I_netrc.cc>:
   21fe0: b538 push {r3, r4, r5, lr}
   21fe2: 4d08 ldr r5, [pc, #32] ; (22004 <_GLOBAL__sub_I_netrc.cc+0x24>)
   21fe4: 4c08 ldr r4, [pc, #32] ; (22008 <_GLOBAL__sub_I_netrc.cc+0x28>)
   21fe6: 447d add r5, pc
   21fe8: 447c add r4, pc
   21fea: 4628 mov r0, r5
   21fec: f7ff e86c blx 210c8 <_init+0x290>
   21ff0: 4b06 ldr r3, [pc, #24] ; (2200c <_GLOBAL__sub_I_netrc.cc+0x2c>)
   21ff2: 4628 mov r0, r5
   21ff4: 58e1 ldr r1, [r4, r3]
   21ff6: 4b06 ldr r3, [pc, #24] ; (22010 <_GLOBAL__sub_I_netrc.cc+0x30>)
   21ff8: 58e2 ldr r2, [r4, r3]
   21ffa: e8bd 4038 ldmia.w sp!, {r3, r4, r5, lr}
   21ffe: f081 bfa3 b.w a3f48

Now that branch is on a word boundary which kicks the a8 erratum case, so it has planted:

   a3f48: f77d bc20 b.w 2178c <_init+0x954>

Now note we're in Thumb at the moment.

which unfortunately branches to the following ARM code in the PLT - it should have branched a few bytes earlier, which is what Richard's fix in bug 745843 fixed.
   2178c: e28fc600 add ip, pc, #0
   21790: e28cca9f add ip, ip, #651264 ; 0x9f000
   21794: e5bcfb84 ldr pc, [ip, #2948]! ; 0xb84

We reckoned that was a really unlikely bug to hit - maybe one in a few thousand executables - trust it to land in apt!

Dave

This should be fixed by Richard's patch here:

http://sourceware.org/ml/binutils/2011-04/msg00177.html

It's probably worth pulling it in relatively quickly since this bug could trigger on anything else.

Dave

fixed in 2.21.51.20110421-0ubuntu5

apt rebuilt in oneiric

I am out of the office until 15/05/2011.

I will respond to your message when I return.

Note: This is an automated response to your message "[Bug 774175] Re: apt
segfaults on armel in oneiric" sent on 8/5/2011 3:56:48.

This is the only notification you will receive while this person is away.

Do we have evidence that this also breaks stuff in natty? Making a change to such a central piece of toolchain in an SRU bears a lot of regression potential, so it should only be considered if it fixes actually known bugs.

Well we originally found this in a Natty ftbfs in bug 745843 (vtk), so yes it breaks something - whether it breaks anything important however is a different matter; it's a bit of a timebomb - it's a case of a branch landing on a particular boundary under certain conditions and you never know what other change is going to trip that.

Dave

Accepted binutils into natty-proposed, the package will build now and be available in a few hours. Please test and give feedback here. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Thank you in advance!

Merged bug 745843 as a dupe of this one; tested that the natty proposed binutils package fixes vtk's FTBFS.

Dave

On Thu, May 19, 2011 at 08:39:13AM -0000, Dr. David Alan Gilbert wrote:
> Merged bug 745843 as a dupe of this one; tested that the natty proposed
> binutils package fixes vtk's FTBFS.

Based off your comment it sounds like the verification of the fix in the
proposed package has been completed. If that is the case please change
the bug tag from verification-needed to verification-done.

Thanks!

This bug was fixed in the package binutils - 2.21.0.20110327-2ubuntu3

---------------
binutils (2.21.0.20110327-2ubuntu3) natty-proposed; urgency=medium

  * Cortex A8 workarounds for PLT tail calls (Richard Sandiford). LP: #774175.
 -- Matthias Klose <email address hidden> Sun, 08 May 2011 03:00:48 +0200