bzr crashed with ShortReadvError in _seek_and_read()

Bug #697815 reported by Henry Gomersall
10
This bug affects 1 person
Affects Status Importance Assigned to Milestone
Bazaar
Confirmed
High
Vincent Ladeuil
bzr (Ubuntu)
Confirmed
Undecided
Unassigned

Bug Description

Binary package hint: bzr

My bzr metadata has somehow become corrupted making any bzr operations fail with a crash like that submitted (although bzr status returns fine).

The doing md5sum on the files in packs/ gives me one file with an incorrect name (that given by 4d46aaa30ce4a05d9a71342fb858bfe7.pack with the same name as the .rix file referenced in the report).

To fail like this is pretty serious. How much of the version data is lost? Is there any way to revert it? It seems the data itself is fine, so its not a catastrophe, but I'm currently unable to commit changes.

ProblemType: Crash
DistroRelease: Ubuntu 10.10
Package: bzr 2.2.1-0ubuntu1
Uname: Linux 2.6.36-020636-generic x86_64
Architecture: amd64
BzrDebugFlags: set()
BzrPlugins:
 bash_completion /usr/lib/python2.6/dist-packages/bzrlib/plugins/bash_completion [2.2.1]
   bzrtools /usr/lib/python2.6/dist-packages/bzrlib/plugins/bzrtools [2.2.0]
   launchpad /usr/lib/python2.6/dist-packages/bzrlib/plugins/launchpad [2.2.1]
   netrc_credential_store /usr/lib/python2.6/dist-packages/bzrlib/plugins/netrc_credential_store [2.2.1]
   news_merge /usr/lib/python2.6/dist-packages/bzrlib/plugins/news_merge [2.2.1]
BzrVersion: 2.2.1
CommandLine: ['/usr/bin/bzr', 'commit', '-m', 'not sure']
CrashDb: bzr
Date: Wed Jan 5 19:14:29 2011
ExecutablePath: /usr/bin/bzr
FileSystemEncoding: UTF-8
InstallationMedia: Ubuntu 10.04 "Lucid Lynx" - Beta amd64 (20100318)
InterpreterPath: /usr/bin/python2.6
Locale: en_GB.utf8
Platform: Linux-2.6.36-020636-generic-x86_64-with-Ubuntu-10.10-maverick
ProcCmdline: /usr/bin/python /usr/bin/bzr commit -m not\ sure
ProcEnviron:
 SHELL=/bin/bash
 LANG=en_GB.utf8
PythonVersion: 2.6.6
SourcePackage: bzr
Title: bzr crashed with ShortReadvError in _seek_and_read()
UserEncoding: UTF-8
UserGroups: adm admin cdrom daemon dialout lpadmin plugdev sambashare

Revision history for this message
Henry Gomersall (hgomersall) wrote :
tags: removed: need-duplicate-check
visibility: private → public
Vincent Ladeuil (vila)
Changed in bzr (Ubuntu):
status: New → Confirmed
Changed in bzr:
status: New → Confirmed
importance: Undecided → High
Revision history for this message
Vincent Ladeuil (vila) wrote :

As seen in the ML thread, this is caused by an empty pack file.

The attached script will help restore the *repository* in a working state (by getting rid of the reference to the empty pack file) but will not fix any branch that refer to it (nor restore its content which is lost for good :-/).

Fixing the branch itself can be achieved while keeping the changes in the working tree and committing again which will create a new pack file.

To get there:
- create a branch based on a valid revision,
- copy the 'last-revision' file in '.bzr/branch' from the good branch to the broken one
- 'bzr st' and 'bzr diff' should show you the uncommitted changes

Revision history for this message
Vincent Ladeuil (vila) wrote :

Rescue script that should be integrated into bzr with proper tests.

Revision history for this message
Henry Gomersall (hgomersall) wrote :

Thanks, the script seems to have done as advertised, but I can't seem to create a new branch. bzr branch foo bar -r N always returns the following error:
bzr: ERROR: Revision {<email address hidden>} not present in "CHKInventoryRepository('file:///path/to/repository/.bzr/repository/')".

The other problem is that I don't know which the last working revision is.

Revision history for this message
Vincent Ladeuil (vila) wrote :

Right, finding the last working revision is the tricky bit, try searching your '.bzr.log' file for messages like:

  201.324 Committed revid <email address hidden> as revno 60.

You should find one for '<email address hidden>' which has been lost (I dunno if you edited this one to inject '<email address hidden>' but the date '20101204' seem to match your empty pack file so it's probably the lost revision), and a previous one which should be working.

Once you have that one you can use it as -rrevid:<the_good_one> in your branch command.

Revision history for this message
Henry Gomersall (hgomersall) wrote :

Hmmm, I don't seem to have a .bzr.log file anywhere I can see (not ~/, not repository/, not /var/log/, not ~/.bazaar/). Should I have enabled a switch somewhere to turn on the logging?

Revision history for this message
Vincent Ladeuil (vila) wrote :

'bzr version' should tell you, on unices it's $HOME/.bzr.log

There is no switch to activate it, but you can redefine it by setting BZR_LOG... did you do that ?

Revision history for this message
Henry Gomersall (hgomersall) wrote :

Its set to ~/.bzr.log, but that file doesn't exist. Could this be another manifestation of the pack problem?

Ultimately, its not a catastrophe as the code is all there, but I would like to get to the bottom of it so it doesn't happen again.

Revision history for this message
Vincent Ladeuil (vila) wrote :

Nope, no relation at all with the pack problem (well, no obvious one that is).
Even issuing 'bzr version' should create it, so you have *another* problem here... pretty weird though, we've never had report of the .bzr.log file not being created, are you sure you're not tricked by using some utility that doesn't reveal the hidden files (.bzr.log starts with a dot to be considered an hidden file and not being shown by default).

Doing:

  cd
  ls -al

should reveal it.

Otherwise please post the output of 'bzr version'.

Revision history for this message
Henry Gomersall (hgomersall) wrote :

So it seems ~/.bzr.log was there. Not quite sure how I missed it (apologies for being a fool).

Anyway, I was having trouble with getting a sensible internal state for the code using the suggested method, so I branched the tree to an old commit and then replaced the .bzr directory in my working tree with the .bzr from the new branch. A few commits later and it all seems to be working fine.

Thanks for all the help. Is the recommendation then that ext4 should not be used?

Revision history for this message
Vincent Ladeuil (vila) wrote :

Ha good thing it was there and that you're now out of trouble.

No we don't recommend *against* ext4, this kind of issue have only been reported a few times and mostly on irc, so it's still unclear which kind of crash can lead to such situations. I use ext4 myself.

Revision history for this message
Vincent Ladeuil (vila) wrote :

I'll keep this bug open to track the integration of the script with proper tests into bzr-core.

Changed in bzr:
assignee: nobody → Vincent Ladeuil (vila)
Revision history for this message
Martin Pool (mbp) wrote :

There's a different case possible here, which is in bug 413430, where the pack file is fine, but the index file is truncated. (Probably ext4 again.)

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.