1
=== modified file 'NEWS'
2
--- NEWS	2009-06-15 15:20:24 +0000
3
+++ NEWS	2009-06-16 05:29:42 +0000
4
@@ -17,6 +17,12 @@
5
17
  diverged-branches`` when a push fails because the branches have
17
  diverged-branches`` when a push fails because the branches have
6
18
  diverged.  (Neil Martinsen-Burrell, #269477)
18
  diverged.  (Neil Martinsen-Burrell, #269477)
7
19
19
8
20
API Changes
9
21
***********
10
22
11
23
* ``WorkingTree._check`` now requires a references dict with keys matching
12
24
  those returned by ``WorkingTree._get_check_refs``. (Robert Collins)
13
25
14
20
Internals
26
Internals
15
21
*********
27
*********
16
22
28
17
@@ -402,6 +408,9 @@
18
402
  cause mismatched physical locks to cause test errors rather than just
408
  cause mismatched physical locks to cause test errors rather than just
19
403
  reporting to the screen. (Robert Collins)
409
  reporting to the screen. (Robert Collins)
20
404
410
21
411
* -Dprogress will cause pdb to start up if a progress view jumps
22
412
  backwards. (Robert Collins)
23
413
24
405
* Fallback ``CredentialStore`` instances registered with ``fallback=True``
414
* Fallback ``CredentialStore`` instances registered with ``fallback=True``
25
406
  are now be able to provide credentials if obtaining credentials 
415
  are now be able to provide credentials if obtaining credentials 
26
407
  via ~/.bazaar/authentication.conf fails. (Jelmer Vernooij, 
416
  via ~/.bazaar/authentication.conf fails. (Jelmer Vernooij, 
27
408
417
28
=== modified file 'bzrlib/branch.py'
29
--- bzrlib/branch.py	2009-06-10 03:56:49 +0000
30
+++ bzrlib/branch.py	2009-06-16 05:29:42 +0000
31
@@ -125,6 +125,14 @@
32
125
            raise errors.UnstackableRepositoryFormat(self.repository._format,
125
            raise errors.UnstackableRepositoryFormat(self.repository._format,
33
126
                self.repository.base)
126
                self.repository.base)
34
127
127
35
128
    def _get_check_refs(self):
36
129
        """Get the references needed for check().
37
130
38
131
        See bzrlib.check.
39
132
        """
40
133
        revid = self.last_revision()
41
134
        return [('revision-existence', revid), ('lefthand-distance', revid)]
42
135
43
128
    @staticmethod
136
    @staticmethod
44
129
    def open(base, _unsupported=False, possible_transports=None):
137
    def open(base, _unsupported=False, possible_transports=None):
45
130
        """Open the branch rooted at base.
138
        """Open the branch rooted at base.
46
@@ -1132,7 +1140,7 @@
47
1132
        target._set_all_reference_info(target_reference_dict)
1140
        target._set_all_reference_info(target_reference_dict)
48
1133
1141
49
1134
    @needs_read_lock
1142
    @needs_read_lock
51
1135
    def check(self):
1143
    def check(self, refs):
52
1136
        """Check consistency of the branch.
1144
        """Check consistency of the branch.
53
1137
1145
54
1138
        In particular this checks that revisions given in the revision-history
1146
        In particular this checks that revisions given in the revision-history
55
@@ -1141,42 +1149,23 @@
56
1141
1149
57
1142
        Callers will typically also want to check the repository.
1150
        Callers will typically also want to check the repository.
58
1143
1151
59
1152
        :param refs: Calculated refs for this branch as specified by
60
1153
            branch._get_check_refs()
61
1144
        :return: A BranchCheckResult.
1154
        :return: A BranchCheckResult.
62
1145
        """
1155
        """
65
1146
        ret = BranchCheckResult(self)
1156
        result = BranchCheckResult(self)
64
1147
        mainline_parent_id = None
66
1148
        last_revno, last_revision_id = self.last_revision_info()
1157
        last_revno, last_revision_id = self.last_revision_info()
98
1149
        real_rev_history = []
1158
        actual_revno = refs[('lefthand-distance', last_revision_id)]
99
1150
        try:
1159
        if actual_revno != last_revno:
100
1151
            for revid in self.repository.iter_reverse_revision_history(
1160
            result.errors.append(errors.BzrCheckError(
101
1152
                last_revision_id):
1161
                'revno does not match len(mainline) %s != %s' % (
102
1153
                real_rev_history.append(revid)
1162
                last_revno, actual_revno)))
103
1154
        except errors.RevisionNotPresent:
1163
        # TODO: We should probably also check that self.revision_history
104
1155
            ret.ghosts_in_mainline = True
1164
        # matches the repository for older branch formats.
105
1156
        else:
1165
        # If looking for the code that cross-checks repository parents against
106
1157
            ret.ghosts_in_mainline = False
1166
        # the iter_reverse_revision_history output, that is now a repository
107
1158
        real_rev_history.reverse()
1167
        # specific check.
108
1159
        if len(real_rev_history) != last_revno:
1168
        return result
78
1160
            raise errors.BzrCheckError('revno does not match len(mainline)'
79
1161
                ' %s != %s' % (last_revno, len(real_rev_history)))
80
1162
        # TODO: We should probably also check that real_rev_history actually
81
1163
        #       matches self.revision_history()
82
1164
        for revision_id in real_rev_history:
83
1165
            try:
84
1166
                revision = self.repository.get_revision(revision_id)
85
1167
            except errors.NoSuchRevision, e:
86
1168
                raise errors.BzrCheckError("mainline revision {%s} not in repository"
87
1169
                            % revision_id)
88
1170
            # In general the first entry on the revision history has no parents.
89
1171
            # But it's not illegal for it to have parents listed; this can happen
90
1172
            # in imports from Arch when the parents weren't reachable.
91
1173
            if mainline_parent_id is not None:
92
1174
                if mainline_parent_id not in revision.parent_ids:
93
1175
                    raise errors.BzrCheckError("previous revision {%s} not listed among "
94
1176
                                        "parents of {%s}"
95
1177
                                        % (mainline_parent_id, revision_id))
96
1178
            mainline_parent_id = revision_id
97
1179
        return ret
109
1180
1169
110
1181
    def _get_checkout_format(self):
1170
    def _get_checkout_format(self):
111
1182
        """Return the most suitable metadir for a checkout of this branch.
1171
        """Return the most suitable metadir for a checkout of this branch.
112
@@ -2784,7 +2773,7 @@
113
2784
2773
114
2785
    def __init__(self, branch):
2774
    def __init__(self, branch):
115
2786
        self.branch = branch
2775
        self.branch = branch
117
2787
        self.ghosts_in_mainline = False
2776
        self.errors = []
118
2788
2777
119
2789
    def report_results(self, verbose):
2778
    def report_results(self, verbose):
120
2790
        """Report the check results via trace.note.
2779
        """Report the check results via trace.note.
121
@@ -2792,11 +2781,10 @@
122
2792
        :param verbose: Requests more detailed display of what was checked,
2781
        :param verbose: Requests more detailed display of what was checked,
123
2793
            if any.
2782
            if any.
124
2794
        """
2783
        """
130
2795
        note('checked branch %s format %s',
2784
        note('checked branch %s format %s', self.branch.base,
131
2796
             self.branch.base,
2785
            self.branch._format)
132
2797
             self.branch._format)
2786
        for error in self.errors:
133
2798
        if self.ghosts_in_mainline:
2787
            note('found error:%s', error)
129
2799
            note('branch contains ghosts in mainline')
134
2800
2788
135
2801
2789
136
2802
class Converter5to6(object):
2790
class Converter5to6(object):
137
2803
2791
138
=== modified file 'bzrlib/check.py'
139
--- bzrlib/check.py	2009-03-23 14:59:43 +0000
140
+++ bzrlib/check.py	2009-06-16 00:53:41 +0000
141
@@ -32,6 +32,20 @@
142
32
# raising them.  If there's more than one exception it'd be good to see them
32
# raising them.  If there's more than one exception it'd be good to see them
143
33
# all.
33
# all.
144
34
34
145
35
"""Checking of bzr objects.
146
36
147
37
check_refs is a concept used for optimising check. Objects that depend on other
148
38
objects (e.g. tree on repository) can list the objects they would be requesting
149
39
so that when the dependent object is checked, matches can be pulled out and
150
40
evaluated in-line rather than re-reading the same data many times.
151
41
check_refs are tuples (kind, value). Currently defined kinds are:
152
42
* 'trees', where value is a revid and the looked up objects are revision trees.
153
43
* 'lefthand-distance', where value is a revid and the looked up objects are the
154
44
  distance along the lefthand path to NULL for that revid.
155
45
* 'revision-existence', where value is a revid, and the result is True or False
156
46
  indicating that the revision was found/not found.
157
47
"""
158
48
159
35
from bzrlib import errors, osutils
49
from bzrlib import errors, osutils
160
36
from bzrlib import repository as _mod_repository
50
from bzrlib import repository as _mod_repository
161
37
from bzrlib import revision
51
from bzrlib import revision
162
@@ -39,6 +53,7 @@
163
39
from bzrlib.bzrdir import BzrDir
53
from bzrlib.bzrdir import BzrDir
164
40
from bzrlib.errors import BzrCheckError
54
from bzrlib.errors import BzrCheckError
165
41
from bzrlib.repository import Repository
55
from bzrlib.repository import Repository
166
56
from bzrlib.revision import NULL_REVISION
167
42
from bzrlib.symbol_versioning import deprecated_function, deprecated_in
57
from bzrlib.symbol_versioning import deprecated_function, deprecated_in
168
43
from bzrlib.trace import log_error, note
58
from bzrlib.trace import log_error, note
169
44
import bzrlib.ui
59
import bzrlib.ui
170
@@ -49,76 +64,145 @@
171
49
64
172
50
    # The Check object interacts with InventoryEntry.check, etc.
65
    # The Check object interacts with InventoryEntry.check, etc.
173
51
66
175
52
    def __init__(self, repository):
67
    def __init__(self, repository, check_repo=True):
176
53
        self.repository = repository
68
        self.repository = repository
177
54
        self.checked_text_cnt = 0
178
55
        self.checked_rev_cnt = 0
69
        self.checked_rev_cnt = 0
181
56
        self.ghosts = []
70
        self.ghosts = set()
180
57
        self.repeated_text_cnt = 0
182
58
        self.missing_parent_links = {}
71
        self.missing_parent_links = {}
183
59
        self.missing_inventory_sha_cnt = 0
72
        self.missing_inventory_sha_cnt = 0
184
60
        self.missing_revision_cnt = 0
73
        self.missing_revision_cnt = 0
185
61
        # maps (file-id, version) -> sha1; used by InventoryFile._check
186
62
        self.checked_texts = {}
187
63
        self.checked_weaves = set()
74
        self.checked_weaves = set()
188
64
        self.unreferenced_versions = set()
75
        self.unreferenced_versions = set()
189
65
        self.inconsistent_parents = []
76
        self.inconsistent_parents = []
190
66
        self.rich_roots = repository.supports_rich_root()
77
        self.rich_roots = repository.supports_rich_root()
191
67
        self.text_key_references = {}
78
        self.text_key_references = {}
192
79
        self.check_repo = check_repo
193
80
        self.other_results = []
194
81
        # Plain text lines to include in the report
195
82
        self._report_items = []
196
83
        # Keys we are looking for; may be large and need spilling to disk.
197
84
        # key->(type(revision/inventory/text/signature/map), sha1, first-referer)
198
85
        self.pending_keys = {}
199
86
        # Ancestors map for all of revisions being checked; while large helper
200
87
        # functions we call would create it anyway, so better to have once and
201
88
        # keep.
202
89
        self.ancestors = {}
203
68
90
205
69
    def check(self):
91
    def check(self, callback_refs=None, check_repo=True):
206
92
        if callback_refs is None:
207
93
            callback_refs = {}
208
70
        self.repository.lock_read()
94
        self.repository.lock_read()
209
71
        self.progress = bzrlib.ui.ui_factory.nested_progress_bar()
95
        self.progress = bzrlib.ui.ui_factory.nested_progress_bar()
210
72
        try:
96
        try:
228
73
            self.progress.update('retrieving inventory', 0, 2)
97
            self.progress.update('check', 0, 4)
229
74
            # do not put in init, as it should be done with progess,
98
            if self.check_repo:
230
75
            # and inside the lock.
99
                self.progress.update('checking revisions', 0)
231
76
            self.inventory_weave = self.repository.inventories
100
                self.check_revisions()
232
77
            self.progress.update('checking revision graph', 1)
101
                self.progress.update('checking commit contents', 1)
233
78
            self.check_revision_graph()
102
                self.repository._check_inventories(self)
234
79
            self.plan_revisions()
103
                self.progress.update('checking file graphs', 2)
235
80
            revno = 0
104
                # check_weaves is done after the revision scan so that
236
81
            while revno < len(self.planned_revisions):
105
                # revision index is known to be valid.
237
82
                rev_id = self.planned_revisions[revno]
106
                self.check_weaves()
238
83
                self.progress.update('checking revision', revno,
107
            self.progress.update('checking branches and trees', 3)
239
84
                                     len(self.planned_revisions))
108
            if callback_refs:
240
85
                revno += 1
109
                repo = self.repository
241
86
                self.check_one_rev(rev_id)
110
                # calculate all refs, and callback the objects requesting them.
242
87
            # check_weaves is done after the revision scan so that
111
                refs = {}
243
88
            # revision index is known to be valid.
112
                wanting_items = set()
244
89
            self.check_weaves()
113
                # Current crude version calculates everything and calls
245
114
                # everything at once. Doing a queue and popping as things are
246
115
                # satisfied would be cheaper on memory [but few people have
247
116
                # huge numbers of working trees today. TODO: fix before
248
117
                # landing].
249
118
                distances = set()
250
119
                existences = set()
251
120
                for ref, wantlist in callback_refs.iteritems():
252
121
                    wanting_items.update(wantlist)
253
122
                    kind, value = ref
254
123
                    if kind == 'trees':
255
124
                        refs[ref] = repo.revision_tree(value)
256
125
                    elif kind == 'lefthand-distance':
257
126
                        distances.add(value)
258
127
                    elif kind == 'revision-existence':
259
128
                        existences.add(value)
260
129
                    else:
261
130
                        raise AssertionError(
262
131
                            'unknown ref kind for ref %s' % ref)
263
132
                node_distances = repo.get_graph().find_lefthand_distances(distances)
264
133
                for key, distance in node_distances.iteritems():
265
134
                    refs[('lefthand-distance', key)] = distance
266
135
                    if key in existences and distance > 0:
267
136
                        refs[('revision-existence', key)] = True
268
137
                        existences.remove(key)
269
138
                parent_map = repo.get_graph().get_parent_map(existences)
270
139
                for key in parent_map:
271
140
                    refs[('revision-existence', key)] = True
272
141
                    existences.remove(key)
273
142
                for key in existences:
274
143
                    refs[('revision-existence', key)] = False
275
144
                for item in wanting_items:
276
145
                    if isinstance(item, WorkingTree):
277
146
                        item._check(refs)
278
147
                    if isinstance(item, Branch):
279
148
                        self.other_results.append(item.check(refs))
280
90
        finally:
149
        finally:
281
91
            self.progress.finished()
150
            self.progress.finished()
282
92
            self.repository.unlock()
151
            self.repository.unlock()
283
93
152
285
94
    def check_revision_graph(self):
153
    def _check_revisions(self, revisions_iterator):
286
154
        """Check revision objects by decorating a generator.
287
155
288
156
        :param revisions_iterator: An iterator of(revid, Revision-or-None).
289
157
        :return: A generator of the contents of revisions_iterator.
290
158
        """
291
159
        self.planned_revisions = set()
292
160
        for revid, revision in revisions_iterator:
293
161
            yield revid, revision
294
162
            self._check_one_rev(revid, revision)
295
163
        # Flatten the revisions we found to guarantee consistent later
296
164
        # iteration.
297
165
        self.planned_revisions = list(self.planned_revisions)
298
166
        # TODO: extract digital signatures as items to callback on too.
299
167
300
168
    def check_revisions(self):
301
169
        """Scan revisions, checking data directly available as we go."""
302
170
        revision_iterator = self.repository._iter_revisions(None)
303
171
        revision_iterator = self._check_revisions(revision_iterator)
304
172
        # We read the all revisions here:
305
173
        # - doing this allows later code to depend on the revision index.
306
174
        # - we can fill out existence flags at this point
307
175
        # - we can read the revision inventory sha at this point
308
176
        # - we can check properties and serialisers etc.
309
95
        if not self.repository.revision_graph_can_have_wrong_parents():
177
        if not self.repository.revision_graph_can_have_wrong_parents():
311
96
            # This check is not necessary.
178
            # The check against the index isn't needed.
312
97
            self.revs_with_bad_parents_in_index = None
179
            self.revs_with_bad_parents_in_index = None
326
98
            return
180
            for thing in revision_iterator:
327
99
        bad_revisions = self.repository._find_inconsistent_revision_parents()
181
                pass
328
100
        self.revs_with_bad_parents_in_index = list(bad_revisions)
182
        else:
329
101
183
            bad_revisions = self.repository._find_inconsistent_revision_parents(
330
102
    def plan_revisions(self):
184
                revision_iterator)
331
103
        repository = self.repository
185
            self.revs_with_bad_parents_in_index = list(bad_revisions)
319
104
        self.planned_revisions = repository.all_revision_ids()
320
105
        self.progress.clear()
321
106
        inventoried = set(key[-1] for key in self.inventory_weave.keys())
322
107
        awol = set(self.planned_revisions) - inventoried
323
108
        if len(awol) > 0:
324
109
            raise BzrCheckError('Stored revisions missing from inventory'
325
110
                '{%s}' % ','.join([f for f in awol]))
332
111
186
333
112
    def report_results(self, verbose):
187
    def report_results(self, verbose):
334
188
        if self.check_repo:
335
189
            self._report_repo_results(verbose)
336
190
        for result in self.other_results:
337
191
            result.report_results(verbose)
338
192
339
193
    def _report_repo_results(self, verbose):
340
113
        note('checked repository %s format %s',
194
        note('checked repository %s format %s',
341
114
             self.repository.bzrdir.root_transport,
195
             self.repository.bzrdir.root_transport,
342
115
             self.repository._format)
196
             self.repository._format)
343
116
        note('%6d revisions', self.checked_rev_cnt)
197
        note('%6d revisions', self.checked_rev_cnt)
344
117
        note('%6d file-ids', len(self.checked_weaves))
198
        note('%6d file-ids', len(self.checked_weaves))
349
118
        note('%6d unique file texts', self.checked_text_cnt)
199
        if verbose:
350
119
        note('%6d repeated file texts', self.repeated_text_cnt)
200
            note('%6d unreferenced text versions',
351
120
        note('%6d unreferenced text versions',
201
                len(self.unreferenced_versions))
352
121
             len(self.unreferenced_versions))
202
        if verbose and len(self.unreferenced_versions):
353
203
                for file_id, revision_id in self.unreferenced_versions:
354
204
                    log_error('unreferenced version: {%s} in %s', revision_id,
355
205
                        file_id)
356
122
        if self.missing_inventory_sha_cnt:
206
        if self.missing_inventory_sha_cnt:
357
123
            note('%6d revisions are missing inventory_sha1',
207
            note('%6d revisions are missing inventory_sha1',
358
124
                 self.missing_inventory_sha_cnt)
208
                 self.missing_inventory_sha_cnt)
359
@@ -138,10 +222,6 @@
360
138
                    note('      %s should be in the ancestry for:', link)
222
                    note('      %s should be in the ancestry for:', link)
361
139
                    for linker in linkers:
223
                    for linker in linkers:
362
140
                        note('       * %s', linker)
224
                        note('       * %s', linker)
363
141
            if verbose:
364
142
                for file_id, revision_id in self.unreferenced_versions:
365
143
                    log_error('unreferenced version: {%s} in %s', revision_id,
366
144
                        file_id)
367
145
        if len(self.inconsistent_parents):
225
        if len(self.inconsistent_parents):
368
146
            note('%6d inconsistent parents', len(self.inconsistent_parents))
226
            note('%6d inconsistent parents', len(self.inconsistent_parents))
369
147
            if verbose:
227
            if verbose:
370
@@ -161,60 +241,75 @@
371
161
                        '       %s has wrong parents in index: '
241
                        '       %s has wrong parents in index: '
372
162
                        '%r should be %r',
242
                        '%r should be %r',
373
163
                        revision_id, index_parents, actual_parents)
243
                        revision_id, index_parents, actual_parents)
379
164
244
        for item in self._report_items:
380
165
    def check_one_rev(self, rev_id):
245
            note(item)
381
166
        """Check one revision.
246
382
167
247
    def _check_one_rev(self, rev_id, rev):
383
168
        rev_id - the one to check
248
        """Cross-check one revision.
384
249
385
250
        :param rev_id: A revision id to check.
386
251
        :param rev: A revision or None to indicate a missing revision.
387
169
        """
252
        """
388
170
        rev = self.repository.get_revision(rev_id)
389
171
390
172
        if rev.revision_id != rev_id:
253
        if rev.revision_id != rev_id:
394
173
            raise BzrCheckError('wrong internal revision id in revision {%s}'
254
            self._report_items.append(
395
174
                                % rev_id)
255
                'Mismatched internal revid {%s} and index revid {%s}' % (
396
175
256
                rev.revision_id, rev_id))
397
257
            rev_id = rev.revision_id
398
258
        # Check this revision tree etc, and count as seen when we encounter a
399
259
        # reference to it.
400
260
        self.planned_revisions.add(rev_id)
401
261
        # It is not a ghost
402
262
        self.ghosts.discard(rev_id)
403
263
        # Count all parents as ghosts if we haven't seen them yet.
404
176
        for parent in rev.parent_ids:
264
        for parent in rev.parent_ids:
405
177
            if not parent in self.planned_revisions:
265
            if not parent in self.planned_revisions:
432
178
                # rev has a parent we didn't know about.
266
                self.ghosts.add(parent)
433
179
                missing_links = self.missing_parent_links.get(parent, [])
267
        
434
180
                missing_links.append(rev_id)
268
        self.ancestors[rev_id] = tuple(rev.parent_ids) or (NULL_REVISION,)
435
181
                self.missing_parent_links[parent] = missing_links
269
        self.add_pending_item(rev_id, ('inventories', rev_id), 'inventory',
436
182
                # list based so somewhat slow,
270
            rev.inventory_sha1)
411
183
                # TODO have a planned_revisions list and set.
412
184
                if self.repository.has_revision(parent):
413
185
                    missing_ancestry = self.repository.get_ancestry(parent)
414
186
                    for missing in missing_ancestry:
415
187
                        if (missing is not None
416
188
                            and missing not in self.planned_revisions):
417
189
                            self.planned_revisions.append(missing)
418
190
                else:
419
191
                    self.ghosts.append(rev_id)
420
192
421
193
        if rev.inventory_sha1:
422
194
            # Loopback - this is currently circular logic as the
423
195
            # knit get_inventory_sha1 call returns rev.inventory_sha1.
424
196
            # Repository.py's get_inventory_sha1 should instead return
425
197
            # inventories.get_record_stream([(revid,)]).next().sha1 or
426
198
            # similar.
427
199
            inv_sha1 = self.repository.get_inventory_sha1(rev_id)
428
200
            if inv_sha1 != rev.inventory_sha1:
429
201
                raise BzrCheckError('Inventory sha1 hash doesn\'t match'
430
202
                    ' value in revision {%s}' % rev_id)
431
203
        self._check_revision_tree(rev_id)
437
204
        self.checked_rev_cnt += 1
271
        self.checked_rev_cnt += 1
438
205
272
439
273
    def add_pending_item(self, referer, key, kind, sha1):
440
274
        """Add a reference to a sha1 to be cross checked against a key.
441
275
442
276
        :param referer: The referer that expects key to have sha1.
443
277
        :param key: A storage key e.g. ('texts', 'foo@bar-20040504-1234')
444
278
        :param kind: revision/inventory/text/map/signature
445
279
        :param sha1: A hex sha1 or None if no sha1 is known.
446
280
        """
447
281
        existing = self.pending_keys.get(key)
448
282
        if existing:
449
283
            if sha1 != existing[1]:
450
284
                self._report_items.append('Multiple expected sha1s for %s. {%s}'
451
285
                    ' expects {%s}, {%s} expects {%s}', (
452
286
                    key, referer, sha1, existing[1], existing[0]))
453
287
        else:
454
288
            self.pending_keys[key] = (kind, sha1, referer)
455
289
456
206
    def check_weaves(self):
290
    def check_weaves(self):
457
207
        """Check all the weaves we can get our hands on.
291
        """Check all the weaves we can get our hands on.
458
208
        """
292
        """
459
209
        weave_ids = []
293
        weave_ids = []
466
210
        self.progress.update('checking inventory', 0, 2)
294
        storebar = bzrlib.ui.ui_factory.nested_progress_bar()
467
211
        self.inventory_weave.check(progress_bar=self.progress)
295
        try:
468
212
        self.progress.update('checking text storage', 1, 2)
296
            self._check_weaves(storebar)
469
213
        self.repository.texts.check(progress_bar=self.progress)
297
        finally:
470
214
        weave_checker = self.repository._get_versioned_file_checker(
298
            storebar.finished()
471
215
            text_key_references=self.text_key_references)
299
472
300
    def _check_weaves(self, storebar):
473
301
        storebar.update('text-index', 0, 2)
474
302
        if self.repository._format.fast_deltas:
475
303
            # We haven't considered every fileid instance so far.
476
304
            weave_checker = self.repository._get_versioned_file_checker(
477
305
                ancestors=self.ancestors)
478
306
        else:
479
307
            weave_checker = self.repository._get_versioned_file_checker(
480
308
                text_key_references=self.text_key_references,
481
309
                ancestors=self.ancestors)
482
310
        storebar.update('file-graph', 1)
483
216
        result = weave_checker.check_file_version_parents(
311
        result = weave_checker.check_file_version_parents(
485
217
            self.repository.texts, progress_bar=self.progress)
312
            self.repository.texts)
486
218
        self.checked_weaves = weave_checker.file_ids
313
        self.checked_weaves = weave_checker.file_ids
487
219
        bad_parents, unused_versions = result
314
        bad_parents, unused_versions = result
488
220
        bad_parents = bad_parents.items()
315
        bad_parents = bad_parents.items()
489
@@ -228,28 +323,8 @@
490
228
                (revision_id, weave_id, weave_parents, correct_parents))
323
                (revision_id, weave_id, weave_parents, correct_parents))
491
229
        self.unreferenced_versions.update(unused_versions)
324
        self.unreferenced_versions.update(unused_versions)
492
230
325
493
231
    def _check_revision_tree(self, rev_id):
494
232
        tree = self.repository.revision_tree(rev_id)
495
233
        inv = tree.inventory
496
234
        seen_ids = set()
497
235
        seen_names = set()
498
236
        for path, ie in inv.iter_entries():
499
237
            self._add_entry_to_text_key_references(inv, ie)
500
238
            file_id = ie.file_id
501
239
            if file_id in seen_ids:
502
240
                raise BzrCheckError('duplicated file_id {%s} '
503
241
                                    'in inventory for revision {%s}'
504
242
                                    % (file_id, rev_id))
505
243
            seen_ids.add(file_id)
506
244
            ie.check(self, rev_id, inv, tree)
507
245
            if path in seen_names:
508
246
                raise BzrCheckError('duplicated path %s '
509
247
                                    'in inventory for revision {%s}'
510
248
                                    % (path, rev_id))
511
249
            seen_names.add(path)
512
250
513
251
    def _add_entry_to_text_key_references(self, inv, entry):
326
    def _add_entry_to_text_key_references(self, inv, entry):
515
252
        if not self.rich_roots and entry == inv.root:
327
        if not self.rich_roots and entry.name == '':
516
253
            return
328
            return
517
254
        key = (entry.file_id, entry.revision)
329
        key = (entry.file_id, entry.revision)
518
255
        self.text_key_references.setdefault(key, False)
330
        self.text_key_references.setdefault(key, False)
519
@@ -263,13 +338,14 @@
520
263
338
521
264
    Results are reported through logging.
339
    Results are reported through logging.
522
265
340
524
266
    Deprecated in 1.6.  Please use check_branch instead.
341
    Deprecated in 1.6.  Please use check_dwim instead.
525
267
342
526
268
    :raise BzrCheckError: if there's a consistency error.
343
    :raise BzrCheckError: if there's a consistency error.
527
269
    """
344
    """
528
270
    check_branch(branch, verbose)
345
    check_branch(branch, verbose)
529
271
346
530
272
347
531
348
@deprecated_function(deprecated_in((1,16,0)))
532
273
def check_branch(branch, verbose):
349
def check_branch(branch, verbose):
533
274
    """Run consistency checks on a branch.
350
    """Run consistency checks on a branch.
534
275
351
535
@@ -279,56 +355,108 @@
536
279
    """
355
    """
537
280
    branch.lock_read()
356
    branch.lock_read()
538
281
    try:
357
    try:
540
282
        branch_result = branch.check()
358
        needed_refs = {}
541
359
        for ref in branch._get_check_refs():
542
360
            needed_refs.setdefault(ref, []).append(branch)
543
361
        result = branch.repository.check([branch.last_revision()], needed_refs)
544
362
        branch_result = result.other_results[0]
545
283
    finally:
363
    finally:
546
284
        branch.unlock()
364
        branch.unlock()
547
285
    branch_result.report_results(verbose)
365
    branch_result.report_results(verbose)
548
286
366
549
287
367
550
368
def scan_branch(branch, needed_refs, to_unlock):
551
369
    """Scan a branch for refs.
552
370
553
371
    :param branch:  The branch to schedule for checking.
554
372
    :param needed_refs: Refs we are accumulating.
555
373
    :param to_unlock: The unlock list accumulating.
556
374
    """
557
375
    note("Checking branch at '%s'." % (branch.base,))
558
376
    branch.lock_read()
559
377
    to_unlock.append(branch)
560
378
    branch_refs = branch._get_check_refs()
561
379
    for ref in branch_refs:
562
380
        reflist = needed_refs.setdefault(ref, [])
563
381
        reflist.append(branch)
564
382
565
383
566
384
def scan_tree(base_tree, tree, needed_refs, to_unlock):
567
385
    """Scan a tree for refs.
568
386
569
387
    :param base_tree: The original tree check opened, used to detect duplicate
570
388
        tree checks.
571
389
    :param tree:  The tree to schedule for checking.
572
390
    :param needed_refs: Refs we are accumulating.
573
391
    :param to_unlock: The unlock list accumulating.
574
392
    """
575
393
    if base_tree is not None and tree.basedir == base_tree.basedir:
576
394
        return
577
395
    note("Checking working tree at '%s'." % (tree.basedir,))
578
396
    tree.lock_read()
579
397
    to_unlock.append(tree)
580
398
    tree_refs = tree._get_check_refs()
581
399
    for ref in tree_refs:
582
400
        reflist = needed_refs.setdefault(ref, [])
583
401
        reflist.append(tree)
584
402
585
403
586
288
def check_dwim(path, verbose, do_branch=False, do_repo=False, do_tree=False):
404
def check_dwim(path, verbose, do_branch=False, do_repo=False, do_tree=False):
587
289
    try:
405
    try:
589
290
        tree, branch, repo, relpath = \
406
        base_tree, branch, repo, relpath = \
590
291
                        BzrDir.open_containing_tree_branch_or_repository(path)
407
                        BzrDir.open_containing_tree_branch_or_repository(path)
591
292
    except errors.NotBranchError:
408
    except errors.NotBranchError:
618
293
        tree = branch = repo = None
409
        base_tree = branch = repo = None
619
294
410
620
295
    if do_tree:
411
    to_unlock = []
621
296
        if tree is not None:
412
    needed_refs= {}
622
297
            note("Checking working tree at '%s'."
413
    try:
623
298
                 % (tree.bzrdir.root_transport.base,))
414
        if base_tree is not None:
624
299
            tree._check()
415
            # If the tree is a lightweight checkout we won't see it in
625
300
        else:
416
            # repo.find_branches - add now.
626
301
            log_error("No working tree found at specified location.")
417
            if do_tree:
627
302
418
                scan_tree(None, base_tree, needed_refs, to_unlock)
628
303
    if branch is not None:
419
            branch = base_tree.branch
629
304
        # We have a branch
420
        if branch is not None:
630
305
        if repo is None:
421
            # We have a branch
631
306
            # The branch is in a shared repository
422
            if repo is None:
632
307
            repo = branch.repository
423
                # The branch is in a shared repository
633
308
        branches = [branch]
424
                repo = branch.repository
634
309
    elif repo is not None:
425
        if repo is not None:
635
310
        branches = repo.find_branches(using=True)
426
            repo.lock_read()
636
311
427
            to_unlock.append(repo)
637
312
    if repo is not None:
428
            branches = repo.find_branches(using=True)
638
313
        repo.lock_read()
429
            saw_tree = False
639
314
        try:
430
            if do_branch or do_tree:
640
315
            if do_repo:
431
                for branch in branches:
641
316
                note("Checking repository at '%s'."
432
                    if do_tree:
642
317
                     % (repo.bzrdir.root_transport.base,))
433
                        try:
643
318
                result = repo.check()
434
                            tree = branch.bzrdir.open_workingtree()
644
435
                            saw_tree = True
645
436
                        except (errors.NotLocalUrl, errors.NoWorkingTree):
646
437
                            pass
647
438
                        else:
648
439
                            scan_tree(base_tree, tree, needed_refs, to_unlock)
649
440
                    if do_branch:
650
441
                        scan_branch(branch, needed_refs, to_unlock)
651
442
            if do_branch and not branches:
652
443
                log_error("No branch found at specified location.")
653
444
            if do_tree and base_tree is None and not saw_tree:
654
445
                log_error("No working tree found at specified location.")
655
446
            if do_repo or do_branch or do_tree:
656
447
                if do_repo:
657
448
                    note("Checking repository at '%s'."
658
449
                         % (repo.bzrdir.root_transport.base,))
659
450
                result = repo.check(None, callback_refs=needed_refs,
660
451
                    check_repo=do_repo)
661
319
                result.report_results(verbose)
452
                result.report_results(verbose)
662
453
        else:
663
454
            if do_tree:
664
455
                log_error("No working tree found at specified location.")
665
320
            if do_branch:
456
            if do_branch:
680
321
                if branches == []:
457
                log_error("No branch found at specified location.")
681
322
                    log_error("No branch found at specified location.")
458
            if do_repo:
682
323
                else:
459
                log_error("No repository found at specified location.")
683
324
                    for branch in branches:
460
    finally:
684
325
                        note("Checking branch at '%s'."
461
        for thing in to_unlock:
685
326
                             % (branch.bzrdir.root_transport.base,))
462
            thing.unlock()
672
327
                        check_branch(branch, verbose)
673
328
        finally:
674
329
            repo.unlock()
675
330
    else:
676
331
        if do_branch:
677
332
            log_error("No branch found at specified location.")
678
333
        if do_repo:
679
334
            log_error("No repository found at specified location.")
686
335
463
687
=== modified file 'bzrlib/graph.py'
688
--- bzrlib/graph.py	2009-06-10 03:56:49 +0000
689
+++ bzrlib/graph.py	2009-06-16 05:29:42 +0000
690
@@ -311,6 +311,27 @@
691
311
        # get there.
311
        # get there.
692
312
        return known_revnos[cur_tip] + num_steps
312
        return known_revnos[cur_tip] + num_steps
693
313
313
694
314
    def find_lefthand_distances(self, keys):
695
315
        """Find the distance to null for all the keys in keys.
696
316
697
317
        :param keys: keys to lookup.
698
318
        :return: A dict key->distance for all of keys.
699
319
        """
700
320
        # Optimisable by concurrent searching, but a random spread should get
701
321
        # some sort of hit rate.
702
322
        result = {}
703
323
        known_revnos = []
704
324
        ghosts = []
705
325
        for key in keys:
706
326
            try:
707
327
                known_revnos.append(
708
328
                    (key, self.find_distance_to_null(key, known_revnos)))
709
329
            except errors.GhostRevisionsHaveNoRevno:
710
330
                ghosts.append(key)
711
331
        for key in ghosts:
712
332
            known_revnos.append((key, -1))
713
333
        return dict(known_revnos)
714
334
715
314
    def find_unique_ancestors(self, unique_revision, common_revisions):
335
    def find_unique_ancestors(self, unique_revision, common_revisions):
716
315
        """Find the unique ancestors for a revision versus others.
336
        """Find the unique ancestors for a revision versus others.
717
316
337
718
317
338
719
=== modified file 'bzrlib/groupcompress.py'
720
--- bzrlib/groupcompress.py	2009-06-10 03:56:49 +0000
721
+++ bzrlib/groupcompress.py	2009-06-16 05:29:42 +0000
722
@@ -1049,11 +1049,14 @@
723
1049
                reannotate(parent_lines, lines, key, None, head_cache))
1049
                reannotate(parent_lines, lines, key, None, head_cache))
724
1050
        return parent_cache[key]
1050
        return parent_cache[key]
725
1051
1051
727
1052
    def check(self, progress_bar=None):
1052
    def check(self, progress_bar=None, keys=None):
728
1053
        """See VersionedFiles.check()."""
1053
        """See VersionedFiles.check()."""
732
1054
        keys = self.keys()
1054
        if keys is None:
733
1055
        for record in self.get_record_stream(keys, 'unordered', True):
1055
            keys = self.keys()
734
1056
            record.get_bytes_as('fulltext')
1056
            for record in self.get_record_stream(keys, 'unordered', True):
735
1057
                record.get_bytes_as('fulltext')
736
1058
        else:
737
1059
            return self.get_record_stream(keys, 'unordered', True)
738
1057
1060
739
1058
    def _check_add(self, key, lines, random_id, check_content):
1061
    def _check_add(self, key, lines, random_id, check_content):
740
1059
        """check that version_id and lines are safe to add."""
1062
        """check that version_id and lines are safe to add."""
741
1060
1063
742
=== modified file 'bzrlib/help_topics/en/debug-flags.txt'
743
--- bzrlib/help_topics/en/debug-flags.txt	2009-03-18 23:43:51 +0000
744
+++ bzrlib/help_topics/en/debug-flags.txt	2009-05-12 06:30:56 +0000
745
@@ -19,6 +19,7 @@
746
19
-Dindex           Trace major index operations.
19
-Dindex           Trace major index operations.
747
20
-Dknit            Trace knit operations.
20
-Dknit            Trace knit operations.
748
21
-Dlock            Trace when lockdir locks are taken or released.
21
-Dlock            Trace when lockdir locks are taken or released.
749
22
-Dprogress        Trace progress bar operations.
750
22
-Dmerge           Emit information for debugging merges.
23
-Dmerge           Emit information for debugging merges.
751
23
-Dpack            Emit information about pack operations.
24
-Dpack            Emit information about pack operations.
752
24
-Dsftp            Trace SFTP internals.
25
-Dsftp            Trace SFTP internals.
753
25
26
754
=== modified file 'bzrlib/inventory.py'
755
--- bzrlib/inventory.py	2009-06-10 03:56:49 +0000
756
+++ bzrlib/inventory.py	2009-06-16 05:29:42 +0000
757
@@ -262,7 +262,7 @@
758
262
    def versionable_kind(kind):
262
    def versionable_kind(kind):
759
263
        return (kind in ('file', 'directory', 'symlink', 'tree-reference'))
263
        return (kind in ('file', 'directory', 'symlink', 'tree-reference'))
760
264
264
762
265
    def check(self, checker, rev_id, inv, tree):
265
    def check(self, checker, rev_id, inv):
763
266
        """Check this inventory entry is intact.
266
        """Check this inventory entry is intact.
764
267
267
765
268
        This is a template method, override _check for kind specific
268
        This is a template method, override _check for kind specific
766
@@ -274,18 +274,18 @@
767
274
        :param rev_id: Revision id from which this InventoryEntry was loaded.
274
        :param rev_id: Revision id from which this InventoryEntry was loaded.
768
275
             Not necessarily the last-changed revision for this file.
275
             Not necessarily the last-changed revision for this file.
769
276
        :param inv: Inventory from which the entry was loaded.
276
        :param inv: Inventory from which the entry was loaded.
770
277
        :param tree: RevisionTree for this entry.
771
278
        """
277
        """
772
279
        if self.parent_id is not None:
278
        if self.parent_id is not None:
773
280
            if not inv.has_id(self.parent_id):
279
            if not inv.has_id(self.parent_id):
774
281
                raise BzrCheckError('missing parent {%s} in inventory for revision {%s}'
280
                raise BzrCheckError('missing parent {%s} in inventory for revision {%s}'
775
282
                        % (self.parent_id, rev_id))
281
                        % (self.parent_id, rev_id))
777
283
        self._check(checker, rev_id, tree)
282
        checker._add_entry_to_text_key_references(inv, self)
778
283
        self._check(checker, rev_id)
779
284
284
781
285
    def _check(self, checker, rev_id, tree):
285
    def _check(self, checker, rev_id):
782
286
        """Check this inventory entry for kind specific errors."""
286
        """Check this inventory entry for kind specific errors."""
785
287
        raise BzrCheckError('unknown entry kind %r in revision {%s}' %
287
        checker._report_items.append(
786
288
                            (self.kind, rev_id))
288
            'unknown entry kind %r in revision {%s}' % (self.kind, rev_id))
787
289
289
788
290
    def copy(self):
290
    def copy(self):
789
291
        """Clone this inventory entry."""
291
        """Clone this inventory entry."""
790
@@ -404,7 +404,7 @@
791
404
                 'text_id', 'parent_id', 'children', 'executable',
404
                 'text_id', 'parent_id', 'children', 'executable',
792
405
                 'revision', 'symlink_target', 'reference_revision']
405
                 'revision', 'symlink_target', 'reference_revision']
793
406
406
795
407
    def _check(self, checker, rev_id, tree):
407
    def _check(self, checker, rev_id):
796
408
        """See InventoryEntry._check"""
408
        """See InventoryEntry._check"""
797
409
409
798
410
    def __init__(self, file_id):
410
    def __init__(self, file_id):
799
@@ -433,11 +433,16 @@
800
433
                 'text_id', 'parent_id', 'children', 'executable',
433
                 'text_id', 'parent_id', 'children', 'executable',
801
434
                 'revision', 'symlink_target', 'reference_revision']
434
                 'revision', 'symlink_target', 'reference_revision']
802
435
435
804
436
    def _check(self, checker, rev_id, tree):
436
    def _check(self, checker, rev_id):
805
437
        """See InventoryEntry._check"""
437
        """See InventoryEntry._check"""
808
438
        if self.text_sha1 is not None or self.text_size is not None or self.text_id is not None:
438
        if (self.text_sha1 is not None or self.text_size is not None or
809
439
            raise BzrCheckError('directory {%s} has text in revision {%s}'
439
            self.text_id is not None):
810
440
            checker._report_items.append('directory {%s} has text in revision {%s}'
811
440
                                % (self.file_id, rev_id))
441
                                % (self.file_id, rev_id))
812
442
        # Directories are stored as ''.
813
443
        checker.add_pending_item(rev_id,
814
444
            ('texts', self.file_id, self.revision), 'text',
815
445
             'da39a3ee5e6b4b0d3255bfef95601890afd80709')
816
441
446
817
442
    def copy(self):
447
    def copy(self):
818
443
        other = InventoryDirectory(self.file_id, self.name, self.parent_id)
448
        other = InventoryDirectory(self.file_id, self.name, self.parent_id)
819
@@ -476,27 +481,16 @@
820
476
                 'text_id', 'parent_id', 'children', 'executable',
481
                 'text_id', 'parent_id', 'children', 'executable',
821
477
                 'revision', 'symlink_target', 'reference_revision']
482
                 'revision', 'symlink_target', 'reference_revision']
822
478
483
824
479
    def _check(self, checker, tree_revision_id, tree):
484
    def _check(self, checker, tree_revision_id):
825
480
        """See InventoryEntry._check"""
485
        """See InventoryEntry._check"""
845
481
        key = (self.file_id, self.revision)
486
        # TODO: check size too.
846
482
        if key in checker.checked_texts:
487
        checker.add_pending_item(tree_revision_id,
847
483
            prev_sha = checker.checked_texts[key]
488
            ('texts', self.file_id, self.revision), 'text',
848
484
            if prev_sha != self.text_sha1:
489
             self.text_sha1)
849
485
                raise BzrCheckError(
490
        if self.text_size is None:
850
486
                    'mismatched sha1 on {%s} in {%s} (%s != %s) %r' %
491
            checker._report_items.append(
851
487
                    (self.file_id, tree_revision_id, prev_sha, self.text_sha1,
492
                'fileid {%s} in {%s} has None for text_size' % (self.file_id,
852
488
                     t))
493
                tree_revision_id))
834
489
            else:
835
490
                checker.repeated_text_cnt += 1
836
491
                return
837
492
838
493
        checker.checked_text_cnt += 1
839
494
        # We can't check the length, because Weave doesn't store that
840
495
        # information, and the whole point of looking at the weave's
841
496
        # sha1sum is that we don't have to extract the text.
842
497
        if (self.text_sha1 != tree._repository.texts.get_sha1s([key])[key]):
843
498
            raise BzrCheckError('text {%s} version {%s} wrong sha1' % key)
844
499
        checker.checked_texts[key] = self.text_sha1
853
500
494
854
501
    def copy(self):
495
    def copy(self):
855
502
        other = InventoryFile(self.file_id, self.name, self.parent_id)
496
        other = InventoryFile(self.file_id, self.name, self.parent_id)
856
@@ -600,14 +594,20 @@
857
600
                 'text_id', 'parent_id', 'children', 'executable',
594
                 'text_id', 'parent_id', 'children', 'executable',
858
601
                 'revision', 'symlink_target', 'reference_revision']
595
                 'revision', 'symlink_target', 'reference_revision']
859
602
596
861
603
    def _check(self, checker, rev_id, tree):
597
    def _check(self, checker, rev_id):
862
604
        """See InventoryEntry._check"""
598
        """See InventoryEntry._check"""
863
605
        if self.text_sha1 is not None or self.text_size is not None or self.text_id is not None:
599
        if self.text_sha1 is not None or self.text_size is not None or self.text_id is not None:
865
606
            raise BzrCheckError('symlink {%s} has text in revision {%s}'
600
            checker._report_items.append(
866
601
               'symlink {%s} has text in revision {%s}'
867
607
                    % (self.file_id, rev_id))
602
                    % (self.file_id, rev_id))
868
608
        if self.symlink_target is None:
603
        if self.symlink_target is None:
870
609
            raise BzrCheckError('symlink {%s} has no target in revision {%s}'
604
            checker._report_items.append(
871
605
                'symlink {%s} has no target in revision {%s}'
872
610
                    % (self.file_id, rev_id))
606
                    % (self.file_id, rev_id))
873
607
        # Symlinks are stored as ''
874
608
        checker.add_pending_item(tree_revision_id,
875
609
            ('texts', self.file_id, self.revision), 'text',
876
610
             'da39a3ee5e6b4b0d3255bfef95601890afd80709')
877
611
611
878
612
    def copy(self):
612
    def copy(self):
879
613
        other = InventoryLink(self.file_id, self.name, self.parent_id)
613
        other = InventoryLink(self.file_id, self.name, self.parent_id)
880
614
614
881
=== modified file 'bzrlib/knit.py'
882
--- bzrlib/knit.py	2009-06-10 03:56:49 +0000
883
+++ bzrlib/knit.py	2009-06-16 05:29:42 +0000
884
@@ -1005,8 +1005,15 @@
885
1005
        """See VersionedFiles.annotate."""
1005
        """See VersionedFiles.annotate."""
886
1006
        return self._factory.annotate(self, key)
1006
        return self._factory.annotate(self, key)
887
1007
1007
889
1008
    def check(self, progress_bar=None):
1008
    def check(self, progress_bar=None, keys=None):
890
1009
        """See VersionedFiles.check()."""
1009
        """See VersionedFiles.check()."""
891
1010
        if keys is None:
892
1011
            return self._logical_check()
893
1012
        else:
894
1013
            # At the moment, check does not extra work over get_record_stream
895
1014
            return self.get_record_stream(keys, 'unordered', True)
896
1015
897
1016
    def _logical_check(self):
898
1010
        # This doesn't actually test extraction of everything, but that will
1017
        # This doesn't actually test extraction of everything, but that will
899
1011
        # impact 'bzr check' substantially, and needs to be integrated with
1018
        # impact 'bzr check' substantially, and needs to be integrated with
900
1012
        # care. However, it does check for the obvious problem of a delta with
1019
        # care. However, it does check for the obvious problem of a delta with
901
1013
1020
902
=== modified file 'bzrlib/remote.py'
903
--- bzrlib/remote.py	2009-06-11 09:11:21 +0000
904
+++ bzrlib/remote.py	2009-06-16 05:29:42 +0000
905
@@ -1414,9 +1414,10 @@
906
1414
        return self._real_repository.get_revision_reconcile(revision_id)
1414
        return self._real_repository.get_revision_reconcile(revision_id)
907
1415
1415
908
1416
    @needs_read_lock
1416
    @needs_read_lock
910
1417
    def check(self, revision_ids=None):
1417
    def check(self, revision_ids=None, callback_refs=None):
911
1418
        self._ensure_real()
1418
        self._ensure_real()
913
1419
        return self._real_repository.check(revision_ids=revision_ids)
1419
        return self._real_repository.check(revision_ids=revision_ids,
914
1420
            callback_refs=callback_refs)
915
1420
1421
916
1421
    def copy_content_into(self, destination, revision_id=None):
1422
    def copy_content_into(self, destination, revision_id=None):
917
1422
        self._ensure_real()
1423
        self._ensure_real()
918
1423
1424
919
=== modified file 'bzrlib/repofmt/groupcompress_repo.py'
920
--- bzrlib/repofmt/groupcompress_repo.py	2009-06-12 01:11:00 +0000
921
+++ bzrlib/repofmt/groupcompress_repo.py	2009-06-16 05:29:42 +0000
922
@@ -718,6 +718,10 @@
923
718
        finally:
718
        finally:
924
719
            basis_tree.unlock()
719
            basis_tree.unlock()
925
720
720
926
721
    def deserialise_inventory(self, revision_id, bytes):
927
722
        return inventory.CHKInventory.deserialise(self.chk_bytes, bytes,
928
723
            (revision_id,))
929
724
930
721
    def _iter_inventories(self, revision_ids):
725
    def _iter_inventories(self, revision_ids):
931
722
        """Iterate over many inventory objects."""
726
        """Iterate over many inventory objects."""
932
723
        keys = [(revision_id,) for revision_id in revision_ids]
727
        keys = [(revision_id,) for revision_id in revision_ids]
933
724
728
934
=== modified file 'bzrlib/repofmt/knitrepo.py'
935
--- bzrlib/repofmt/knitrepo.py	2009-06-10 03:56:49 +0000
936
+++ bzrlib/repofmt/knitrepo.py	2009-06-16 05:29:42 +0000
937
@@ -229,24 +229,29 @@
938
229
    def _make_parents_provider(self):
229
    def _make_parents_provider(self):
939
230
        return _KnitsParentsProvider(self.revisions)
230
        return _KnitsParentsProvider(self.revisions)
940
231
231
942
232
    def _find_inconsistent_revision_parents(self):
232
    def _find_inconsistent_revision_parents(self, revisions_iterator=None):
943
233
        """Find revisions with different parent lists in the revision object
233
        """Find revisions with different parent lists in the revision object
944
234
        and in the index graph.
234
        and in the index graph.
945
235
235
946
236
        :param revisions_iterator: None, or an iterator of (revid,
947
237
            Revision-or-None). This iterator controls the revisions checked.
948
236
        :returns: an iterator yielding tuples of (revison-id, parents-in-index,
238
        :returns: an iterator yielding tuples of (revison-id, parents-in-index,
949
237
            parents-in-revision).
239
            parents-in-revision).
950
238
        """
240
        """
951
239
        if not self.is_locked():
241
        if not self.is_locked():
952
240
            raise AssertionError()
242
            raise AssertionError()
953
241
        vf = self.revisions
243
        vf = self.revisions
956
242
        for index_version in vf.keys():
244
        if revisions_iterator is None:
957
243
            parent_map = vf.get_parent_map([index_version])
245
            revisions_iterator = self._iter_revisions(None)
958
246
        for revid, revision in revisions_iterator:
959
247
            if revision is None:
960
248
                pass
961
249
            parent_map = vf.get_parent_map([(revid,)])
962
244
            parents_according_to_index = tuple(parent[-1] for parent in
250
            parents_according_to_index = tuple(parent[-1] for parent in
965
245
                parent_map[index_version])
251
                parent_map[(revid,)])
964
246
            revision = self.get_revision(index_version[-1])
966
247
            parents_according_to_revision = tuple(revision.parent_ids)
252
            parents_according_to_revision = tuple(revision.parent_ids)
967
248
            if parents_according_to_index != parents_according_to_revision:
253
            if parents_according_to_index != parents_according_to_revision:
969
249
                yield (index_version[-1], parents_according_to_index,
254
                yield (revid, parents_according_to_index,
970
250
                    parents_according_to_revision)
255
                    parents_according_to_revision)
971
251
256
972
252
    def _check_for_inconsistent_revision_parents(self):
257
    def _check_for_inconsistent_revision_parents(self):
973
253
258
974
=== modified file 'bzrlib/repofmt/pack_repo.py'
975
--- bzrlib/repofmt/pack_repo.py	2009-06-10 03:56:49 +0000
976
+++ bzrlib/repofmt/pack_repo.py	2009-06-16 05:29:42 +0000
977
@@ -2219,52 +2219,6 @@
978
2219
        self.revisions._index._key_dependencies.refs.clear()
2219
        self.revisions._index._key_dependencies.refs.clear()
979
2220
        self._pack_collection._abort_write_group()
2220
        self._pack_collection._abort_write_group()
980
2221
2221
981
2222
    def _find_inconsistent_revision_parents(self):
982
2223
        """Find revisions with incorrectly cached parents.
983
2224
984
2225
        :returns: an iterator yielding tuples of (revison-id, parents-in-index,
985
2226
            parents-in-revision).
986
2227
        """
987
2228
        if not self.is_locked():
988
2229
            raise errors.ObjectNotLocked(self)
989
2230
        pb = ui.ui_factory.nested_progress_bar()
990
2231
        result = []
991
2232
        try:
992
2233
            revision_nodes = self._pack_collection.revision_index \
993
2234
                .combined_index.iter_all_entries()
994
2235
            index_positions = []
995
2236
            # Get the cached index values for all revisions, and also the
996
2237
            # location in each index of the revision text so we can perform
997
2238
            # linear IO.
998
2239
            for index, key, value, refs in revision_nodes:
999
2240
                node = (index, key, value, refs)
1000
2241
                index_memo = self.revisions._index._node_to_position(node)
1001
2242
                if index_memo[0] != index:
1002
2243
                    raise AssertionError('%r != %r' % (index_memo[0], index))
1003
2244
                index_positions.append((index_memo, key[0],
1004
2245
                                       tuple(parent[0] for parent in refs[0])))
1005
2246
                pb.update("Reading revision index", 0, 0)
1006
2247
            index_positions.sort()
1007
2248
            batch_size = 1000
1008
2249
            pb.update("Checking cached revision graph", 0,
1009
2250
                      len(index_positions))
1010
2251
            for offset in xrange(0, len(index_positions), 1000):
1011
2252
                pb.update("Checking cached revision graph", offset)
1012
2253
                to_query = index_positions[offset:offset + batch_size]
1013
2254
                if not to_query:
1014
2255
                    break
1015
2256
                rev_ids = [item[1] for item in to_query]
1016
2257
                revs = self.get_revisions(rev_ids)
1017
2258
                for revision, item in zip(revs, to_query):
1018
2259
                    index_parents = item[2]
1019
2260
                    rev_parents = tuple(revision.parent_ids)
1020
2261
                    if index_parents != rev_parents:
1021
2262
                        result.append((revision.revision_id, index_parents,
1022
2263
                                       rev_parents))
1023
2264
        finally:
1024
2265
            pb.finished()
1025
2266
        return result
1026
2267
1027
2268
    def _make_parents_provider(self):
2222
    def _make_parents_provider(self):
1028
2269
        return graph.CachingParentsProvider(self)
2223
        return graph.CachingParentsProvider(self)
1029
2270
2224
1030
2271
2225
1031
=== modified file 'bzrlib/repository.py'
1032
--- bzrlib/repository.py	2009-06-12 01:11:00 +0000
1033
+++ bzrlib/repository.py	2009-06-16 05:29:42 +0000
1034
@@ -1154,6 +1154,119 @@
1035
1154
        # The old API returned a list, should this actually be a set?
1154
        # The old API returned a list, should this actually be a set?
1036
1155
        return parent_map.keys()
1155
        return parent_map.keys()
1037
1156
1156
1038
1157
    def _check_inventories(self, checker):
1039
1158
        """Check the inventories found from the revision scan.
1040
1159
        
1041
1160
        This is responsible for verifying the sha1 of inventories and
1042
1161
        creating a pending_keys set that covers data referenced by inventories.
1043
1162
        """
1044
1163
        bar = ui.ui_factory.nested_progress_bar()
1045
1164
        try:
1046
1165
            self._do_check_inventories(checker, bar)
1047
1166
        finally:
1048
1167
            bar.finished()
1049
1168
1050
1169
    def _do_check_inventories(self, checker, bar):
1051
1170
        """Helper for _check_inventories."""
1052
1171
        revno = 0
1053
1172
        keys = {'chk_bytes':set(), 'inventories':set(), 'texts':set()}
1054
1173
        kinds = ['chk_bytes', 'texts']
1055
1174
        count = len(checker.pending_keys)
1056
1175
        bar.update("inventories", 0, 2)
1057
1176
        current_keys = checker.pending_keys
1058
1177
        checker.pending_keys = {}
1059
1178
        # Accumulate current checks.
1060
1179
        for key in current_keys:
1061
1180
            if key[0] != 'inventories' and key[0] not in kinds:
1062
1181
                checker._report_items.append('unknown key type %r' % (key,))
1063
1182
            keys[key[0]].add(key[1:])
1064
1183
        if keys['inventories']:
1065
1184
            # NB: output order *should* be roughly sorted - topo or
1066
1185
            # inverse topo depending on repository - either way decent
1067
1186
            # to just delta against. However, pre-CHK formats didn't
1068
1187
            # try to optimise inventory layout on disk. As such the
1069
1188
            # pre-CHK code path does not use inventory deltas.
1070
1189
            last_object = None
1071
1190
            for record in self.inventories.check(keys=keys['inventories']):
1072
1191
                if record.storage_kind == 'absent':
1073
1192
                    checker._report_items.append(
1074
1193
                        'Missing inventory {%s}' % (record.key,))
1075
1194
                else:
1076
1195
                    last_object = self._check_record('inventories', record,
1077
1196
                        checker, last_object,
1078
1197
                        current_keys[('inventories',) + record.key])
1079
1198
            del keys['inventories']
1080
1199
        else:
1081
1200
            return
1082
1201
        bar.update("texts", 1)
1083
1202
        while (checker.pending_keys or keys['chk_bytes']
1084
1203
            or keys['texts']):
1085
1204
            # Something to check.
1086
1205
            current_keys = checker.pending_keys
1087
1206
            checker.pending_keys = {}
1088
1207
            # Accumulate current checks.
1089
1208
            for key in current_keys:
1090
1209
                if key[0] not in kinds:
1091
1210
                    checker._report_items.append('unknown key type %r' % (key,))
1092
1211
                keys[key[0]].add(key[1:])
1093
1212
            # Check the outermost kind only - inventories || chk_bytes || texts
1094
1213
            for kind in kinds:
1095
1214
                if keys[kind]:
1096
1215
                    last_object = None
1097
1216
                    for record in getattr(self, kind).check(keys=keys[kind]):
1098
1217
                        if record.storage_kind == 'absent':
1099
1218
                            checker._report_items.append(
1100
1219
                                'Missing inventory {%s}' % (record.key,))
1101
1220
                        else:
1102
1221
                            last_object = self._check_record(kind, record,
1103
1222
                                checker, last_object, current_keys[(kind,) + record.key])
1104
1223
                    keys[kind] = set()
1105
1224
                    break
1106
1225
1107
1226
    def _check_record(self, kind, record, checker, last_object, item_data):
1108
1227
        """Check a single text from this repository."""
1109
1228
        if kind == 'inventories':
1110
1229
            rev_id = record.key[0]
1111
1230
            inv = self.deserialise_inventory(rev_id,
1112
1231
                record.get_bytes_as('fulltext'))
1113
1232
            if last_object is not None:
1114
1233
                delta = inv._make_delta(last_object)
1115
1234
                for old_path, path, file_id, ie in delta:
1116
1235
                    if ie is None:
1117
1236
                        continue
1118
1237
                    ie.check(checker, rev_id, inv)
1119
1238
            else:
1120
1239
                for path, ie in inv.iter_entries():
1121
1240
                    ie.check(checker, rev_id, inv)
1122
1241
            if self._format.fast_deltas:
1123
1242
                return inv
1124
1243
        elif kind == 'chk_bytes':
1125
1244
            # No code written to check chk_bytes for this repo format.
1126
1245
            checker._report_items.append(
1127
1246
                'unsupported key type chk_bytes for %s' % (record.key,))
1128
1247
        elif kind == 'texts':
1129
1248
            self._check_text(record, checker, item_data)
1130
1249
        else:
1131
1250
            checker._report_items.append(
1132
1251
                'unknown key type %s for %s' % (kind, record.key))
1133
1252
1134
1253
    def _check_text(self, record, checker, item_data):
1135
1254
        """Check a single text."""
1136
1255
        # Check it is extractable.
1137
1256
        # TODO: check length.
1138
1257
        if record.storage_kind == 'chunked':
1139
1258
            chunks = record.get_bytes_as(record.storage_kind)
1140
1259
            sha1 = osutils.sha_strings(chunks)
1141
1260
            length = sum(map(len, chunks))
1142
1261
        else:
1143
1262
            content = record.get_bytes_as('fulltext')
1144
1263
            sha1 = osutils.sha_string(content)
1145
1264
            length = len(content)
1146
1265
        if item_data and sha1 != item_data[1]:
1147
1266
            checker._report_items.append(
1148
1267
                'sha1 mismatch: %s has sha1 %s expected %s referenced by %s' %
1149
1268
                (record.key, sha1, item_data[1], item_data[2]))
1150
1269
1151
1157
    @staticmethod
1270
    @staticmethod
1152
1158
    def create(a_bzrdir):
1271
    def create(a_bzrdir):
1153
1159
        """Construct the current default format repository in a_bzrdir."""
1272
        """Construct the current default format repository in a_bzrdir."""
1154
@@ -1700,25 +1813,49 @@
1155
1700
1813
1156
1701
    @needs_read_lock
1814
    @needs_read_lock
1157
1702
    def get_revisions(self, revision_ids):
1815
    def get_revisions(self, revision_ids):
1159
1703
        """Get many revisions at once."""
1816
        """Get many revisions at once.
1160
1817
        
1161
1818
        Repositories that need to check data on every revision read should 
1162
1819
        subclass this method.
1163
1820
        """
1164
1704
        return self._get_revisions(revision_ids)
1821
        return self._get_revisions(revision_ids)
1165
1705
1822
1166
1706
    @needs_read_lock
1823
    @needs_read_lock
1167
1707
    def _get_revisions(self, revision_ids):
1824
    def _get_revisions(self, revision_ids):
1168
1708
        """Core work logic to get many revisions without sanity checks."""
1825
        """Core work logic to get many revisions without sanity checks."""
1172
1709
        for rev_id in revision_ids:
1826
        revs = {}
1173
1710
            if not rev_id or not isinstance(rev_id, basestring):
1827
        for revid, rev in self._iter_revisions(revision_ids):
1174
1711
                raise errors.InvalidRevisionId(revision_id=rev_id, branch=self)
1828
            if rev is None:
1175
1829
                raise errors.NoSuchRevision(self, revid)
1176
1830
            revs[revid] = rev
1177
1831
        return [revs[revid] for revid in revision_ids]
1178
1832
1179
1833
    def _iter_revisions(self, revision_ids):
1180
1834
        """Iterate over revision objects.
1181
1835
1182
1836
        :param revision_ids: An iterable of revisions to examine. None may be
1183
1837
            passed to request all revisions known to the repository. Note that
1184
1838
            not all repositories can find unreferenced revisions; for those
1185
1839
            repositories only referenced ones will be returned.
1186
1840
        :return: An iterator of (revid, revision) tuples. Absent revisions (
1187
1841
            those asked for but not available) are returned as (revid, None).
1188
1842
        """
1189
1843
        if revision_ids is None:
1190
1844
            revision_ids = self.all_revision_ids()
1191
1845
        else:
1192
1846
            for rev_id in revision_ids:
1193
1847
                if not rev_id or not isinstance(rev_id, basestring):
1194
1848
                    raise errors.InvalidRevisionId(revision_id=rev_id, branch=self)
1195
1712
        keys = [(key,) for key in revision_ids]
1849
        keys = [(key,) for key in revision_ids]
1196
1713
        stream = self.revisions.get_record_stream(keys, 'unordered', True)
1850
        stream = self.revisions.get_record_stream(keys, 'unordered', True)
1197
1714
        revs = {}
1198
1715
        for record in stream:
1851
        for record in stream:
1199
1852
            revid = record.key[0]
1200
1716
            if record.storage_kind == 'absent':
1853
            if record.storage_kind == 'absent':
1206
1717
                raise errors.NoSuchRevision(self, record.key[0])
1854
                yield (revid, None)
1207
1718
            text = record.get_bytes_as('fulltext')
1855
            else:
1208
1719
            rev = self._serializer.read_revision_from_string(text)
1856
                text = record.get_bytes_as('fulltext')
1209
1720
            revs[record.key[0]] = rev
1857
                rev = self._serializer.read_revision_from_string(text)
1210
1721
        return [revs[revid] for revid in revision_ids]
1858
                yield (revid, rev)
1211
1722
1859
1212
1723
    @needs_read_lock
1860
    @needs_read_lock
1213
1724
    def get_revision_xml(self, revision_id):
1861
    def get_revision_xml(self, revision_id):
1214
@@ -2056,8 +2193,7 @@
1215
2056
                batch_size]
2193
                batch_size]
1216
2057
            if not to_query:
2194
            if not to_query:
1217
2058
                break
2195
                break
1220
2059
            for rev_tree in self.revision_trees(to_query):
2196
            for revision_id in to_query:
1219
2060
                revision_id = rev_tree.get_revision_id()
1221
2061
                parent_ids = ancestors[revision_id]
2197
                parent_ids = ancestors[revision_id]
1222
2062
                for text_key in revision_keys[revision_id]:
2198
                for text_key in revision_keys[revision_id]:
1223
2063
                    pb.update("Calculating text parents", processed_texts)
2199
                    pb.update("Calculating text parents", processed_texts)
1224
@@ -2417,7 +2553,8 @@
1225
2417
                [parents_provider, other_repository._make_parents_provider()])
2553
                [parents_provider, other_repository._make_parents_provider()])
1226
2418
        return graph.Graph(parents_provider)
2554
        return graph.Graph(parents_provider)
1227
2419
2555
1229
2420
    def _get_versioned_file_checker(self, text_key_references=None):
2556
    def _get_versioned_file_checker(self, text_key_references=None,
1230
2557
        ancestors=None):
1231
2421
        """Return an object suitable for checking versioned files.
2558
        """Return an object suitable for checking versioned files.
1232
2422
        
2559
        
1233
2423
        :param text_key_references: if non-None, an already built
2560
        :param text_key_references: if non-None, an already built
1234
@@ -2425,9 +2562,12 @@
1235
2425
            to whether they were referred to by the inventory of the
2562
            to whether they were referred to by the inventory of the
1236
2426
            revision_id that they contain. If None, this will be
2563
            revision_id that they contain. If None, this will be
1237
2427
            calculated.
2564
            calculated.
1238
2565
        :param ancestors: Optional result from
1239
2566
            self.get_graph().get_parent_map(self.all_revision_ids()) if already
1240
2567
            available.
1241
2428
        """
2568
        """
1242
2429
        return _VersionedFileChecker(self,
2569
        return _VersionedFileChecker(self,
1244
2430
            text_key_references=text_key_references)
2570
            text_key_references=text_key_references, ancestors=ancestors)
1245
2431
2571
1246
2432
    def revision_ids_to_search_result(self, result_set):
2572
    def revision_ids_to_search_result(self, result_set):
1247
2433
        """Convert a set of revision ids to a graph SearchResult."""
2573
        """Convert a set of revision ids to a graph SearchResult."""
1248
@@ -2483,19 +2623,25 @@
1249
2483
        return record.get_bytes_as('fulltext')
2623
        return record.get_bytes_as('fulltext')
1250
2484
2624
1251
2485
    @needs_read_lock
2625
    @needs_read_lock
1253
2486
    def check(self, revision_ids=None):
2626
    def check(self, revision_ids=None, callback_refs=None, check_repo=True):
1254
2487
        """Check consistency of all history of given revision_ids.
2627
        """Check consistency of all history of given revision_ids.
1255
2488
2628
1256
2489
        Different repository implementations should override _check().
2629
        Different repository implementations should override _check().
1257
2490
2630
1258
2491
        :param revision_ids: A non-empty list of revision_ids whose ancestry
2631
        :param revision_ids: A non-empty list of revision_ids whose ancestry
1259
2492
             will be checked.  Typically the last revision_id of a branch.
2632
             will be checked.  Typically the last revision_id of a branch.
1260
2633
        :param callback_refs: A dict of check-refs to resolve and callback
1261
2634
            the check/_check method on the items listed as wanting the ref.
1262
2635
            see bzrlib.check.
1263
2636
        :param check_repo: If False do not check the repository contents, just 
1264
2637
            calculate the data callback_refs requires and call them back.
1265
2493
        """
2638
        """
1267
2494
        return self._check(revision_ids)
2639
        return self._check(revision_ids, callback_refs=callback_refs,
1268
2640
            check_repo=check_repo)
1269
2495
2641
1273
2496
    def _check(self, revision_ids):
2642
    def _check(self, revision_ids, callback_refs, check_repo):
1274
2497
        result = check.Check(self)
2643
        result = check.Check(self, check_repo=check_repo)
1275
2498
        result.check()
2644
        result.check(callback_refs)
1276
2499
        return result
2645
        return result
1277
2500
2646
1278
2501
    def _warn_if_deprecated(self):
2647
    def _warn_if_deprecated(self):
1279
@@ -3923,10 +4069,10 @@
1280
3923
4069
1281
3924
class _VersionedFileChecker(object):
4070
class _VersionedFileChecker(object):
1282
3925
4071
1284
3926
    def __init__(self, repository, text_key_references=None):
4072
    def __init__(self, repository, text_key_references=None, ancestors=None):
1285
3927
        self.repository = repository
4073
        self.repository = repository
1286
3928
        self.text_index = self.repository._generate_text_key_index(
4074
        self.text_index = self.repository._generate_text_key_index(
1288
3929
            text_key_references=text_key_references)
4075
            text_key_references=text_key_references, ancestors=ancestors)
1289
3930
4076
1290
3931
    def calculate_file_version_parents(self, text_key):
4077
    def calculate_file_version_parents(self, text_key):
1291
3932
        """Calculate the correct parents for a file version according to
4078
        """Calculate the correct parents for a file version according to
1292
@@ -3950,6 +4096,18 @@
1293
3950
            revision_id) tuples for versions that are present in this versioned
4096
            revision_id) tuples for versions that are present in this versioned
1294
3951
            file, but not used by the corresponding inventory.
4097
            file, but not used by the corresponding inventory.
1295
3952
        """
4098
        """
1296
4099
        local_progress = None
1297
4100
        if progress_bar is None:
1298
4101
            local_progress = ui.ui_factory.nested_progress_bar()
1299
4102
            progress_bar = local_progress
1300
4103
        try:
1301
4104
            return self._check_file_version_parents(texts, progress_bar)
1302
4105
        finally:
1303
4106
            if local_progress:
1304
4107
                local_progress.finished()
1305
4108
1306
4109
    def _check_file_version_parents(self, texts, progress_bar):
1307
4110
        """See check_file_version_parents."""
1308
3953
        wrong_parents = {}
4111
        wrong_parents = {}
1309
3954
        self.file_ids = set([file_id for file_id, _ in
4112
        self.file_ids = set([file_id for file_id, _ in
1310
3955
            self.text_index.iterkeys()])
4113
            self.text_index.iterkeys()])
1311
@@ -3964,8 +4122,7 @@
1312
3964
        text_keys = self.repository.texts.keys()
4122
        text_keys = self.repository.texts.keys()
1313
3965
        unused_keys = frozenset(text_keys) - set(self.text_index)
4123
        unused_keys = frozenset(text_keys) - set(self.text_index)
1314
3966
        for num, key in enumerate(self.text_index.iterkeys()):
4124
        for num, key in enumerate(self.text_index.iterkeys()):
1317
3967
            if progress_bar is not None:
4125
            progress_bar.update('checking text graph', num, n_versions)
1316
3968
                progress_bar.update('checking text graph', num, n_versions)
1318
3969
            correct_parents = self.calculate_file_version_parents(key)
4126
            correct_parents = self.calculate_file_version_parents(key)
1319
3970
            try:
4127
            try:
1320
3971
                knit_parents = parent_map[key]
4128
                knit_parents = parent_map[key]
1321
3972
4129
1322
=== modified file 'bzrlib/smart/medium.py'
1323
--- bzrlib/smart/medium.py	2009-06-10 03:56:49 +0000
1324
+++ bzrlib/smart/medium.py	2009-06-16 05:29:42 +0000
1325
@@ -37,7 +37,6 @@
1326
37
from bzrlib import (
37
from bzrlib import (
1327
38
    debug,
38
    debug,
1328
39
    errors,
39
    errors,
1329
40
    osutils,
1330
41
    symbol_versioning,
40
    symbol_versioning,
1331
42
    trace,
41
    trace,
1332
43
    ui,
42
    ui,
1333
@@ -46,7 +45,8 @@
1334
46
from bzrlib.smart import client, protocol, request, vfs
45
from bzrlib.smart import client, protocol, request, vfs
1335
47
from bzrlib.transport import ssh
46
from bzrlib.transport import ssh
1336
48
""")
47
""")
1338
49
48
#usually already imported, and getting IllegalScoperReplacer on it here.
1339
49
from bzrlib import osutils
1340
50
50
1341
51
# We must not read any more than 64k at a time so we don't risk "no buffer
51
# We must not read any more than 64k at a time so we don't risk "no buffer
1342
52
# space available" errors on some platforms.  Windows in particular is likely
52
# space available" errors on some platforms.  Windows in particular is likely
1343
53
53
1344
=== modified file 'bzrlib/tests/blackbox/test_check.py'
1345
--- bzrlib/tests/blackbox/test_check.py	2009-03-23 14:59:43 +0000
1346
+++ bzrlib/tests/blackbox/test_check.py	2009-05-12 07:27:33 +0000
1347
@@ -34,23 +34,21 @@
1348
34
        tree = self.make_branch_and_tree('.')
34
        tree = self.make_branch_and_tree('.')
1349
35
        tree.commit('hallelujah')
35
        tree.commit('hallelujah')
1350
36
        out, err = self.run_bzr('check')
36
        out, err = self.run_bzr('check')
1354
37
        self.assertContainsRe(err, r"^Checking working tree at '.*'\.\n"
37
        self.assertContainsRe(err, r"Checking working tree at '.*'\.\n")
1355
38
                                   r"Checking repository at '.*'\.\n"
38
        self.assertContainsRe(err, r"Checking repository at '.*'\.\n")
1356
39
                                   r"checked repository.*\n"
39
        self.assertContainsRe(err, r"checked repository.*\n"
1357
40
                                   r"     1 revisions\n"
40
                                   r"     1 revisions\n"
1358
41
                                   r"     0 file-ids\n"
41
                                   r"     0 file-ids\n"
1364
42
                                   r"     0 unique file texts\n"
42
                                   )
1365
43
                                   r"     0 repeated file texts\n"
43
        self.assertContainsRe(err, r"Checking branch at '.*'\.\n")
1366
44
                                   r"     0 unreferenced text versions\n"
44
        self.assertContainsRe(err, r"checked branch.*")
1362
45
                                   r"Checking branch at '.*'\.\n"
1363
46
                                   r"checked branch.*\n$")
1367
47
45
1368
48
    def test_check_branch(self):
46
    def test_check_branch(self):
1369
49
        tree = self.make_branch_and_tree('.')
47
        tree = self.make_branch_and_tree('.')
1370
50
        tree.commit('foo')
48
        tree.commit('foo')
1371
51
        out, err = self.run_bzr('check --branch')
49
        out, err = self.run_bzr('check --branch')
1372
52
        self.assertContainsRe(err, r"^Checking branch at '.*'\.\n"
50
        self.assertContainsRe(err, r"^Checking branch at '.*'\.\n"
1374
53
                                   r"checked branch.*\n$")
51
                                   r"checked branch.*")
1375
54
52
1376
55
    def test_check_repository(self):
53
    def test_check_repository(self):
1377
56
        tree = self.make_branch_and_tree('.')
54
        tree = self.make_branch_and_tree('.')
1378
@@ -60,9 +58,7 @@
1379
60
                                   r"checked repository.*\n"
58
                                   r"checked repository.*\n"
1380
61
                                   r"     1 revisions\n"
59
                                   r"     1 revisions\n"
1381
62
                                   r"     0 file-ids\n"
60
                                   r"     0 file-ids\n"
1385
63
                                   r"     0 unique file texts\n"
61
                                   )
1383
64
                                   r"     0 repeated file texts\n"
1384
65
                                   r"     0 unreferenced text versions$")
1386
66
62
1387
67
    def test_check_tree(self):
63
    def test_check_tree(self):
1388
68
        tree = self.make_branch_and_tree('.')
64
        tree = self.make_branch_and_tree('.')
1389
@@ -76,7 +72,7 @@
1390
76
        out, err = self.run_bzr('check --tree --branch')
72
        out, err = self.run_bzr('check --tree --branch')
1391
77
        self.assertContainsRe(err, r"^Checking working tree at '.*'\.\n"
73
        self.assertContainsRe(err, r"^Checking working tree at '.*'\.\n"
1392
78
                                   r"Checking branch at '.*'\.\n"
74
                                   r"Checking branch at '.*'\.\n"
1394
79
                                   r"checked branch.*\n$")
75
                                   r"checked branch.*")
1395
80
76
1396
81
    def test_check_missing_tree(self):
77
    def test_check_missing_tree(self):
1397
82
        branch = self.make_branch('.')
78
        branch = self.make_branch('.')
1398
@@ -87,9 +83,9 @@
1399
87
        branch = self.make_branch('.')
83
        branch = self.make_branch('.')
1400
88
        out, err = self.run_bzr('check --tree --branch')
84
        out, err = self.run_bzr('check --tree --branch')
1401
89
        self.assertContainsRe(err,
85
        self.assertContainsRe(err,
1402
90
            r"^No working tree found at specified location\.\n"
1403
91
            r"Checking branch at '.*'\.\n"
86
            r"Checking branch at '.*'\.\n"
1405
92
            r"checked branch.*\n$")
87
            r"No working tree found at specified location\.\n"
1406
88
            r"checked branch.*")
1407
93
89
1408
94
    def test_check_missing_branch_in_shared_repo(self):
90
    def test_check_missing_branch_in_shared_repo(self):
1409
95
        self.make_repository('shared', shared=True)
91
        self.make_repository('shared', shared=True)
1410
96
92
1411
=== modified file 'bzrlib/tests/branch_implementations/test_check.py'
1412
--- bzrlib/tests/branch_implementations/test_check.py	2009-06-10 03:56:49 +0000
1413
+++ bzrlib/tests/branch_implementations/test_check.py	2009-06-16 05:29:42 +0000
1414
@@ -16,7 +16,9 @@
1415
16
16
1416
17
"""Tests for branch implementations - test check() functionality"""
17
"""Tests for branch implementations - test check() functionality"""
1417
18
18
1419
19
from bzrlib import errors
19
from StringIO import StringIO
1420
20
1421
21
from bzrlib import errors, tests, ui
1422
20
from bzrlib.tests.branch_implementations import TestCaseWithBranch
22
from bzrlib.tests.branch_implementations import TestCaseWithBranch
1423
21
23
1424
22
24
1425
@@ -54,24 +56,57 @@
1426
54
            # with set_last_revision_info
56
            # with set_last_revision_info
1427
55
            tree.branch.set_last_revision_info(3, r5)
57
            tree.branch.set_last_revision_info(3, r5)
1428
56
58
1433
57
        e = self.assertRaises(errors.BzrCheckError,
59
        tree.lock_read()
1434
58
                              tree.branch.check)
60
        self.addCleanup(tree.unlock)
1435
59
        self.assertEqual('Internal check failed:'
61
        refs = self.make_refs(tree.branch)
1436
60
                         ' revno does not match len(mainline) 3 != 5', str(e))
62
        result = tree.branch.check(refs)
1437
63
        ui.ui_factory = tests.TestUIFactory(stdout=StringIO())
1438
64
        result.report_results(True)
1439
65
        self.assertContainsRe('revno does not match len',
1440
66
            ui.ui_factory.stdout.getvalue())
1441
61
67
1442
62
    def test_check_branch_report_results(self):
68
    def test_check_branch_report_results(self):
1443
63
        """Checking a branch produces results which can be printed"""
69
        """Checking a branch produces results which can be printed"""
1444
64
        branch = self.make_branch('.')
70
        branch = self.make_branch('.')
1446
65
        result = branch.check()
71
        branch.lock_read()
1447
72
        self.addCleanup(branch.unlock)
1448
73
        result = branch.check(self.make_refs(branch))
1449
66
        # reports results through logging
74
        # reports results through logging
1450
67
        result.report_results(verbose=True)
75
        result.report_results(verbose=True)
1451
68
        result.report_results(verbose=False)
76
        result.report_results(verbose=False)
1452
69
77
1460
70
    def test_check_detects_ghosts_in_mainline(self):
78
    def test__get_check_refs(self):
1461
71
        tree = self.make_branch_and_tree('test')
79
        tree = self.make_branch_and_tree('.')
1462
72
        tree.set_parent_ids(['thisisaghost'], allow_leftmost_as_ghost=True)
80
        revid = tree.commit('foo')
1463
73
        r1 = tree.commit('one')
81
        self.assertEqual(
1464
74
        r2 = tree.commit('two')
82
            set([('revision-existence', revid), ('lefthand-distance', revid)]),
1465
75
        result = tree.branch.check()
83
            set(tree.branch._get_check_refs()))
1459
76
        self.assertEquals(True, result.ghosts_in_mainline)
1466
77
84
1467
85
    def make_refs(self, branch):
1468
86
        needed_refs = branch._get_check_refs()
1469
87
        refs = {}
1470
88
        distances = set()
1471
89
        existences = set()
1472
90
        for ref in needed_refs:
1473
91
            kind, value = ref
1474
92
            if kind == 'lefthand-distance':
1475
93
                distances.add(value)
1476
94
            elif kind == 'revision-existence':
1477
95
                existences.add(value)
1478
96
            else:
1479
97
                raise AssertionError(
1480
98
                    'unknown ref kind for ref %s' % ref)
1481
99
        node_distances = branch.repository.get_graph().find_lefthand_distances(
1482
100
            distances)
1483
101
        for key, distance in node_distances.iteritems():
1484
102
            refs[('lefthand-distance', key)] = distance
1485
103
            if key in existences and distance > 0:
1486
104
                refs[('revision-existence', key)] = True
1487
105
                existences.remove(key)
1488
106
        parent_map = branch.repository.get_graph().get_parent_map(existences)
1489
107
        for key in parent_map:
1490
108
            refs[('revision-existence', key)] = True
1491
109
            existences.remove(key)
1492
110
        for key in existences:
1493
111
            refs[('revision-existence', key)] = False
1494
112
        return refs
1495
78
113
1496
=== modified file 'bzrlib/tests/per_repository/test_check.py'
1497
--- bzrlib/tests/per_repository/test_check.py	2009-04-09 20:23:07 +0000
1498
+++ bzrlib/tests/per_repository/test_check.py	2009-05-12 05:34:15 +0000
1499
@@ -36,18 +36,14 @@
1500
36
        tree = self.make_branch_and_tree('.')
36
        tree = self.make_branch_and_tree('.')
1501
37
        self.build_tree(['foo'])
37
        self.build_tree(['foo'])
1502
38
        tree.smart_add(['.'])
38
        tree.smart_add(['.'])
1504
39
        tree.commit('1')
39
        revid1 = tree.commit('1')
1505
40
        self.build_tree(['bar'])
40
        self.build_tree(['bar'])
1506
41
        tree.smart_add(['.'])
41
        tree.smart_add(['.'])
1511
42
        tree.commit('2')
42
        revid2 = tree.commit('2')
1512
43
        # XXX: check requires a non-empty revision IDs list, but it ignores the
43
        check_object = tree.branch.repository.check([revid1, revid2])
1509
44
        # contents of it!
1510
45
        check_object = tree.branch.repository.check(['ignored'])
1513
46
        check_object.report_results(verbose=True)
44
        check_object.report_results(verbose=True)
1514
47
        log = self._get_log(keep_log_file=True)
45
        log = self._get_log(keep_log_file=True)
1518
48
        self.assertContainsRe(
46
        self.assertContainsRe(log, "0 unreferenced text versions")
1516
49
            log,
1517
50
            "0 unreferenced text versions")
1519
51
47
1520
52
48
1521
53
class TestFindInconsistentRevisionParents(TestCaseWithBrokenRevisionIndex):
49
class TestFindInconsistentRevisionParents(TestCaseWithBrokenRevisionIndex):
1522
@@ -100,3 +96,32 @@
1523
100
            "revision-id has wrong parents in index: "
96
            "revision-id has wrong parents in index: "
1524
101
            r"\('incorrect-parent',\) should be \(\)")
97
            r"\('incorrect-parent',\) should be \(\)")
1525
102
98
1526
99
1527
100
class TestCallbacks(TestCaseWithRepository):
1528
101
1529
102
    def test_callback_tree_and_branch(self):
1530
103
        # use a real tree to get actual refs that will work
1531
104
        tree = self.make_branch_and_tree('foo')
1532
105
        revid = tree.commit('foo')
1533
106
        tree.lock_read()
1534
107
        self.addCleanup(tree.unlock)
1535
108
        needed_refs = {}
1536
109
        for ref in tree._get_check_refs():
1537
110
            needed_refs.setdefault(ref, []).append(tree)
1538
111
        for ref in tree.branch._get_check_refs():
1539
112
            needed_refs.setdefault(ref, []).append(tree.branch)
1540
113
        self.tree_check = tree._check
1541
114
        self.branch_check = tree.branch.check
1542
115
        tree._check = self.tree_callback
1543
116
        tree.branch.check = self.branch_callback
1544
117
        self.callbacks = []
1545
118
        tree.branch.repository.check([revid], callback_refs=needed_refs)
1546
119
        self.assertNotEqual([], self.callbacks)
1547
120
1548
121
    def tree_callback(self, refs):
1549
122
        self.callbacks.append(('tree', refs))
1550
123
        return self.tree_check(refs)
1551
124
1552
125
    def branch_callback(self, refs):
1553
126
        self.callbacks.append(('branch', refs))
1554
127
        return self.branch_check(refs)
1555
103
128
1556
=== modified file 'bzrlib/tests/test_graph.py'
1557
--- bzrlib/tests/test_graph.py	2009-06-10 03:56:49 +0000
1558
+++ bzrlib/tests/test_graph.py	2009-06-16 05:29:42 +0000
1559
@@ -526,6 +526,19 @@
1560
526
        graph = self.make_graph(history_shortcut)
526
        graph = self.make_graph(history_shortcut)
1561
527
        self.assertEqual(set(['rev2b']), graph.find_lca('rev3a', 'rev3b'))
527
        self.assertEqual(set(['rev2b']), graph.find_lca('rev3a', 'rev3b'))
1562
528
528
1563
529
    def test_lefthand_distance_smoke(self):
1564
530
        """A simple does it work test for graph.lefthand_distance(keys)."""
1565
531
        graph = self.make_graph(history_shortcut)
1566
532
        distance_graph = graph.find_lefthand_distances(['rev3b', 'rev2a'])
1567
533
        self.assertEqual({'rev2a': 2, 'rev3b': 3}, distance_graph)
1568
534
1569
535
    def test_lefthand_distance_ghosts(self):
1570
536
        """A simple does it work test for graph.lefthand_distance(keys)."""
1571
537
        nodes = {'nonghost':[NULL_REVISION], 'toghost':['ghost']}
1572
538
        graph = self.make_graph(nodes)
1573
539
        distance_graph = graph.find_lefthand_distances(['nonghost', 'toghost'])
1574
540
        self.assertEqual({'nonghost': 1, 'toghost': -1}, distance_graph)
1575
541
1576
529
    def test_recursive_unique_lca(self):
542
    def test_recursive_unique_lca(self):
1577
530
        """Test finding a unique least common ancestor.
543
        """Test finding a unique least common ancestor.
1578
531
544
1579
532
545
1580
=== modified file 'bzrlib/tests/test_versionedfile.py'
1581
--- bzrlib/tests/test_versionedfile.py	2009-05-01 18:09:24 +0000
1582
+++ bzrlib/tests/test_versionedfile.py	2009-06-15 00:24:04 +0000
1583
@@ -30,6 +30,7 @@
1584
30
    knit as _mod_knit,
30
    knit as _mod_knit,
1585
31
    osutils,
31
    osutils,
1586
32
    progress,
32
    progress,
1587
33
    ui,
1588
33
    )
34
    )
1589
34
from bzrlib.errors import (
35
from bzrlib.errors import (
1590
35
                           RevisionNotPresent,
36
                           RevisionNotPresent,
1591
@@ -1510,6 +1511,28 @@
1592
1510
        self.assertRaises(RevisionNotPresent,
1511
        self.assertRaises(RevisionNotPresent,
1593
1511
            files.annotate, prefix + ('missing-key',))
1512
            files.annotate, prefix + ('missing-key',))
1594
1512
1513
1595
1514
    def test_check_no_parameters(self):
1596
1515
        files = self.get_versionedfiles()
1597
1516
1598
1517
    def test_check_progressbar_parameter(self):
1599
1518
        """A progress bar can be supplied because check can be a generator."""
1600
1519
        pb = ui.ui_factory.nested_progress_bar()
1601
1520
        self.addCleanup(pb.finished)
1602
1521
        files = self.get_versionedfiles()
1603
1522
        files.check(progress_bar=pb)
1604
1523
1605
1524
    def test_check_with_keys_becomes_generator(self):
1606
1525
        files = self.get_versionedfiles()
1607
1526
        self.get_diamond_files(files)
1608
1527
        keys = files.keys()
1609
1528
        entries = files.check(keys=keys)
1610
1529
        seen = set()
1611
1530
        # Texts output should be fulltexts.
1612
1531
        self.capture_stream(files, entries, seen.add,
1613
1532
            files.get_parent_map(keys), require_fulltext=True)
1614
1533
        # All texts should be output.
1615
1534
        self.assertEqual(set(keys), seen)
1616
1535
1617
1513
    def test_construct(self):
1536
    def test_construct(self):
1618
1514
        """Each parameterised test can be constructed on a transport."""
1537
        """Each parameterised test can be constructed on a transport."""
1619
1515
        files = self.get_versionedfiles()
1538
        files = self.get_versionedfiles()
1620
@@ -1669,7 +1692,8 @@
1621
1669
             'knit-delta-closure', 'knit-delta-closure-ref',
1692
             'knit-delta-closure', 'knit-delta-closure-ref',
1622
1670
             'groupcompress-block', 'groupcompress-block-ref'])
1693
             'groupcompress-block', 'groupcompress-block-ref'])
1623
1671
1694
1625
1672
    def capture_stream(self, f, entries, on_seen, parents):
1695
    def capture_stream(self, f, entries, on_seen, parents,
1626
1696
        require_fulltext=False):
1627
1673
        """Capture a stream for testing."""
1697
        """Capture a stream for testing."""
1628
1674
        for factory in entries:
1698
        for factory in entries:
1629
1675
            on_seen(factory.key)
1699
            on_seen(factory.key)
1630
@@ -1680,6 +1704,8 @@
1631
1680
            self.assertEqual(parents[factory.key], factory.parents)
1704
            self.assertEqual(parents[factory.key], factory.parents)
1632
1681
            self.assertIsInstance(factory.get_bytes_as(factory.storage_kind),
1705
            self.assertIsInstance(factory.get_bytes_as(factory.storage_kind),
1633
1682
                str)
1706
                str)
1634
1707
            if require_fulltext:
1635
1708
                factory.get_bytes_as('fulltext')
1636
1683
1709
1637
1684
    def test_get_record_stream_interface(self):
1710
    def test_get_record_stream_interface(self):
1638
1685
        """each item in a stream has to provide a regular interface."""
1711
        """each item in a stream has to provide a regular interface."""
1639
@@ -2547,8 +2573,8 @@
1640
2547
        self.assertRaises(NotImplementedError,
2573
        self.assertRaises(NotImplementedError,
1641
2548
                self.texts.add_mpdiffs, [])
2574
                self.texts.add_mpdiffs, [])
1642
2549
2575
1645
2550
    def test_check(self):
2576
    def test_check_noerrors(self):
1646
2551
        self.assertTrue(self.texts.check())
2577
        self.texts.check()
1647
2552
2578
1648
2553
    def test_insert_record_stream(self):
2579
    def test_insert_record_stream(self):
1649
2554
        self.assertRaises(NotImplementedError, self.texts.insert_record_stream,
2580
        self.assertRaises(NotImplementedError, self.texts.insert_record_stream,
1650
2555
2581
1651
=== modified file 'bzrlib/tests/workingtree_implementations/__init__.py'
1652
--- bzrlib/tests/workingtree_implementations/__init__.py	2009-06-10 03:56:49 +0000
1653
+++ bzrlib/tests/workingtree_implementations/__init__.py	2009-06-16 05:29:42 +0000
1654
@@ -60,45 +60,49 @@
1655
60
60
1656
61
61
1657
62
def load_tests(standard_tests, module, loader):
62
def load_tests(standard_tests, module, loader):
1658
63
    test_names = [
1659
64
        'add_reference',
1660
65
        'add',
1661
66
        'basis_inventory',
1662
67
        'basis_tree',
1663
68
        'break_lock',
1664
69
        'changes_from',
1665
70
        'check',
1666
71
        'content_filters',
1667
72
        'commit',
1668
73
        'eol_conversion',
1669
74
        'executable',
1670
75
        'flush',
1671
76
        'get_file_mtime',
1672
77
        'get_parent_ids',
1673
78
        'inv',
1674
79
        'is_control_filename',
1675
80
        'is_ignored',
1676
81
        'locking',
1677
82
        'merge_from_branch',
1678
83
        'mkdir',
1679
84
        'move',
1680
85
        'nested_specifics',
1681
86
        'parents',
1682
87
        'paths2ids',
1683
88
        'pull',
1684
89
        'put_file',
1685
90
        'readonly',
1686
91
        'read_working_inventory',
1687
92
        'remove',
1688
93
        'rename_one',
1689
94
        'revision_tree',
1690
95
        'set_root_id',
1691
96
        'smart_add',
1692
97
        'uncommit',
1693
98
        'unversion',
1694
99
        'views',
1695
100
        'walkdirs',
1696
101
        'workingtree',
1697
102
        ]
1698
63
    test_workingtree_implementations = [
103
    test_workingtree_implementations = [
1737
64
        'bzrlib.tests.workingtree_implementations.test_add_reference',
104
        'bzrlib.tests.workingtree_implementations.test_' + name for
1738
65
        'bzrlib.tests.workingtree_implementations.test_add',
105
        name in test_names]
1701
66
        'bzrlib.tests.workingtree_implementations.test_basis_inventory',
1702
67
        'bzrlib.tests.workingtree_implementations.test_basis_tree',
1703
68
        'bzrlib.tests.workingtree_implementations.test_break_lock',
1704
69
        'bzrlib.tests.workingtree_implementations.test_changes_from',
1705
70
        'bzrlib.tests.workingtree_implementations.test_content_filters',
1706
71
        'bzrlib.tests.workingtree_implementations.test_commit',
1707
72
        'bzrlib.tests.workingtree_implementations.test_eol_conversion',
1708
73
        'bzrlib.tests.workingtree_implementations.test_executable',
1709
74
        'bzrlib.tests.workingtree_implementations.test_flush',
1710
75
        'bzrlib.tests.workingtree_implementations.test_get_file_mtime',
1711
76
        'bzrlib.tests.workingtree_implementations.test_get_parent_ids',
1712
77
        'bzrlib.tests.workingtree_implementations.test_inv',
1713
78
        'bzrlib.tests.workingtree_implementations.test_is_control_filename',
1714
79
        'bzrlib.tests.workingtree_implementations.test_is_ignored',
1715
80
        'bzrlib.tests.workingtree_implementations.test_locking',
1716
81
        'bzrlib.tests.workingtree_implementations.test_merge_from_branch',
1717
82
        'bzrlib.tests.workingtree_implementations.test_mkdir',
1718
83
        'bzrlib.tests.workingtree_implementations.test_move',
1719
84
        'bzrlib.tests.workingtree_implementations.test_nested_specifics',
1720
85
        'bzrlib.tests.workingtree_implementations.test_parents',
1721
86
        'bzrlib.tests.workingtree_implementations.test_paths2ids',
1722
87
        'bzrlib.tests.workingtree_implementations.test_pull',
1723
88
        'bzrlib.tests.workingtree_implementations.test_put_file',
1724
89
        'bzrlib.tests.workingtree_implementations.test_readonly',
1725
90
        'bzrlib.tests.workingtree_implementations.test_read_working_inventory',
1726
91
        'bzrlib.tests.workingtree_implementations.test_remove',
1727
92
        'bzrlib.tests.workingtree_implementations.test_rename_one',
1728
93
        'bzrlib.tests.workingtree_implementations.test_revision_tree',
1729
94
        'bzrlib.tests.workingtree_implementations.test_set_root_id',
1730
95
        'bzrlib.tests.workingtree_implementations.test_smart_add',
1731
96
        'bzrlib.tests.workingtree_implementations.test_uncommit',
1732
97
        'bzrlib.tests.workingtree_implementations.test_unversion',
1733
98
        'bzrlib.tests.workingtree_implementations.test_views',
1734
99
        'bzrlib.tests.workingtree_implementations.test_walkdirs',
1735
100
        'bzrlib.tests.workingtree_implementations.test_workingtree',
1736
101
        ]
1739
102
106
1740
103
    scenarios = make_scenarios(
107
    scenarios = make_scenarios(
1741
104
        tests.default_transport,
108
        tests.default_transport,
1742
105
109
1743
=== added file 'bzrlib/tests/workingtree_implementations/test_check.py'
1744
--- bzrlib/tests/workingtree_implementations/test_check.py	1970-01-01 00:00:00 +0000
1745
+++ bzrlib/tests/workingtree_implementations/test_check.py	2009-05-08 02:06:36 +0000
1746
@@ -0,0 +1,55 @@
1747
1
# Copyright (C) 2009 Canonical Ltd
1748
2
#
1749
3
# This program is free software; you can redistribute it and/or modify
1750
4
# it under the terms of the GNU General Public License as published by
1751
5
# the Free Software Foundation; either version 2 of the License, or
1752
6
# (at your option) any later version.
1753
7
#
1754
8
# This program is distributed in the hope that it will be useful,
1755
9
# but WITHOUT ANY WARRANTY; without even the implied warranty of
1756
10
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
1757
11
# GNU General Public License for more details.
1758
12
#
1759
13
# You should have received a copy of the GNU General Public License
1760
14
# along with this program; if not, write to the Free Software
1761
15
# Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA
1762
16
1763
17
"""Tests for checking of trees."""
1764
18
1765
19
from bzrlib import (
1766
20
    tests,
1767
21
    )
1768
22
from bzrlib.tests.workingtree_implementations import TestCaseWithWorkingTree
1769
23
1770
24
1771
25
class TestCheck(TestCaseWithWorkingTree):
1772
26
1773
27
    def test__get_check_refs_new(self):
1774
28
        tree = self.make_branch_and_tree('tree')
1775
29
        self.assertEqual(set([('trees', 'null:')]),
1776
30
            set(tree._get_check_refs()))
1777
31
1778
32
    def test__get_check_refs_basis(self):
1779
33
        # with a basis, all current bzr trees cache it and so need the
1780
34
        # inventory to cross-check.
1781
35
        tree = self.make_branch_and_tree('tree')
1782
36
        revid = tree.commit('first post')
1783
37
        self.assertEqual(set([('trees', revid)]),
1784
38
            set(tree._get_check_refs()))
1785
39
1786
40
    def test__check_with_refs(self):
1787
41
        # _check can be called with a dict of the things required.
1788
42
        tree = self.make_branch_and_tree('tree')
1789
43
        tree.lock_write()
1790
44
        self.addCleanup(tree.unlock)
1791
45
        revid = tree.commit('first post')
1792
46
        needed_refs = tree._get_check_refs()
1793
47
        repo = tree.branch.repository
1794
48
        for ref in needed_refs:
1795
49
            kind, revid = ref
1796
50
            refs = {}
1797
51
            if kind == 'trees':
1798
52
                refs[ref] = repo.revision_tree(revid)
1799
53
            else:
1800
54
                self.fail('unknown ref kind')
1801
55
        tree._check(refs)
1802
0
56
1803
=== modified file 'bzrlib/ui/text.py'
1804
--- bzrlib/ui/text.py	2009-04-10 19:37:20 +0000
1805
+++ bzrlib/ui/text.py	2009-05-12 06:30:56 +0000
1806
@@ -26,6 +26,7 @@
1807
26
from bzrlib.lazy_import import lazy_import
26
from bzrlib.lazy_import import lazy_import
1808
27
lazy_import(globals(), """
27
lazy_import(globals(), """
1809
28
from bzrlib import (
28
from bzrlib import (
1810
29
    debug,
1811
29
    progress,
30
    progress,
1812
30
    osutils,
31
    osutils,
1813
31
    symbol_versioning,
32
    symbol_versioning,
1814
@@ -128,6 +129,7 @@
1815
128
        self._last_task = None
129
        self._last_task = None
1816
129
        self._total_byte_count = 0
130
        self._total_byte_count = 0
1817
130
        self._bytes_since_update = 0
131
        self._bytes_since_update = 0
1818
132
        self._fraction = 0
1819
131
133
1820
132
    def _show_line(self, s):
134
    def _show_line(self, s):
1821
133
        n = self._width - 1
135
        n = self._width - 1
1822
@@ -151,9 +153,14 @@
1823
151
            cols = 20
153
            cols = 20
1824
152
            if self._last_task is None:
154
            if self._last_task is None:
1825
153
                completion_fraction = 0
155
                completion_fraction = 0
1826
156
                self._fraction = 0
1827
154
            else:
157
            else:
1828
155
                completion_fraction = \
158
                completion_fraction = \
1829
156
                    self._last_task._overall_completion_fraction() or 0
159
                    self._last_task._overall_completion_fraction() or 0
1830
160
            if (completion_fraction < self._fraction and 'progress' in
1831
161
                debug.debug_flags):
1832
162
                import pdb;pdb.set_trace()
1833
163
            self._fraction = completion_fraction
1834
157
            markers = int(round(float(cols) * completion_fraction)) - 1
164
            markers = int(round(float(cols) * completion_fraction)) - 1
1835
158
            bar_str = '[' + ('#' * markers + spin_str).ljust(cols) + '] '
165
            bar_str = '[' + ('#' * markers + spin_str).ljust(cols) + '] '
1836
159
            return bar_str
166
            return bar_str
1837
160
167
1838
=== modified file 'bzrlib/versionedfile.py'
1839
--- bzrlib/versionedfile.py	2009-06-10 03:56:49 +0000
1840
+++ bzrlib/versionedfile.py	2009-06-16 05:29:42 +0000
1841
@@ -876,7 +876,16 @@
1842
876
        raise NotImplementedError(self.annotate)
876
        raise NotImplementedError(self.annotate)
1843
877
877
1844
878
    def check(self, progress_bar=None):
878
    def check(self, progress_bar=None):
1846
879
        """Check this object for integrity."""
879
        """Check this object for integrity.
1847
880
        
1848
881
        :param progress_bar: A progress bar to output as the check progresses.
1849
882
        :param keys: Specific keys within the VersionedFiles to check. When
1850
883
            this parameter is not None, check() becomes a generator as per
1851
884
            get_record_stream. The difference to get_record_stream is that
1852
885
            more or deeper checks will be performed.
1853
886
        :return: None, or if keys was supplied a generator as per
1854
887
            get_record_stream.
1855
888
        """
1856
880
        raise NotImplementedError(self.check)
889
        raise NotImplementedError(self.check)
1857
881
890
1858
882
    @staticmethod
891
    @staticmethod
1859
@@ -1092,10 +1101,15 @@
1860
1092
            result.append((prefix + (origin,), line))
1101
            result.append((prefix + (origin,), line))
1861
1093
        return result
1102
        return result
1862
1094
1103
1864
1095
    def check(self, progress_bar=None):
1104
    def check(self, progress_bar=None, keys=None):
1865
1096
        """See VersionedFiles.check()."""
1105
        """See VersionedFiles.check()."""
1866
1106
        # XXX: This is over-enthusiastic but as we only thunk for Weaves today
1867
1107
        # this is tolerable. Ideally we'd pass keys down to check() and 
1868
1108
        # have the older VersiondFile interface updated too.
1869
1097
        for prefix, vf in self._iter_all_components():
1109
        for prefix, vf in self._iter_all_components():
1870
1098
            vf.check()
1110
            vf.check()
1871
1111
        if keys is not None:
1872
1112
            return self.get_record_stream(keys, 'unordered', True)
1873
1099
1113
1874
1100
    def get_parent_map(self, keys):
1114
    def get_parent_map(self, keys):
1875
1101
        """Get a map of the parents of keys.
1115
        """Get a map of the parents of keys.
1876
1102
1116
1877
=== modified file 'bzrlib/workingtree.py'
1878
--- bzrlib/workingtree.py	2009-06-15 15:47:45 +0000
1879
+++ bzrlib/workingtree.py	2009-06-16 05:29:42 +0000
1880
@@ -290,6 +290,16 @@
1881
290
        self._control_files.break_lock()
290
        self._control_files.break_lock()
1882
291
        self.branch.break_lock()
291
        self.branch.break_lock()
1883
292
292
1884
293
    def _get_check_refs(self):
1885
294
        """Return the references needed to perform a check of this tree.
1886
295
        
1887
296
        The default implementation returns no refs, and is only suitable for
1888
297
        trees that have no local caching and can commit on ghosts at any time.
1889
298
1890
299
        :seealso: bzrlib.check for details about check_refs.
1891
300
        """
1892
301
        return []
1893
302
1894
293
    def requires_rich_root(self):
303
    def requires_rich_root(self):
1895
294
        return self._format.requires_rich_root
304
        return self._format.requires_rich_root
1896
295
305
1897
@@ -2515,12 +2525,17 @@
1898
2515
        return un_resolved, resolved
2525
        return un_resolved, resolved
1899
2516
2526
1900
2517
    @needs_read_lock
2527
    @needs_read_lock
1902
2518
    def _check(self):
2528
    def _check(self, references):
1903
2529
        """Check the tree for consistency.
1904
2530
1905
2531
        :param references: A dict with keys matching the items returned by
1906
2532
            self._get_check_refs(), and values from looking those keys up in
1907
2533
            the repository.
1908
2534
        """
1909
2519
        tree_basis = self.basis_tree()
2535
        tree_basis = self.basis_tree()
1910
2520
        tree_basis.lock_read()
2536
        tree_basis.lock_read()
1911
2521
        try:
2537
        try:
1914
2522
            repo_basis = self.branch.repository.revision_tree(
2538
            repo_basis = references[('trees', self.last_revision())]
1913
2523
                self.last_revision())
1915
2524
            if len(list(repo_basis.iter_changes(tree_basis))) > 0:
2539
            if len(list(repo_basis.iter_changes(tree_basis))) > 0:
1916
2525
                raise errors.BzrCheckError(
2540
                raise errors.BzrCheckError(
1917
2526
                    "Mismatched basis inventory content.")
2541
                    "Mismatched basis inventory content.")
1918
@@ -2572,6 +2587,10 @@
1919
2572
        if self._inventory is None:
2587
        if self._inventory is None:
1920
2573
            self.read_working_inventory()
2588
            self.read_working_inventory()
1921
2574
2589
1922
2590
    def _get_check_refs(self):
1923
2591
        """Return the references needed to perform a check of this tree."""
1924
2592
        return [('trees', self.last_revision())]
1925
2593
1926
2575
    def lock_tree_write(self):
2594
    def lock_tree_write(self):
1927
2576
        """See WorkingTree.lock_tree_write().
2595
        """See WorkingTree.lock_tree_write().
1928
2577
2596
1929
@@ -2634,6 +2653,10 @@
1930
2634
                mode=self.bzrdir._get_file_mode())
2653
                mode=self.bzrdir._get_file_mode())
1931
2635
            return True
2654
            return True
1932
2636
2655
1933
2656
    def _get_check_refs(self):
1934
2657
        """Return the references needed to perform a check of this tree."""
1935
2658
        return [('trees', self.last_revision())]
1936
2659
1937
2637
    @needs_tree_write_lock
2660
    @needs_tree_write_lock
1938
2638
    def set_conflicts(self, conflicts):
2661
    def set_conflicts(self, conflicts):
1939
2639
        self._put_rio('conflicts', conflicts.to_stanzas(),
2662
        self._put_rio('conflicts', conflicts.to_stanzas(),
1940
2640
2663
1941
=== added file 'doc/developers/check.txt'
1942
--- doc/developers/check.txt	1970-01-01 00:00:00 +0000
1943
+++ doc/developers/check.txt	2009-05-12 03:50:39 +0000
1944
@@ -0,0 +1,63 @@
1945
1
Check Notes
1946
2
===========
1947
3
1948
4
.. contents:: :local:
1949
5
1950
6
Overview
1951
7
--------
1952
8
1953
9
Check has multiple responsibilities:
1954
10
1955
11
* Ensure that the data as recorded on disk is accessible intact and unaltered.
1956
12
* Ensure that a branch/repository/tree/whatever is ready for upgrade.
1957
13
* Look for and report on recorded-data issues where previous bzr's, or changing
1958
14
  situations have lead so some form of inconsistency.
1959
15
* Report sufficient information for a user to either fix the issue themselves
1960
16
  or report a bug that will hopefully be sufficiently detailed we can fix based
1961
17
  on the initial report.
1962
18
* Not scare users when run if everything is okey-dokey.
1963
19
1964
20
Ideally one check invocation can do all these things.
1965
21
1966
22
Repository
1967
23
----------
1968
24
1969
25
Things that can go wrong:
1970
26
* Bit errors or alterations may occur in raw data.
1971
27
* Data that is referenced may be missing
1972
28
* There could be a lot of garbage in upload etc.
1973
29
* File graphs may be inconsistent with inventories and parents.
1974
30
* The revision graph cache can be inconsistent with the revision data.
1975
31
1976
32
Branch
1977
33
------
1978
34
1979
35
Things that can go wrong:
1980
36
* Tag or tip revision ids may be missing from the repo.
1981
37
* The revno tip cache may be wrong.
1982
38
* Various urls could be problematic (not inaccessible, just invalid)
1983
39
* Stacked-on branch could be inaccessible.
1984
40
1985
41
Tree
1986
42
----
1987
43
1988
44
Things that can go wrong:
1989
45
* Bit errors in dirstate.
1990
46
* Corrupt or invalid shelves.
1991
47
* Corrupt dirstates written to disk.
1992
48
* Cached inventories might not match repository.
1993
49
1994
50
Duplicate work
1995
51
--------------
1996
52
1997
53
If we check every branch in a repo separately we will encounter duplicate
1998
54
effort in assessing things like missing tags/tips, revno cache etc.
1999
55
2000
56
Outline of approach
2001
57
-------------------
2002
58
2003
59
To check a repository, we scan for branches, open their trees and generate
2004
60
summary data. We then collect all the summary data in as compact a form as
2005
61
possible and do a detailed check on the repository, calling back out to branch
2006
62
and trees as we encounter the actual data that that tree/branch requires to
2007
63
perform its check.
2008
0
64
Status:	Merged
Merged at revision:	not available
Proposed branch:	lp:~lifeless/bzr/check-1
Merge into:	lp:~bzr/bzr/trunk-old
Diff against target:	None lines
To merge this branch:	bzr merge lp:~lifeless/bzr/check-1
Related bugs:	Link a bug report
Reviewer	Date Requested	Status
Vincent Ladeuil	2009-07-07	Approve on 2009-07-07
bzr-core	2009-06-16	Pending
Review via email: mp+7489@code.launchpad.net