lp:~jameinel/bzr/2.3-gcb-peak-mem

Branch merges

Propose for merging

No branches dependent on this one.

Merged into lp:bzr at revision 5451

Andrew Bennetts: Approve on 2010-09-24

Diff: 289 lines (+99/-42)

4 files modified

NEWS (+3/-0)
bzrlib/groupcompress.py (+69/-39)
bzrlib/tests/per_lock/test_lock.py (+1/-1)
bzrlib/tests/test_groupcompress.py (+26/-2)

api

Related source package recipes

No recipes using this branch. (?)

Create packaging recipe

Related snap packages

Related bugs

Bug #602614: Out of memory error in _ensure_content on auto repacking

High

Fix Released

Link a bug report

Related blueprints

5543. By John A Meinel on 2010-12-02

A TODO entry.

Overall this experiment hasn't been particularly beneficial. The final speed
is still slower than the existing code, and the primary knob that reduced
peak memory size is to change the stride for large content. Which can be
trivially added to the existing match code.

I like the code quite a bit, but I wish I had more to show for the amount
of effort put into it.

5542. By John A Meinel on 2010-12-02

sorting-on-repack doesn't seem to help a lot.
It adds a couple of seconds to 'build' time, because of more overhead and lots of
'old' search steps.
I'm a bit surprised at how many old steps are taken, maybe because the table is
now 'overly full'? which is why we are resizing in the first place.
Total time is a fairly significant loss (2m2s). Only a couple seconds due to
the time to resize, so probably somehow 'less optimal' distribution now...

5541. By John A Meinel on 2010-12-02

Revert to previous code.

5540. By John A Meinel on 2010-12-02

(broken, experiment) Lowering the load factor did improve the peek-skip rate

Changing initial hash to (rabin_val ^ (rabin_val >> 16)) is actually
quite a bit worse overall. Though it looks like it was 'unstable' and not giving
identical pack file output. >>24 seems to be better, but doesn't improve anything.
In both cases, it is clear the extra bits don't help peek skip rates, so it isn't
helpful.

5539. By John A Meinel on 2010-12-02

Moving the key functionality into inline functions drops us down to 1m44.4s

How do you value the cost of complexity of code, vs the performance you get from the code?
It is hard for me to determine 'good enough'. I at least wanted to experiment with it
to determine where the threshold might be.
(if it shaved 30s out of 100s, then it seems more worthwhile.)

Note the current best time with inlining is 1m45s, the current best time for
existing bzr code is 1m38s, which is still faster...

5538. By John A Meinel on 2010-12-01

peek() seems to shave about 5% (5s reduction, over ~100s total time).

I don't know that we need it as a separate 'inline' function, but it is definitely
useful to peek() before doing the full test.

With peek, we get about 217M peek skips, 166M no-actual-matches, out of 392M
matches that aren't good enough, and 420M total matches.

5537. By John A Meinel on 2010-12-01

More profiling results. It appears that a vast majority of rabin lookups fail.

Something like 90+% (384M vs 420M total lookups). So adding a 'peek()' function that does
the most-trivial amount of work I can find to determine if this is worth persuing further.
Eliminates lots of .next() and .start() calls. Might want to turn it into an inline
separate function, since it is pretty cheap. peek() seems to skip 217M of those calls.

5536. By John A Meinel on 2010-12-01

Some interesting results from using cython: profile=True,

Having 'next_with_dummy()' defined as 'except? NULL' was quite a bit of
overhead. My guess is that NULL is quite common, and PyErr_Occurred is
a little expensive. All the callers would handle it gracefully (.next()
knows to handle falling off the end, and the limiting loop will just
treat it as no more entries to fill in, which is valid.

Now within spitting distance of the existing code, lots faster and lower
memory against large file content, but need to think closely about some
tweaks for match rules.

5535. By John A Meinel on 2010-11-30

More instrumentation.

5534. By John A Meinel on 2010-11-23

Bits of TODO markup.

Initial results from pulling the 'RabinEntrySearch' allocation out of
_find_better_match seems to drop specific time from 120s down to 66s.
Didn't seem to affect wall-time, so not sure if that is accurate.