Code review comment for lp:~jameinel/bzr-fastimport/mooloolaba
Revision history for this message
| John A Meinel (jameinel) wrote | # |
© 2004
Canonical Ltd.
•
Terms of use
•
Data privacy
•
Contact Launchpad Support
•
Blog
•
Careers
•
System status
•
35a32c5
(Get the code!)
This branch is in --2a format, but trunk isn't, so this request may fail.
Anyway, this is just a rollup of the stuff I worked on in Mooloolaba. Stuff like:
1) Memory reduction by referencing texts as strings rather than lines. This alone caused my peak qt import memory to go to 1.5GB from >2GB (I don't know how big, because I go OOM @2GB on my 32-bit machine).
2) Memory reduction by not caching all texts in memory. Instead, dump them into temp files. 'small' blob files are all collected into a single temp file, everything else gets a unique file that will be deleted when the last reference goes away. This got me down to between 500MB => 900MB during the import.
3) Switch the code that adds things to the repo to use CommitBuilder. It cheats a little bit here and there to get around deficiencies in the api. Namely, CommitBuilder wants to start_write_group and commit_write_group for every build, and we wanted to work around that.
I'm not sure what else is included, but that is the basics.