scanner doesn't flush/close

While it'd be good for meliae to explicitly close the file, I think this indicates that meliae is creating a reference cycle. From the look of it, you'd have the following cycle:

 1. dump_all_objects execution frame
 2. the all_objs list held as a local in that frame
 3. the dump_all_objects frame as an item in the all_objs list.

This would likely keep a reference to the open file object (since the frame holds a reference to it too) until the cyclic garbage collector kicks in at some later point in time.

Making dump_all_objects() clear the all_objs list before exiting is probably enough to get rid of the cycle.

This may be addressed by trunk revno 114. I did both things:

1) del all_objs[:] (why doesn't list have .clear()...)
2) if the file was opened, call 'outf.close()' otherwise call outf.flush() when the dump is done.

Using meliae 0.2.0 (r138, it seems), similar problems still exist. We have generated seven memory dumps for Launchpad so far (bug 531071). Five of them exhibit this problem. Here are the endings for each file, on the off-chance it is pertinent.

{"address": 46912590194560, "type": "tuple", "size": 80, "len": 3, "refs": [189248576, 46912590253760, 205904976]}
{"address": 205904976, "type": "TrustedZopePathExpr", "size": 64, "refs": [46912588292336, 23860992]}
{"address": 469125

{"address": 224506656, "type": "list", "size": 104, "len": 1, "refs": [46912626040576]}
{"address": 46912626040576, "type": "tuple", "size": 80, "len": 3, "refs": [21379520, 46912616607008, 46912553812640]}
{"address": 46912633222856, "type": "list

{"address": 44941312, "type": "dict", "size": 280, "len": 4, "refs": [26791752, 193153232, 26790912, 28441424, 13309536, 187535152, 26814192, 28443856]}
{"address": 187535152, "type": "str", "size": 44, "len": 4, "value": "view", "refs": []}
{"address": 193153232, "type": "tuple", "size": 64, "len": 1, "refs": [119181256

{"address": 46912626386144, "type": "unicode", "size": 68, "len": 5, "value": "maria", "refs": []}
{"address": 408007856, "type": "dict", "size": 280, "len": 1, "refs": [46913061766496, 7521504]}
{"address": 46913061766496, "type": "tuple", "size": 72, "len": 2, "r

{"address": 283744464, "type": "unicode", "size": 92, "len": 11, "value": "End of file", "refs": []}
{"address": 343873696, "type": "dict", "size": 280, "len": 3, "refs": [47683651817072, 7747160, 47683630069776, 56286928, 47683630069872, 283744464]}
{"address": 46912729200672, "type": "Gpgme

These files ranged in size from 84M to 334M. The two files that did not exhibit this problem were 371M and 381M, respectively.

As the scanner is given an open object, its the callers responsibility
to flush and close (going from memory of the code).

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Robert Collins wrote:
> As the scanner is given an open object, its the callers responsibility
> to flush and close (going from memory of the code).
>

Which function is being used? The recommended one is:
def dump_all_objects(outf):
...
    if isinstance(outf, basestring):
        opened = True
        outf = open(outf, 'wb')
    else:
        opened = False
    all_objs = gc.get_objects()
    dump_all_referenced(outf, all_objs, is_pending=True)
    del all_objs[:]
    if opened:
        outf.close()
    else:
        outf.flush()

Which I'll note both flushes if handed an object, and closes otherwise.

dump_gc_objects() also does the same.

dump_all_referenced() does not, but

 a) Probably could
 b) Isn't really meant as the top-level function to call.

(Note: I'm not 100% sure what ended up in the release, I generally
recommend running from trunk right now.)
John
=:->
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.9 (Cygwin)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/

iEYEARECAAYFAkvxwHYACgkQJdeBCYSNAAMjbQCgrE0KyPvlMsvg+NEh5oxsRcmS
M/kAoLPYwl4lkUaTO35XUetO61aKWcZC
=93jT
-----END PGP SIGNATURE-----

We are using dump_all_objects.

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Gary Poster wrote:
> We are using dump_all_objects.
>

Weird. I went ahead and added an fflush() call at a lower level, just in
case PyFile_AsFile(file_obj) doesn't sync properly with
file_obj.flush(). I don't know if you have a way to trigger the problem,
but maybe you could try with trunk@141 and see if the problem goes away?
(Basically, try with @140 and see if you can trigger it, and @141 and
see if it doesn't.)

John
=:->

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.9 (Cygwin)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/

iEYEARECAAYFAkv1QwkACgkQJdeBCYSNAAPdoACfZLBnCvgFLOWdUqiRVu0wP3e7
kRgAn0VE4qqn4px7lAKSQyT1SBjsibBK
=39XW
-----END PGP SIGNATURE-----

We were flushing the Python object, but not the FILE* object that we got. I would have thought they would be synchronized, but .... Anyway, added an fflush() call where appropriate.