Merge lp:~cjwatson/meliae/py3-loader-source-bytes into lp:meliae

This is an interesting one. I think fundamentally we should just drop the _from_line decoder, since 'json' is now a core part of the Python stdlib.
The question remains whether we want to leverage the fact that Bytes might be more memory efficient than PyUnicode objects.

>>> import sys
>>> for i in [0, 1, 10, 50, 500]:
... x = b'1'*i
... y = '1'*i
... print(i, sys.getsizeof(x), sys.getsizeof(y))
...
   0 33 49
   1 34 50
  10 43 59
  50 83 99
500 533 549

It is a modest win at low sizes (59 vs 43 at 10 bytes is 37% more overhead).

I don't know how much it really matters, but I do remember playing a lot of tricks to make it easier to look at a large memory dump without then needing at least as much memory as you were using live. (its why there is a MemObjectCollection which is functionally a dict but typed, and why there are Proxy objects that live long enough to look like a fleshed out Python object instead of just a C struct.)

I hadn't considered the question of large dumps. In that case the difference in size between bytes and text isn't quite the point; the issue would be that decoding would add another copy of the data in memory. Indeed, while (simple)json.loads is a more accurate parser, it unpacks it into a much less memory-efficient representation along the way. This is only an issue if there are single objects in the dump that are very large (e.g. a large string), but of course that's quite possible.

So I guess if I just arranged for the _from_line decoder to cope with bytes to save the extra intermediate representation and dropped the line.decode call, that would be good enough? (json.loads accepts either bytes or text on Python 3, so there's no type-safety issue for _from_json.)

So I believe the file format is such that it *could* be just-one-big json.loads(), but you don't get any progress, etc when doing that. I believe since it is just [\n{data},\n{data},...] you can load each line separately with json, which is what I thought I was doing (trimming off the trailing ,).

I'm happy to continue doing so, as any given line isn't going to be a major overhead, and the memory savings are by putting it into a compacted data structure. (though that followed the Py2 Dict model of a table with holes in it vs the python 3 table-with-no-holes and an index with holes.)

I think the scanner already truncates long strings, so you only see the prefix, so we shouldn't have to worry about that too much.

OK, it took me a while to get my head around what you were driving at here, but I think I now understand. How's this? I've turned type_str and name (and sometimes value) back into bytes objects, and everything should now be more compact again even on Python 3.

I'm happy to land this as is. I'm also willing for you to push back and say "the memory saving probably isn't worth disrupting people using the library who end up seeing b'' strings where they just expect strings".

The fact that you play around casting type_str back to string makes me wonder.
I'll wait to merge until I hear back from you.

As discussed on IRC, I've made type_str be an interned str instead, which indeed does make things generally easier to deal with. I think keeping the less-shareable fields as bytes rather than str is reasonably justifiable in the context of a memory debugging tool though; it's still a little surprising, but seems tolerable.