Launchpad itself

Merge lp:~gary/launchpad/pythonpath into lp:launchpad/db-devel

  1. pythonpath
  2. Merge into db-devel
Proposed by Gary Poster
Status: Merged
Merged at revision: not available
Proposed branch: lp:~gary/launchpad/pythonpath
Merge into: lp:launchpad/db-devel
Diff against target: 99 lines (+51/-23)
1 file modified
buildout-templates/_pythonpath.py.in (+51/-23)
To merge this branch: bzr merge lp:~gary/launchpad/pythonpath
Reviewer Review Type Date Requested Status
Francis J. Lacoste (community) release-critical Approve
Barry Warsaw (community) code Approve
Review via email: mp+14490@code.launchpad.net
To post a comment you must log in.
Revision history for this message
Gary Poster (gary) wrote :

Fixes bug 475550.

Diff and example usage can be found here: http://paste.ubuntu.com/310788/

I hope that the bug report, the extensive comment, and the example usage can provide the necessary context. If not, of course, please ask!

Thank you

Gary

Revision history for this message
Barry Warsaw (barry) wrote :

Thanks for working on this hairy problem! We talked about it enough in irc that this approach seems reasonable to me.

Python should really make our lives easier.

review: Approve (code)
Revision history for this message
Francis J. Lacoste (flacoste) :
review: Approve (release-critical)

Preview Diff

[H/L] Next/Prev Comment, [J/K] Next/Prev File, [N/P] Next/Prev Hunk
1=== modified file 'buildout-templates/_pythonpath.py.in'
2--- buildout-templates/_pythonpath.py.in 2009-10-19 19:33:01 +0000
3+++ buildout-templates/_pythonpath.py.in 2009-11-05 20:20:37 +0000
4@@ -6,6 +6,7 @@
5
6 __metaclass__ = type
7
8+import new
9 import os
10 import site
11 import sys
12@@ -19,22 +20,58 @@
13 site_dirs = [
14 ${indented-dir-paths}
15 ]
16-clean_modules = [
17- 'os',
18- '_pythonpath',
19- 'site',
20- 'sitecustomize',
21- 'sys',
22- ${clean-sys-modules}
23- ]
24
25 def set_path():
26- previously_imported = {}
27- # We get a copy of the keys rather than simply iterating because we will
28- # be mutating the dictionary.
29- for k in sys.modules.keys():
30- if k not in clean_modules:
31- previously_imported[k] = sys.modules.pop(k)
32+ # We need to remove any modules installed using the setuptools
33+ # namespace package approach that generates .pth files that mutate
34+ # sys.modules. They can mask the namespace modules we actually
35+ # want. We will try to recognize these sorts of namespace modules by
36+ # looking for modules that do not have a __file__, like the ones
37+ # generated by these .pth files. However, that's not good enough: some
38+ # C modules look like this. We then look for any names in the
39+ # module that are not special Python names (__*__) that are also not
40+ # in sys.modules. If any one of these exist, this is not a
41+ # namespace module. Otherwise, it is a namespace module. Moreover,
42+ # if there were any sys.modules names from the namespace module,
43+ # this _pythonpath module was imported too late: we should complain.
44+ #
45+ # If you have not seen the code in these .pth files, here's an
46+ # example. It is all in one line in the .pth files, because of the
47+ # .pth syntax for this feature.
48+ #
49+ # import sys,new,os;
50+ # p = os.path.join(sys._getframe(1).f_locals['sitedir'], *('lazr',));
51+ # ie = os.path.exists(os.path.join(p,'__init__.py'));
52+ # m = not ie and sys.modules.setdefault('lazr',new.module('lazr'));
53+ # mp = (m or []) and m.__dict__.setdefault('__path__',[]);
54+ # (p not in mp) and mp.append(p)
55+ #
56+ # Note that we will be mutating sys.modules, so we want to make sure to be
57+ # getting a copy, not an iterable.
58+ marker = object()
59+ for k, v in sys.modules.items():
60+ if k == '__main__' or not isinstance(v, new.module):
61+ # These are some special cases that we'll pass by.
62+ continue
63+ if getattr(v, '__file__', marker) is marker:
64+ # This is a .pth-generated namespace or C module.
65+ for name in dir(v):
66+ if not name.startswith('__') and not name.endswith('__'):
67+ full_name = '.'.join((k, name))
68+ if full_name in sys.modules:
69+ # This is a .pth-generated namespace module that
70+ # has had one of its sub-packages imported.
71+ raise RuntimeError(
72+ 'Found unexpected module %s. '
73+ 'Import _pythonpath earlier!' % (full_name,))
74+ else:
75+ # This is a C module or something like that. Nothing
76+ # to see here: move along.
77+ break
78+ else:
79+ # It is a .pth-generated namespace module. Remove it
80+ # so we can let our eggs make their own.
81+ del sys.modules[k]
82
83 # We keep the very first path because that is typically the directory
84 # of the file that imported us, which should continue to have precedence.
85@@ -53,15 +90,6 @@
86 for p in site_dirs:
87 site.addsitedir(p)
88
89- # We don't want to have dropped any packages that weren't already added
90- # back by what we just did. If we did, there's a good chance that the
91- # world will now be insane. Quit now, and let's fix it.
92- unexpected = set(previously_imported).difference(sys.modules)
93- if unexpected:
94- raise RuntimeError(
95- 'Found unexpected module(s): %s\n\nImport _pythonpath earlier!' %
96- (', '.join(sorted(unexpected)),))
97-
98 # Make subprocesses have the same environment.
99 os.environ['PYTHONPATH'] = os.pathsep.join(sys.path)
100

Subscribers

People subscribed via source and target branches

to status/vote changes: