Buildout

Merge lp:~gary/zc.buildout/python-support-2-bootstrap into lp:zc.buildout

  1. python-support-2-bootstrap
  2. Merge into trunk
Proposed by Gary Poster
Status: Needs review
Proposed branch: lp:~gary/zc.buildout/python-support-2-bootstrap
Merge into: lp:zc.buildout
Prerequisite: lp:~gary/zc.buildout/python-support-1-cleanup
Diff against target: 415 lines (+224/-60)
5 files modified
CHANGES.txt (+15/-3)
bootstrap/bootstrap.py (+110/-47)
dev.py (+1/-1)
src/zc/buildout/bootstrap.txt (+74/-6)
src/zc/buildout/buildout.py (+24/-3)
To merge this branch: bzr merge lp:~gary/zc.buildout/python-support-2-bootstrap
Reviewer Review Type Date Requested Status
Francis J. Lacoste (community) Approve
Review via email: mp+19538@code.launchpad.net
To post a comment you must log in.
Revision history for this message
Gary Poster (gary) wrote :

This branch tries to bring to zc.buildout the changes that Launchpad has used for its bootstrap script for awhile. It allows you to specify locations for files that it uses so that you do not need to go online to use bootstrap.

This is different from the one in the Launchpad tree because it is merged with zc.buildout's trunk. The newer version in trunk does not allow arbitrary buildout options to be passed, which means that we can use standard-library Python option parsing, which makes the code a bit simpler.

Reply
lp:~gary/zc.buildout/python-support-2-bootstrap updated
540. By Gary Poster

propagate merge from trunk from gary-1

Revision history for this message
Francis J. Lacoste (flacoste) wrote :

Looks good. No comments here.

review: Approve
Reply

Unmerged revisions

540. By Gary Poster

propagate merge from trunk from gary-1

539. By gary

flow the gary-1 changes through the connected branches

538. By gary

windows fix

537. By gary

merge of gary-2-bootstrap-changes into trunk, resolving lots of conflicts with the distribute changes.

536. By gary

look, more branches from Gary!

Preview Diff

[H/L] Next/Prev Comment, [J/K] Next/Prev File, [N/P] Next/Prev Hunk
1=== modified file 'CHANGES.txt'
2--- CHANGES.txt 2009-12-10 16:19:55 +0000
3+++ CHANGES.txt 2010-02-17 21:25:19 +0000
4@@ -1,9 +1,21 @@
5 Change History
6 **************
7
8-1.4.4 (?)
9-=========
10-
11+1.?.? (201?-??-??)
12+==================
13+
14+New Features:
15+
16+- Improve bootstrap.
17+
18+ * New options let you specify where to find ez_setup.py and where to find
19+ a download cache. These options can keep bootstrap from going over the
20+ network.
21+
22+ * Another new option lets you specify where to put generated eggs.
23+
24+ * The buildout script generated by bootstrap honors more of the settings
25+ in the designated configuration file (e.g., buildout.cfg).
26
27 1.4.3 (2009-12-10)
28 ==================
29
30=== modified file 'bootstrap/bootstrap.py'
31--- bootstrap/bootstrap.py 2010-02-12 02:40:18 +0000
32+++ bootstrap/bootstrap.py 2010-02-17 21:25:19 +0000
33@@ -20,21 +20,62 @@
34 $Id$
35 """
36
37-import os, shutil, sys, tempfile, urllib2
38+import os, shutil, sys, tempfile, textwrap, urllib, urllib2
39 from optparse import OptionParser
40
41-tmpeggs = tempfile.mkdtemp()
42-
43 is_jython = sys.platform.startswith('java')
44
45+setuptools_source = 'http://peak.telecommunity.com/dist/ez_setup.py'
46+distribute_source = 'http://python-distribute.org/distribute_setup.py'
47+
48 # parsing arguments
49-parser = OptionParser()
50+def normalize_to_url(option, opt_str, value, parser):
51+ if value:
52+ if '://' not in value: # It doesn't smell like a URL.
53+ value = 'file://%s' % (
54+ urllib.pathname2url(
55+ os.path.abspath(os.path.expanduser(value))),)
56+ if opt_str == '--download-base' and not value.endswith('/'):
57+ # Download base needs a trailing slash to make the world happy.
58+ value += '/'
59+ else:
60+ value = None
61+ name = opt_str[2:].replace('-', '_')
62+ setattr(parser.values, name, value)
63+
64+usage = '''\
65+[DESIRED PYTHON FOR BUILDOUT] bootstrap.py [options]
66+
67+Bootstraps a buildout-based project.
68+
69+Simply run this script in a directory containing a buildout.cfg, using the
70+Python that you want bin/buildout to use.
71+
72+Note that by using --setup-source and --download-base to point to
73+local resources, you can keep this script from going over the network.
74+'''
75+
76+parser = OptionParser(usage=usage)
77 parser.add_option("-v", "--version", dest="version",
78 help="use a specific zc.buildout version")
79 parser.add_option("-d", "--distribute",
80- action="store_true", dest="distribute", default=False,
81+ action="store_true", dest="use_distribute", default=False,
82 help="Use Distribute rather than Setuptools.")
83-
84+parser.add_option("--setup-source", action="callback", dest="setup_source",
85+ callback=normalize_to_url, nargs=1, type="string",
86+ help=("Specify a URL or file location for the setup file. "
87+ "If you use Setuptools, this will default to " +
88+ setuptools_source + "; if you use Distribute, this "
89+ "will default to " + distribute_source +"."))
90+parser.add_option("--download-base", action="callback", dest="download_base",
91+ callback=normalize_to_url, nargs=1, type="string",
92+ help=("Specify a URL or directory for downloading "
93+ "zc.buildout and either Setuptools or Distribute. "
94+ "Defaults to PyPI."))
95+parser.add_option("--eggs",
96+ help=("Specify a directory for storing eggs. Defaults to "
97+ "a temporary directory that is deleted when the "
98+ "bootstrap script completes."))
99 parser.add_option("-c", None, action="store", dest="config_file",
100 help=("Specify the path to the buildout configuration "
101 "file to be used."))
102@@ -45,35 +86,48 @@
103 if options.config_file is not None:
104 args += ['-c', options.config_file]
105
106-if options.version is not None:
107- VERSION = '==%s' % options.version
108+if options.eggs:
109+ eggs_dir = os.path.abspath(os.path.expanduser(options.eggs))
110 else:
111- VERSION = ''
112-
113-USE_DISTRIBUTE = options.distribute
114+ eggs_dir = tempfile.mkdtemp()
115+
116+if options.setup_source is None:
117+ if options.use_distribute:
118+ options.setup_source = distribute_source
119+ else:
120+ options.setup_source = setuptools_source
121+
122 args = args + ['bootstrap']
123
124 to_reload = False
125+
126 try:
127+ import setuptools # A flag. Sometimes pkg_resources is installed alone.
128 import pkg_resources
129 if not hasattr(pkg_resources, '_distribute'):
130 to_reload = True
131 raise ImportError
132 except ImportError:
133+ ez_code = urllib2.urlopen(
134+ options.setup_source).read().replace('\r\n', '\n')
135 ez = {}
136- if USE_DISTRIBUTE:
137- exec urllib2.urlopen('http://python-distribute.org/distribute_setup.py'
138- ).read() in ez
139- ez['use_setuptools'](to_dir=tmpeggs, download_delay=0, no_fake=True)
140- else:
141- exec urllib2.urlopen('http://peak.telecommunity.com/dist/ez_setup.py'
142- ).read() in ez
143- ez['use_setuptools'](to_dir=tmpeggs, download_delay=0)
144+ exec ez_code in ez
145+ setup_args = dict(to_dir=eggs_dir, download_delay=0)
146+ if options.download_base:
147+ setup_args['download_base'] = options.download_base
148+ if options.use_distribute:
149+ setup_args['no_fake'] = True
150+ ez['use_setuptools'](**setup_args)
151
152 if to_reload:
153 reload(pkg_resources)
154 else:
155 import pkg_resources
156+ # This does not (always?) update the default working set. We will
157+ # do it.
158+ for path in sys.path:
159+ if path not in pkg_resources.working_set.entries:
160+ pkg_resources.working_set.add_entry(path)
161
162 if sys.platform == 'win32':
163 def quote(c):
164@@ -85,37 +139,46 @@
165 def quote (c):
166 return c
167
168-cmd = 'from setuptools.command.easy_install import main; main()'
169-ws = pkg_resources.working_set
170-
171-if USE_DISTRIBUTE:
172- requirement = 'distribute'
173+cmd = [quote(sys.executable),
174+ '-c',
175+ quote('from setuptools.command.easy_install import main; main()'),
176+ '-mqNxd',
177+ quote(eggs_dir)]
178+
179+if options.download_base:
180+ cmd.extend(['-f', quote(options.download_base)])
181+
182+requirement = 'zc.buildout'
183+if options.version:
184+ requirement = '=='.join((requirement, options.version))
185+cmd.append(requirement)
186+
187+if options.use_distribute:
188+ setup_requirement = 'distribute'
189 else:
190- requirement = 'setuptools'
191+ setup_requirement = 'setuptools'
192+ws = pkg_resources.working_set
193+env = dict(
194+ os.environ,
195+ PYTHONPATH=ws.find(
196+ pkg_resources.Requirement.parse(setup_requirement)).location)
197
198 if is_jython:
199 import subprocess
200-
201- assert subprocess.Popen([sys.executable] + ['-c', quote(cmd), '-mqNxd',
202- quote(tmpeggs), 'zc.buildout' + VERSION],
203- env=dict(os.environ,
204- PYTHONPATH=
205- ws.find(pkg_resources.Requirement.parse(requirement)).location
206- ),
207- ).wait() == 0
208-
209-else:
210- assert os.spawnle(
211- os.P_WAIT, sys.executable, quote (sys.executable),
212- '-c', quote (cmd), '-mqNxd', quote (tmpeggs), 'zc.buildout' + VERSION,
213- dict(os.environ,
214- PYTHONPATH=
215- ws.find(pkg_resources.Requirement.parse(requirement)).location
216- ),
217- ) == 0
218-
219-ws.add_entry(tmpeggs)
220-ws.require('zc.buildout' + VERSION)
221+ exitcode = subprocess.Popen(cmd, env=env).wait()
222+else: # Windows prefers this, apparently; otherwise we would prefer subprocess
223+ exitcode = os.spawnle(*([os.P_WAIT, sys.executable] + cmd + [env]))
224+if exitcode != 0:
225+ sys.stdout.flush()
226+ sys.stderr.flush()
227+ print ("An error occured when trying to install zc.buildout. "
228+ "Look above this message for any errors that "
229+ "were output by easy_install.")
230+ sys.exit(exitcode)
231+
232+ws.add_entry(eggs_dir)
233+ws.require(requirement)
234 import zc.buildout.buildout
235 zc.buildout.buildout.main(args)
236-shutil.rmtree(tmpeggs)
237+if not options.eggs: # clean up temporary egg directory
238+ shutil.rmtree(eggs_dir)
239
240=== modified file 'dev.py'
241--- dev.py 2009-09-08 15:03:44 +0000
242+++ dev.py 2010-02-17 21:25:19 +0000
243@@ -13,7 +13,7 @@
244 ##############################################################################
245 """Bootstrap the buildout project itself.
246
247-This is different from a normal boostrapping process because the
248+This is different from a normal bootstrapping process because the
249 buildout egg itself is installed as a develop egg.
250
251 $Id$
252
253=== modified file 'src/zc/buildout/bootstrap.txt'
254--- src/zc/buildout/bootstrap.txt 2009-11-01 15:15:20 +0000
255+++ src/zc/buildout/bootstrap.txt 2010-02-17 21:25:19 +0000
256@@ -57,7 +57,7 @@
257 ... 'bootstrap.py --version UNKNOWN'); print 'X' # doctest: +ELLIPSIS
258 ...
259 X
260- No local packages or download links found for zc.buildout==UNKNOWN
261+ No local packages or download links found for zc.buildout==UNKNOWN...
262 ...
263
264 Now let's try with `1.1.2`, which happens to exist::
265@@ -119,8 +119,8 @@
266 zc.buildout.buildout.main()
267 <BLANKLINE>
268
269-`zc.buildout` now can also run with `Distribute` with the `--distribute` option::
270-
271+`zc.buildout` now can also run with `Distribute` with the `--distribute`
272+option::
273
274 >>> print 'X'; print system(
275 ... zc.buildout.easy_install._safe_arg(sys.executable)+' '+
276@@ -153,7 +153,8 @@
277
278 >>> print 'X'; print system(
279 ... zc.buildout.easy_install._safe_arg(sys.executable)+' '+
280- ... 'bootstrap.py --distribute --version 1.2.1'); print 'X' # doctest: +ELLIPSIS
281+ ... 'bootstrap.py --distribute --version 1.2.1'); print 'X'
282+ ... # doctest: +ELLIPSIS
283 ...
284 X
285 ...
286@@ -161,7 +162,8 @@
287 <BLANKLINE>
288 X
289
290-Let's make sure the generated `buildout` script uses ``Distribute`` *and* ``zc.buildout-1.2.1``::
291+Let's make sure the generated `buildout` script uses ``Distribute`` *and*
292+``zc.buildout-1.2.1``::
293
294 >>> print open(buildout_script).read() # doctest: +ELLIPSIS
295 #...
296@@ -194,4 +196,70 @@
297 <BLANKLINE>
298 X
299
300-
301+You can specify a location of ez_setup.py or distribute_setup, so you
302+can rely on a local or remote location. We'll write our own ez_setup.py
303+that we will also use to test some other bootstrap options.
304+
305+ >>> write('ez_setup.py', '''\
306+ ... def use_setuptools(**kwargs):
307+ ... import sys, pprint
308+ ... pprint.pprint(kwargs, width=40)
309+ ... sys.exit()
310+ ... ''')
311+ >>> print system(
312+ ... zc.buildout.easy_install._safe_arg(sys.executable)+' '+
313+ ... 'bootstrap.py --setup-source=./ez_setup.py')
314+ ... # doctest: +ELLIPSIS
315+ {'download_delay': 0,
316+ 'to_dir': '...'}
317+ <BLANKLINE>
318+
319+You can also pass a download-cache, and a place in which eggs should be stored
320+(they are normally stored in a temporary directory).
321+
322+ >>> print system(
323+ ... zc.buildout.easy_install._safe_arg(sys.executable)+' '+
324+ ... 'bootstrap.py --setup-source=./ez_setup.py '+
325+ ... '--download-base=./download-cache --eggs=eggs')
326+ ... # doctest: +ELLIPSIS
327+ {'download_base': '/sample/download-cache/',
328+ 'download_delay': 0,
329+ 'to_dir': '/sample/eggs'}
330+ <BLANKLINE>
331+
332+Here's the entire help text.
333+
334+ >>> print system(
335+ ... zc.buildout.easy_install._safe_arg(sys.executable)+' '+
336+ ... 'bootstrap.py --help'),
337+ ... # doctest: +ELLIPSIS
338+ usage: [DESIRED PYTHON FOR BUILDOUT] bootstrap.py [options]
339+ <BLANKLINE>
340+ Bootstraps a buildout-based project.
341+ <BLANKLINE>
342+ Simply run this script in a directory containing a buildout.cfg, using the
343+ Python that you want bin/buildout to use.
344+ <BLANKLINE>
345+ Note that by using --setup-source and --download-base to point to
346+ local resources, you can keep this script from going over the network.
347+ <BLANKLINE>
348+ <BLANKLINE>
349+ options:
350+ -h, --help show this help message and exit
351+ -v VERSION, --version=VERSION
352+ use a specific zc.buildout version
353+ -d, --distribute Use Distribute rather than Setuptools.
354+ --setup-source=SETUP_SOURCE
355+ Specify a URL or file location for the setup file. If
356+ you use Setuptools, this will default to
357+ http://peak.telecommunity.com/dist/ez_setup.py; if you
358+ use Distribute, this will default to http://python-
359+ distribute.org/distribute_setup.py.
360+ --download-base=DOWNLOAD_BASE
361+ Specify a URL or directory for downloading zc.buildout
362+ and either Setuptools or Distribute. Defaults to PyPI.
363+ --eggs=EGGS Specify a directory for storing eggs. Defaults to a
364+ temporary directory that is deleted when the bootstrap
365+ script completes.
366+ -c CONFIG_FILE Specify the path to the buildout configuration file to
367+ be used.
368
369=== modified file 'src/zc/buildout/buildout.py'
370--- src/zc/buildout/buildout.py 2009-11-06 22:33:23 +0000
371+++ src/zc/buildout/buildout.py 2010-02-17 21:25:19 +0000
372@@ -338,11 +338,32 @@
373
374 self._setup_directories()
375
376+ options = self['buildout']
377+
378+ # Get a base working set for our distributions that corresponds to the
379+ # stated desires in the configuration.
380+ distributions = ['setuptools', 'zc.buildout']
381+ if options.get('offline') == 'true':
382+ ws = zc.buildout.easy_install.working_set(
383+ distributions, options['executable'],
384+ [options['develop-eggs-directory'], options['eggs-directory']]
385+ )
386+ else:
387+ ws = zc.buildout.easy_install.install(
388+ distributions, options['eggs-directory'],
389+ links=self._links,
390+ index=options.get('index'),
391+ executable=options['executable'],
392+ path=[options['develop-eggs-directory']],
393+ newest=self.newest,
394+ allow_hosts=self._allow_hosts
395+ )
396+
397 # Now copy buildout and setuptools eggs, and record destination eggs:
398 entries = []
399 for name in 'setuptools', 'zc.buildout':
400 r = pkg_resources.Requirement.parse(name)
401- dist = pkg_resources.working_set.find(r)
402+ dist = ws.find(r)
403 if dist.precedence == pkg_resources.DEVELOP_DIST:
404 dest = os.path.join(self['buildout']['develop-eggs-directory'],
405 name+'.egg-link')
406@@ -362,8 +383,8 @@
407 ws = pkg_resources.WorkingSet(entries)
408 ws.require('zc.buildout')
409 zc.buildout.easy_install.scripts(
410- ['zc.buildout'], ws, sys.executable,
411- self['buildout']['bin-directory'])
412+ ['zc.buildout'], ws, options['executable'],
413+ options['bin-directory'])
414
415 init = bootstrap
416

Subscribers

People subscribed via source and target branches

to all changes: