Hi Bjorn. This is great to see. I think we can simplify the setup. I don't know if we need as many of the tricks that Launchpad uses. Here's a diff that removes a lot of them. This should be regarded as a conversation starter, rather than as a concrete suggestion. I'll call out the ideas here, and then you can see them concretely in the diff. I've marked the items that I strongly think are valid with three stars (***). - Launchpad keeps a separate, shared download-cache and eggs. That makes sense if the build is happening on a machine that does not allow network access, or if we want to prevent being exposed to PyPI service interruptions, or if we want to use custom distributions. On the other hand, if we remove that, we can have a simpler build. If developers want shared eggs, they can set up a global cache (https://dev.launchpad.net/HackingLazrLibraries#Global%20Cache). This change affects .bzrignore, and the Makefile (including the bin/buildout target). - I don't see any reason to specify the BUILDOUT_CFG in the Makefile. There's only one. - ``make clean`` might as well just trash the bin directory. *** - We can just use the interpreter option for zc.recipe.egg to produce the py file, and remove the z3c.recipe.filetemplate dependency (and the buildout-templates directory). We made the py file in Launchpad like that because Launchpad's Makefile wanted to use the -t option. (I have a branch I'm working on for Launchpad that switches it to using the interpreter option also.) *** - The entry points maybe ought to be in the setup.py file, so someone can use it without buildout. Maybe. Anyway, might as well put it in the expected place. *** - No need to specify extra-paths in buildout.cfg. You are already doing the right thing in setup.py. *** - If we change the Makefile to use -S whenever it calls Python, we don't need my custom releases of buildout that handle a system Python (which Jim still has on his queue to review), so we can just use released versions of the various packages. (On the other hand, if Jim approves my changes, we could arguably just get rid of the Makefile entirely and instruct people to run ``python bootstrap.py && bin/buildout && bin/build`` or something to get the build.) - Including ez_setup.py was another artifact of wanting to be able to build on a machine that had no outside net access. If we don't need to support that, we can remove it (and the reference in setup.py). If you agree with all of these thoughts, you can take the diff as is. Otherwise, hopefully it will still give us something to talk about it. If you just change the starred items, and push back convincingly on the rest, that will probably be fine. Thanks! Gary === modified file '.bzrignore' --- .bzrignore 2009-09-23 13:25:32 +0000 +++ .bzrignore 2009-10-07 19:34:10 +0000 @@ -7,6 +7,4 @@ parts src/inlineedit/assets/skins/sam/inlineedit.css src/lazr/assets/skins/sam/lazr.css -./download-cache -./eggs ./MANIFEST === modified file 'Makefile' --- Makefile 2009-10-07 15:42:30 +0000 +++ Makefile 2009-10-07 19:58:20 +0000 @@ -5,35 +5,21 @@ PYTHON=python${PYTHON_VERSION} WD:=$(shell pwd) PY=$(WD)/bin/py -BUILDOUT_CFG=buildout.cfg # Update the build directory build: $(PY) - bin/build + $(PYTHON) -S bin/build lint: bin/jslint -eggs: - mkdir eggs - -download-cache: - @$echo "Missing ./download-cache." - @exit 1 - -# The download-cache dependency comes *before* eggs so that developers get the -# warning before the eggs directory is made. The target for the eggs directory -# is only there for deployment convenience. -bin/buildout: download-cache eggs - $(SHHH) PYTHONPATH= $(PYTHON) bootstrap.py\ - --ez_setup-source=ez_setup.py \ - --download-base=download-cache/dist --eggs=eggs - -$(PY): bin/buildout $(BUILDOUT_CFG) setup.py - PYTHONPATH= ./bin/buildout -c $(BUILDOUT_CFG) - +bin/buildout: + $(PYTHON) -S bootstrap.py + +$(PY): bin/buildout buildout.cfg setup.py + $(PYTHON) -S ./bin/buildout clean: - rm -fr build/* bin/buildout + rm -fr build/* bin .PHONY: build lint clean === removed directory 'buildout-templates' === removed directory 'buildout-templates/bin' === removed file 'buildout-templates/bin/py.in' --- buildout-templates/bin/py.in 2009-09-23 12:17:15 +0000 +++ buildout-templates/bin/py.in 1970-01-01 00:00:00 +0000 @@ -1,2 +0,0 @@ -#!/bin/sh -PYTHONPATH=${os-paths} exec ${buildout:executable} "$@" === modified file 'buildout.cfg' --- buildout.cfg 2009-10-07 15:42:57 +0000 +++ buildout.cfg 2009-10-07 19:33:21 +0000 @@ -1,26 +1,10 @@ [buildout] develop = . parts = - filetemplates scripts - unzip = true -eggs-directory = eggs -download-cache = download-cache - -# Disable this option temporarily if you want buildout to find software -# dependencies *other* than those in our download-cache. Once you have the -# desired software, reenable this option (and check in the new software to -# lp:lp-source-dependencies if this is going to be reviewed/merged/deployed.) -install-from-cache = true - allow-picked-versions = false - -allowed-eggs-from-site-packages = -include-site-packages-for-buildout = false - prefer-final = true - versions = versions [filetemplates] @@ -30,17 +14,12 @@ [scripts] recipe = zc.recipe.egg +interpreter = py eggs = lazr-js -entry-points = - build=lazr.js.build:main - jslint=lazr.js.jslint:main - scaffold=lazr.js.scaffold:main -extra-paths = ${buildout:directory}/src-py [versions] # Alphabetical, case-insensitive, please! :-) bzr = 2.0.0 setuptools = 0.6c9 -zc.buildout = 1.5.0dev-gary-r103553 -zc.recipe.egg = 1.3.0dev-gary-r103515 -z3c.recipe.filetemplate = 2.1dev-gary-r103545 +zc.buildout = 1.4.1 +zc.recipe.egg = 1.2.2 === removed file 'ez_setup.py' --- ez_setup.py 2009-10-06 07:27:36 +0000 +++ ez_setup.py 1970-01-01 00:00:00 +0000 @@ -1,275 +0,0 @@ -#!python -"""Bootstrap setuptools installation - -If you want to use setuptools in your package's setup.py, just include this -file in the same directory with it, and add this to the top of your setup.py:: - - from ez_setup import use_setuptools - use_setuptools() - -If you want to require a specific version of setuptools, set a download -mirror, or use an alternate download directory, you can do so by supplying -the appropriate options to ``use_setuptools()``. - -This file can also be run as a script to install or upgrade setuptools. -""" -import sys -DEFAULT_VERSION = "0.6c9" -DEFAULT_URL = "http://pypi.python.org/packages/%s/s/setuptools/" % sys.version[:3] - -md5_data = { - 'setuptools-0.6b1-py2.3.egg': '8822caf901250d848b996b7f25c6e6ca', - 'setuptools-0.6b1-py2.4.egg': 'b79a8a403e4502fbb85ee3f1941735cb', - 'setuptools-0.6b2-py2.3.egg': '5657759d8a6d8fc44070a9d07272d99b', - 'setuptools-0.6b2-py2.4.egg': '4996a8d169d2be661fa32a6e52e4f82a', - 'setuptools-0.6b3-py2.3.egg': 'bb31c0fc7399a63579975cad9f5a0618', - 'setuptools-0.6b3-py2.4.egg': '38a8c6b3d6ecd22247f179f7da669fac', - 'setuptools-0.6b4-py2.3.egg': '62045a24ed4e1ebc77fe039aa4e6f7e5', - 'setuptools-0.6b4-py2.4.egg': '4cb2a185d228dacffb2d17f103b3b1c4', - 'setuptools-0.6c1-py2.3.egg': 'b3f2b5539d65cb7f74ad79127f1a908c', - 'setuptools-0.6c1-py2.4.egg': 'b45adeda0667d2d2ffe14009364f2a4b', - 'setuptools-0.6c2-py2.3.egg': 'f0064bf6aa2b7d0f3ba0b43f20817c27', - 'setuptools-0.6c2-py2.4.egg': '616192eec35f47e8ea16cd6a122b7277', - 'setuptools-0.6c3-py2.3.egg': 'f181fa125dfe85a259c9cd6f1d7b78fa', - 'setuptools-0.6c3-py2.4.egg': 'e0ed74682c998bfb73bf803a50e7b71e', - 'setuptools-0.6c3-py2.5.egg': 'abef16fdd61955514841c7c6bd98965e', - 'setuptools-0.6c4-py2.3.egg': 'b0b9131acab32022bfac7f44c5d7971f', - 'setuptools-0.6c4-py2.4.egg': '2a1f9656d4fbf3c97bf946c0a124e6e2', - 'setuptools-0.6c4-py2.5.egg': '8f5a052e32cdb9c72bcf4b5526f28afc', - 'setuptools-0.6c5-py2.3.egg': 'ee9fd80965da04f2f3e6b3576e9d8167', - 'setuptools-0.6c5-py2.4.egg': 'afe2adf1c01701ee841761f5bcd8aa64', - 'setuptools-0.6c5-py2.5.egg': 'a8d3f61494ccaa8714dfed37bccd3d5d', - 'setuptools-0.6c6-py2.3.egg': '35686b78116a668847237b69d549ec20', - 'setuptools-0.6c6-py2.4.egg': '3c56af57be3225019260a644430065ab', - 'setuptools-0.6c6-py2.5.egg': 'b2f8a7520709a5b34f80946de5f02f53', - 'setuptools-0.6c7-py2.3.egg': '209fdf9adc3a615e5115b725658e13e2', - 'setuptools-0.6c7-py2.4.egg': '5a8f954807d46a0fb67cf1f26c55a82e', - 'setuptools-0.6c7-py2.5.egg': '45d2ad28f9750e7434111fde831e8372', - 'setuptools-0.6c8-py2.3.egg': '50759d29b349db8cfd807ba8303f1902', - 'setuptools-0.6c8-py2.4.egg': 'cba38d74f7d483c06e9daa6070cce6de', - 'setuptools-0.6c8-py2.5.egg': '1721747ee329dc150590a58b3e1ac95b', - 'setuptools-0.6c9-py2.3.egg': 'a83c4020414807b496e4cfbe08507c03', - 'setuptools-0.6c9-py2.4.egg': '260a2be2e5388d66bdaee06abec6342a', - 'setuptools-0.6c9-py2.5.egg': 'fe67c3e5a17b12c0e7c541b7ea43a8e6', - 'setuptools-0.6c9-py2.6.egg': 'ca37b1ff16fa2ede6e19383e7b59245a', -} - -import sys, os -try: from hashlib import md5 -except ImportError: from md5 import md5 - -def _validate_md5(egg_name, data): - if egg_name in md5_data: - digest = md5(data).hexdigest() - if digest != md5_data[egg_name]: - print >>sys.stderr, ( - "md5 validation of %s failed! (Possible download problem?)" - % egg_name - ) - sys.exit(2) - return data - -def use_setuptools( - version=DEFAULT_VERSION, download_base=DEFAULT_URL, to_dir=os.curdir, - download_delay=15 -): - """Automatically find/download setuptools and make it available on sys.path - - `version` should be a valid setuptools version number that is available - as an egg for download under the `download_base` URL (which should end with - a '/'). `to_dir` is the directory where setuptools will be downloaded, if - it is not already available. If `download_delay` is specified, it should - be the number of seconds that will be paused before initiating a download, - should one be required. If an older version of setuptools is installed, - this routine will print a message to ``sys.stderr`` and raise SystemExit in - an attempt to abort the calling script. - """ - was_imported = 'pkg_resources' in sys.modules or 'setuptools' in sys.modules - def do_download(): - egg = download_setuptools(version, download_base, to_dir, download_delay) - sys.path.insert(0, egg) - import setuptools; setuptools.bootstrap_install_from = egg - try: - import pkg_resources - except ImportError: - return do_download() - try: - pkg_resources.require("setuptools>="+version); return - except pkg_resources.VersionConflict, e: - if was_imported: - print >>sys.stderr, ( - "The required version of setuptools (>=%s) is not available, and\n" - "can't be installed while this script is running. Please install\n" - " a more recent version first, using 'easy_install -U setuptools'." - "\n\n(Currently using %r)" - ) % (version, e.args[0]) - sys.exit(2) - else: - del pkg_resources, sys.modules['pkg_resources'] # reload ok - return do_download() - except pkg_resources.DistributionNotFound: - return do_download() - -def download_setuptools( - version=DEFAULT_VERSION, download_base=DEFAULT_URL, to_dir=os.curdir, - delay = 15 -): - """Download setuptools from a specified location and return its filename - - `version` should be a valid setuptools version number that is available - as an egg for download under the `download_base` URL (which should end - with a '/'). `to_dir` is the directory where the egg will be downloaded. - `delay` is the number of seconds to pause before an actual download attempt. - """ - import urllib2, shutil - egg_name = "setuptools-%s-py%s.egg" % (version,sys.version[:3]) - url = download_base + egg_name - saveto = os.path.join(to_dir, egg_name) - src = dst = None - if not os.path.exists(saveto): # Avoid repeated downloads - try: - from distutils import log - if delay: - log.warn(""" ---------------------------------------------------------------------------- -This script requires setuptools version %s to run (even to display -help). I will attempt to download it for you (from -%s), but -you may need to enable firewall access for this script first. -I will start the download in %d seconds. - -(Note: if this machine does not have network access, please obtain the file - - %s - -and place it in this directory before rerunning this script.) ----------------------------------------------------------------------------""", - version, download_base, delay, url - ); from time import sleep; sleep(delay) - log.warn("Downloading %s", url) - src = urllib2.urlopen(url) - # Read/write all in one block, so we don't create a corrupt file - # if the download is interrupted. - data = _validate_md5(egg_name, src.read()) - dst = open(saveto,"wb"); dst.write(data) - finally: - if src: src.close() - if dst: dst.close() - return os.path.realpath(saveto) - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -def main(argv, version=DEFAULT_VERSION): - """Install or upgrade setuptools and EasyInstall""" - try: - import setuptools - except ImportError: - egg = None - try: - egg = download_setuptools(version, delay=0) - sys.path.insert(0,egg) - from setuptools.command.easy_install import main - return main(list(argv)+[egg]) # we're done here - finally: - if egg and os.path.exists(egg): - os.unlink(egg) - else: - if setuptools.__version__ == '0.0.1': - print >>sys.stderr, ( - "You have an obsolete version of setuptools installed. Please\n" - "remove it from your system entirely before rerunning this script." - ) - sys.exit(2) - - req = "setuptools>="+version - import pkg_resources - try: - pkg_resources.require(req) - except pkg_resources.VersionConflict: - try: - from setuptools.command.easy_install import main - except ImportError: - from easy_install import main - main(list(argv)+[download_setuptools(delay=0)]) - sys.exit(0) # try to force an exit - else: - if argv: - from setuptools.command.easy_install import main - main(argv) - else: - print "Setuptools version",version,"or greater has been installed." - print '(Run "ez_setup.py -U setuptools" to reinstall or upgrade.)' - -def update_md5(filenames): - """Update our built-in md5 registry""" - - import re - - for name in filenames: - base = os.path.basename(name) - f = open(name,'rb') - md5_data[base] = md5(f.read()).hexdigest() - f.close() - - data = [" %r: %r,\n" % it for it in md5_data.items()] - data.sort() - repl = "".join(data) - - import inspect - srcfile = inspect.getsourcefile(sys.modules[__name__]) - f = open(srcfile, 'rb'); src = f.read(); f.close() - - match = re.search("\nmd5_data = {\n([^}]+)}", src) - if not match: - print >>sys.stderr, "Internal error!" - sys.exit(2) - - src = src[:match.start(1)] + repl + src[match.end(1):] - f = open(srcfile,'w') - f.write(src) - f.close() - - -if __name__=='__main__': - if len(sys.argv)>2 and sys.argv[1]=='--md5update': - update_md5(sys.argv[2:]) - else: - main(sys.argv[1:]) - - - - - === modified file 'setup.py' --- setup.py 2009-10-07 15:43:55 +0000 +++ setup.py 2009-10-07 19:35:51 +0000 @@ -1,8 +1,5 @@ # Copyright 2009 Canonical Ltd. All rights reserved. -import ez_setup -ez_setup.use_setuptools() - import sys from setuptools import setup, find_packages @@ -39,6 +36,9 @@ ), entry_points=dict( console_scripts=[ + 'build=lazr.js.build:main', + 'jslint=lazr.js.jslint:main', + 'scaffold=lazr.js.scaffold:main', ] ), )