Merge lp:~leonardr/launchpad/multiversion-apidoc into lp:launchpad/db-devel
- multiversion-apidoc
- Merge into db-devel
Status: | Merged |
---|---|
Approved by: | Aaron Bentley |
Approved revision: | no longer in the source branch. |
Merged at revision: | not available |
Proposed branch: | lp:~leonardr/launchpad/multiversion-apidoc |
Merge into: | lp:launchpad/db-devel |
Prerequisite: | lp:~leonardr/launchpad/no-mutator-named-operations |
Diff against target: |
1156 lines (+535/-157) 21 files modified
.bzrignore (+2/-2) Makefile (+5/-9) lib/canonical/buildd/debian/launchpad-buildd.init (+1/-1) lib/canonical/launchpad/pagetests/webservice/apidoc.txt (+17/-3) lib/canonical/launchpad/pagetests/webservice/multiversion.txt (+32/-0) lib/canonical/launchpad/rest/configuration.py (+21/-0) lib/canonical/launchpad/webapp/tests/test_servers.py (+1/-1) lib/lp/bugs/stories/webservice/xx-bug.txt (+9/-40) lib/lp/buildmaster/interfaces/buildfarmjobbehavior.py (+1/-1) lib/lp/buildmaster/model/buildfarmjobbehavior.py (+94/-10) lib/lp/buildmaster/tests/test_buildfarmjobbehavior.py (+80/-8) lib/lp/code/model/recipebuilder.py (+11/-2) lib/lp/scripts/utilities/apiindex.py (+0/-21) lib/lp/soyuz/doc/buildd-slavescanner.txt (+4/-3) lib/lp/testing/factory.py (+87/-40) lib/lp/translations/doc/translationtemplatesbuildbehavior.txt (+41/-12) lib/lp/translations/model/translationtemplatesbuildbehavior.py (+17/-1) lib/lp/translations/tests/test_translationtemplatesbuildbehavior.py (+37/-0) utilities/apidoc-index.pt (+37/-0) utilities/create-lp-wadl-and-apidoc.py (+35/-0) versions.cfg (+3/-3) |
To merge this branch: | bzr merge lp:~leonardr/launchpad/multiversion-apidoc |
Related bugs: |
Reviewer | Review Type | Date Requested | Status |
---|---|---|---|
Aaron Bentley (community) | Approve | ||
Review via email: mp+21181@code.launchpad.net |
Commit message
Description of the change
This branch makes Launchpad generate an HTML apidoc for every version of the web service, not just the devel version. The apidoc for version "foo" is written to lib/canonical/
This branch renames create-lp-wadl.py to create-
I would really like feedback on the following issues:
1. Is there a way to put apidoc-index.pt somewhere else? Maybe in lib/canonical/
2. Is there a way to look up config/
This branch still needs a little bit of work--I need to fix up the apidoc test so it tests the new system.
Leonard Richardson (leonardr) wrote : | # |
Leonard Richardson (leonardr) wrote : | # |
One test on EC2 failed: the title of the generated apidoc is now "About this service", not "Launchpad Web Service API". Our XSLT stylesheet (which is based on Mark Nottingham's general WADL stylesheet) has this code:
Previously the WADL's <application> tag had no <doc> children, so we used the default. Now there are two <doc> tags, "About this service" and "About version {version}".
This is actually pretty nice; I was afraid I would have to hack the XSLT to add the <doc> tags to the HTML, but it already processes them. The only problem is that the XSLT stylesheet uses the title of the first <doc> tag as the title of the whole document, and ignores the title of all subsequent <doc> tags. I was hoping for this (#1):
---
<h1>Launchpad Web Service API</h1>
<h2>About this service</h2>
The Launchpad web service allows automated clients to access most of the functionality available on the Launchpad web site. For help getting started, see the help wiki.
<h2>About version beta</h2>
This is the first version of the web service ever published. Its end-of-life date is April 2011, the same as the Ubuntu release "Karmic Koala".
---
And upon reflection I think this would be even better (#2):
---
<h1>Launchpad web service, version beta</h1>
The Launchpad web service allows automated clients to access most of the functionality available on the Launchpad web site. For help getting started, see the help wiki.
This is the first version of the web service ever published. Its end-of-life date is April 2011, the same as the Ubuntu release "Karmic Koala".
---
Instead, we have this (#3):
---
<h1>About this service</h1>
The Launchpad web service allows automated clients to access most of the functionality available on the Launchpad web site. For help getting started, see the help wiki.
This is the first version of the web service ever published. Its end-of-life date is April 2011, the same as the Ubuntu release "Karmic Koala".
---
We can change the stylesheet to get #1 and change lazr.restful to get #2. But I think what we have now is good enough that I can file a bug and come back to fix the generated documentation later. What do you think?
Leonard Richardson (leonardr) wrote : | # |
And here's another diff against the precursor branch: http://
Aaron Bentley (abentley) wrote : | # |
Please use extract_text if possible. Otherwise, this looks fine.
Gary Poster (gary) wrote : | # |
Leonard asked me to take a look at this because of his two specific questions.
1. Is there a way to put apidoc-index.pt somewhere else? Maybe in lib/canonical/
The FilePageTemplate just expects a path. I'm relatively confident that you can use os.path fun, combined with module __file__ fun, to do what you describe. However, as we agreed on IRC, I'm fine with what you are doing now.
2. Is there a way to look up config/
The infrequently used syntax for this is "?," I believe. In other words, if you have a variable named "config" and a variable named "version" that contains the string "beta", "config/
On a separate note, also as we discussed on IRC, this segment from create-
1115 + # Finally, create an index.html with links to all the HTML
1116 + # documentation files we just generated.
1117 + template_file = 'apidoc-index.pt'
1118 + template = PageTemplateFil
1119 + namespace = template.
1120 + namespace['config'] = config
1121 + versions_
1122 + for version in config.
1123 + versions_
1124 + dict(version=
1125 + description=
1126 + namespace[
1127 +
1128 + f = open(os.
1129 + f.write(
This could be simplified to the following:
template_file = 'apidoc-index.pt'
template = PageTemplateFil
versions_
for version in config.
f = open(os.
f.write(
Then in your template you would access config and versions_
If you use the ? syntax I mentioned above for options/
template_file = 'apidoc-index.pt'
template = PageTemplateFil
f = open(os.
f.write(
That's way nicer, I think.
Preview Diff
1 | === modified file '.bzrignore' |
2 | --- .bzrignore 2010-03-09 02:42:40 +0000 |
3 | +++ .bzrignore 2010-03-15 21:11:07 +0000 |
4 | @@ -30,9 +30,9 @@ |
5 | database/sampledata/newsampledata-dev.sql |
6 | database/sampledata/lintdata.sql |
7 | database/sampledata/lintdata-dev.sql |
8 | -lib/canonical/launchpad/apidoc/index.html |
9 | +lib/canonical/launchpad/apidoc/*.html |
10 | xxx-report.* |
11 | -lib/canonical/launchpad/apidoc/wadl-development.xml |
12 | +lib/canonical/launchpad/apidoc/wadl-development-*.xml |
13 | lib/canonical/launchpad/apidoc/wadl-test-playground.xml |
14 | lib/canonical/launchpad/icing/build/* |
15 | lib/canonical/launchpad/icing/combo.css |
16 | |
17 | === modified file 'Makefile' |
18 | --- Makefile 2010-03-05 21:47:03 +0000 |
19 | +++ Makefile 2010-03-15 21:11:07 +0000 |
20 | @@ -30,7 +30,6 @@ |
21 | |
22 | APIDOC_DIR = lib/canonical/launchpad/apidoc |
23 | WADL_TEMPLATE = $(APIDOC_DIR).tmp/wadl-$(LPCONFIG)-%(version)s.xml |
24 | -DEVEL_WADL_FILE = $(APIDOC_DIR)/wadl-$(LPCONFIG)-devel.xml |
25 | API_INDEX = $(APIDOC_DIR)/index.html |
26 | |
27 | # Do not add bin/buildout to this list. |
28 | @@ -60,16 +59,12 @@ |
29 | hosted_branches: $(PY) |
30 | $(PY) ./utilities/make-dummy-hosted-branches |
31 | |
32 | -$(DEVEL_WADL_FILE): $(BZR_VERSION_INFO) |
33 | +$(API_INDEX): $(BZR_VERSION_INFO) |
34 | mkdir $(APIDOC_DIR).tmp |
35 | - LPCONFIG=$(LPCONFIG) $(PY) ./utilities/create-lp-wadl.py "$(WADL_TEMPLATE)" |
36 | + LPCONFIG=$(LPCONFIG) $(PY) ./utilities/create-lp-wadl-and-apidoc.py "$(WADL_TEMPLATE)" |
37 | mv $(APIDOC_DIR).tmp/* $(APIDOC_DIR) |
38 | rmdir $(APIDOC_DIR).tmp |
39 | |
40 | -$(API_INDEX): $(DEVEL_WADL_FILE) |
41 | - bin/apiindex $(DEVEL_WADL_FILE) > $@.tmp |
42 | - mv $@.tmp $@ |
43 | - |
44 | apidoc: compile $(API_INDEX) |
45 | |
46 | check_merge: $(PY) |
47 | @@ -259,7 +254,7 @@ |
48 | $(BZR_VERSION_INFO): |
49 | scripts/update-bzr-version-info.sh |
50 | |
51 | -support_files: $(DEVEL_WADL_FILE) $(BZR_VERSION_INFO) |
52 | +support_files: $(API_INDEX) $(BZR_VERSION_INFO) |
53 | |
54 | # Intended for use on developer machines |
55 | start: inplace stop support_files initscript-start |
56 | @@ -330,9 +325,10 @@ |
57 | $(RM) -r $(CODEHOSTING_ROOT) |
58 | mv $(APIDOC_DIR)/wadl-testrunner-devel.xml \ |
59 | $(APIDOC_DIR)/wadl-testrunner-devel.xml.bak |
60 | - $(RM) $(APIDOC_DIR)/wadl*.xml $(API_INDEX) |
61 | + $(RM) $(APIDOC_DIR)/wadl*.xml $(APIDOC_DIR)/*.html |
62 | mv $(APIDOC_DIR)/wadl-testrunner-devel.xml.bak \ |
63 | $(APIDOC_DIR)/wadl-testrunner-devel.xml |
64 | + $(RM) -rf $(APIDOC_DIR).tmp |
65 | $(RM) $(BZR_VERSION_INFO) |
66 | $(RM) _pythonpath.py |
67 | $(RM) -rf \ |
68 | |
69 | === modified file 'lib/canonical/buildd/debian/launchpad-buildd.init' |
70 | --- lib/canonical/buildd/debian/launchpad-buildd.init 2009-12-16 00:16:24 +0000 |
71 | +++ lib/canonical/buildd/debian/launchpad-buildd.init 2010-03-15 21:11:07 +0000 |
72 | @@ -29,7 +29,7 @@ |
73 | CONF=$1 |
74 | PIDFILE="$PIDROOT"/"$CONF".pid |
75 | LOGFILE="$LOGROOT"/"$CONF".log |
76 | - su - buildd -c "BUILDD_SLAVE_CONFIG=$CONFROOT/$CONF PYTHONPATH=/usr/share/launchpad-buildd twistd --no_save --pidfile $PIDFILE --python $TACFILE --logfile $LOGFILE" |
77 | + su - buildd -c "BUILDD_SLAVE_CONFIG=$CONFROOT/$CONF PYTHONPATH=/usr/share/launchpad-buildd twistd --no_save --pidfile $PIDFILE --python $TACFILE --logfile $LOGFILE --umask 022" |
78 | } |
79 | |
80 | # |
81 | |
82 | === modified file 'lib/canonical/launchpad/pagetests/webservice/apidoc.txt' |
83 | --- lib/canonical/launchpad/pagetests/webservice/apidoc.txt 2008-07-31 21:23:02 +0000 |
84 | +++ lib/canonical/launchpad/pagetests/webservice/apidoc.txt 2010-03-15 21:11:07 +0000 |
85 | @@ -1,9 +1,23 @@ |
86 | = API Documentation = |
87 | |
88 | -The up-to-date Launchpad API documentation is available from |
89 | -http://launchpad.dev/+apidoc. |
90 | +The Launchpad API documentation portal is available at |
91 | +http://launchpad.dev/+apidoc. It contains summaries of the different |
92 | +web service versions, and links to version-specific documents. |
93 | |
94 | >>> browser.open('http://launchpad.dev/+apidoc') |
95 | + >>> print extract_text(browser.contents) |
96 | + Launchpad Web Service API |
97 | + Launchpad web service API documentation |
98 | + ... |
99 | + Active versions |
100 | + ... |
101 | + devel: This version of the web service reflects the most recent... |
102 | + ... |
103 | + |
104 | +The documentation for a specific version is located at |
105 | +http://launchpad.dev/+apidoc/{version}.html. |
106 | + |
107 | + >>> browser.open('http://launchpad.dev/+apidoc/devel.html') |
108 | >>> print browser.title |
109 | - Launchpad Web Service API |
110 | + About this service |
111 | |
112 | |
113 | === added file 'lib/canonical/launchpad/pagetests/webservice/multiversion.txt' |
114 | --- lib/canonical/launchpad/pagetests/webservice/multiversion.txt 1970-01-01 00:00:00 +0000 |
115 | +++ lib/canonical/launchpad/pagetests/webservice/multiversion.txt 2010-03-15 21:11:07 +0000 |
116 | @@ -0,0 +1,32 @@ |
117 | +Differences between versions |
118 | +---------------------------- |
119 | + |
120 | +In the 'beta' version of the web service, mutator methods like |
121 | +IBugTask.transitionToStatus are published as named operations. In |
122 | +subsequent versions, those named operations are not published. |
123 | + |
124 | + >>> def get_bugtask_path(version): |
125 | + ... bug_one = webservice.get("/bugs/1", api_version=version).jsonBody() |
126 | + ... bug_one_bugtasks_url = bug_one['bug_tasks_collection_link'] |
127 | + ... bug_one_bugtasks = sorted(webservice.get( |
128 | + ... bug_one_bugtasks_url).jsonBody()['entries']) |
129 | + ... bugtask_path = bug_one_bugtasks[0]['self_link'] |
130 | + ... return bugtask_path |
131 | + |
132 | +Here's the 'beta' version, where the named operation works. |
133 | + |
134 | + >>> url = get_bugtask_path('beta') |
135 | + >>> print webservice.named_post( |
136 | + ... url, 'transitionToImportance', importance='Low', |
137 | + ... api_version='beta') |
138 | + HTTP/1.1 200 Ok |
139 | + ... |
140 | + |
141 | +Now let's try the same thing in the '1.0' version, where it fails. |
142 | + |
143 | + >>> url = get_bugtask_path('1.0') |
144 | + >>> print webservice.named_post( |
145 | + ... url, 'transitionToImportance', importance='Low', |
146 | + ... api_version='devel') |
147 | + HTTP/1.1 405 Method Not Allowed |
148 | + ... |
149 | |
150 | === modified file 'lib/canonical/launchpad/rest/configuration.py' |
151 | --- lib/canonical/launchpad/rest/configuration.py 2010-02-15 02:09:32 +0000 |
152 | +++ lib/canonical/launchpad/rest/configuration.py 2010-03-15 21:11:07 +0000 |
153 | @@ -24,9 +24,30 @@ |
154 | |
155 | path_override = "api" |
156 | active_versions = ["beta", "1.0", "devel"] |
157 | + last_version_with_mutator_named_operations = "beta" |
158 | view_permission = "launchpad.View" |
159 | set_hop_by_hop_headers = True |
160 | |
161 | + service_description = """The Launchpad web service allows automated |
162 | + clients to access most of the functionality available on the |
163 | + Launchpad web site. For help getting started, see |
164 | + <a href="https://help.launchpad.net/API/">the help wiki.</a>""" |
165 | + |
166 | + version_descriptions = { |
167 | + "beta": """This is the first version of the web service ever |
168 | + published. Its end-of-life date is April 2011, the same as the |
169 | + Ubuntu release "Karmic Koala".""", |
170 | + |
171 | + "1.0": """This version of the web service removes unnecessary |
172 | + named operations. It was introduced in March 2010, and its |
173 | + end-of-life date is April 2015, the same as the server version |
174 | + of the Ubuntu release "Lucid Lynx".""", |
175 | + |
176 | + "devel": """This version of the web service reflects the most |
177 | + recent changes made. It may abruptly change without |
178 | + warning. Periodically, these changes are bundled up and given a |
179 | + permanent version number.""" } |
180 | + |
181 | @property |
182 | def use_https(self): |
183 | return config.vhosts.use_https |
184 | |
185 | === modified file 'lib/canonical/launchpad/webapp/tests/test_servers.py' |
186 | --- lib/canonical/launchpad/webapp/tests/test_servers.py 2010-02-24 14:45:33 +0000 |
187 | +++ lib/canonical/launchpad/webapp/tests/test_servers.py 2010-03-15 21:11:07 +0000 |
188 | @@ -14,7 +14,7 @@ |
189 | from lazr.restful.interfaces import ( |
190 | IServiceRootResource, IWebServiceConfiguration) |
191 | from lazr.restful.simple import RootResource |
192 | -from lazr.restful.tests.test_webservice import ( |
193 | +from lazr.restful.testing.webservice import ( |
194 | IGenericCollection, IGenericEntry, WebServiceTestCase) |
195 | |
196 | from lp.testing import TestCase |
197 | |
198 | === modified file 'lib/lp/bugs/stories/webservice/xx-bug.txt' |
199 | --- lib/lp/bugs/stories/webservice/xx-bug.txt 2010-03-10 14:14:38 +0000 |
200 | +++ lib/lp/bugs/stories/webservice/xx-bug.txt 2010-03-15 21:11:07 +0000 |
201 | @@ -357,7 +357,7 @@ |
202 | http://.../~cprov |
203 | |
204 | |
205 | -The task's importance can be modified directly... |
206 | +The task's importance can be modified directly. |
207 | |
208 | >>> body = webservice.get(bugtask_path).jsonBody() |
209 | >>> body['importance'] |
210 | @@ -371,20 +371,6 @@ |
211 | >>> body['importance'] |
212 | u'High' |
213 | |
214 | -...or through a named operation. |
215 | - |
216 | - >>> print webservice.named_post( |
217 | - ... bugtask_path, 'transitionToImportance', importance='Low') |
218 | - HTTP/1.1 200 Ok |
219 | - ... |
220 | - Content-Type: application/json |
221 | - ... |
222 | - <BLANKLINE> |
223 | - null |
224 | - >>> body = webservice.get(bugtask_path).jsonBody() |
225 | - >>> body['importance'] |
226 | - u'Low' |
227 | - |
228 | Only bug supervisors or people who can otherwise edit the bugtask's |
229 | pillar are authorised to edit the importance. |
230 | |
231 | @@ -394,10 +380,9 @@ |
232 | |
233 | >>> body = webservice.get(bugtask_path).jsonBody() |
234 | >>> body['importance'] |
235 | - u'Low' |
236 | + u'High' |
237 | |
238 | -The task's status is another aspect of the task that can be modified |
239 | -either directly or through a named operation. |
240 | +The task's status can also be modified directly. |
241 | |
242 | >>> print webservice.get(bugtask_path).jsonBody()['status'] |
243 | Confirmed |
244 | @@ -409,21 +394,14 @@ |
245 | >>> print webservice.get(bugtask_path).jsonBody()['status'] |
246 | Fix Committed |
247 | |
248 | - >>> print webservice.named_post( |
249 | - ... bugtask_path, 'transitionToStatus', status='New') |
250 | - HTTP/1.1 200 Ok... |
251 | - |
252 | - >>> print webservice.get(bugtask_path).jsonBody()['status'] |
253 | - New |
254 | - |
255 | If an error occurs during a request that sets both 'status' and |
256 | 'importance', neither one will be set. |
257 | |
258 | >>> task = webservice.get(bugtask_path).jsonBody() |
259 | >>> print task['status'] |
260 | - New |
261 | + Fix Committed |
262 | >>> print task['importance'] |
263 | - Low |
264 | + High |
265 | |
266 | >>> patch = {u'importance': 'High', u'status': u'No Such Status'} |
267 | >>> print webservice.patch(bugtask_path, 'application/json', dumps(patch)) |
268 | @@ -431,12 +409,11 @@ |
269 | |
270 | >>> task = webservice.get(bugtask_path).jsonBody() |
271 | >>> print task['status'] |
272 | - New |
273 | + Fix Committed |
274 | >>> print task['importance'] |
275 | - Low |
276 | + High |
277 | |
278 | -Like the importance, the milestone can be set either directly or via a |
279 | -named operation, and only by appropriately privileged users. |
280 | +The milestone can only be set by appropriately privileged users. |
281 | |
282 | >>> print webservice.get(bugtask_path).jsonBody()['milestone_link'] |
283 | None |
284 | @@ -449,20 +426,12 @@ |
285 | >>> print webservice.get(bugtask_path).jsonBody()['milestone_link'] |
286 | http://.../debian/+milestone/3.1 |
287 | |
288 | - >>> print webservice.named_post( |
289 | - ... bugtask_path, 'transitionToMilestone', |
290 | - ... milestone=webservice.getAbsoluteUrl('/debian/+milestone/3.1-rc1')) |
291 | - HTTP/1.1 200 Ok... |
292 | - |
293 | - >>> print webservice.get(bugtask_path).jsonBody()['milestone_link'] |
294 | - http://.../debian/+milestone/3.1-rc1 |
295 | - |
296 | >>> print user_webservice.patch( |
297 | ... bugtask_path, 'application/json', dumps(patch)) |
298 | HTTP/1.1 401 Unauthorized... |
299 | |
300 | >>> print webservice.get(bugtask_path).jsonBody()['milestone_link'] |
301 | - http://.../debian/+milestone/3.1-rc1 |
302 | + http://.../debian/+milestone/3.1 |
303 | |
304 | We can change the task's target. Here we change the task's target from |
305 | the mozilla-firefox package to alsa-utils. |
306 | |
307 | === modified file 'lib/lp/buildmaster/interfaces/buildfarmjobbehavior.py' |
308 | --- lib/lp/buildmaster/interfaces/buildfarmjobbehavior.py 2010-01-21 05:03:16 +0000 |
309 | +++ lib/lp/buildmaster/interfaces/buildfarmjobbehavior.py 2010-03-15 21:11:07 +0000 |
310 | @@ -1,4 +1,4 @@ |
311 | -# Copyright 2009 Canonical Ltd. This software is licensed under the |
312 | +# Copyright 2009-2010 Canonical Ltd. This software is licensed under the |
313 | # GNU Affero General Public License version 3 (see the file LICENSE). |
314 | |
315 | # pylint: disable-msg=E0211,E0213 |
316 | |
317 | === modified file 'lib/lp/buildmaster/model/buildfarmjobbehavior.py' |
318 | --- lib/lp/buildmaster/model/buildfarmjobbehavior.py 2010-03-05 13:52:32 +0000 |
319 | +++ lib/lp/buildmaster/model/buildfarmjobbehavior.py 2010-03-15 21:11:07 +0000 |
320 | @@ -1,4 +1,4 @@ |
321 | -# Copyright 2009 Canonical Ltd. This software is licensed under the |
322 | +# Copyright 2009-2010 Canonical Ltd. This software is licensed under the |
323 | # GNU Affero General Public License version 3 (see the file LICENSE). |
324 | |
325 | # pylint: disable-msg=E0211,E0213 |
326 | @@ -17,17 +17,22 @@ |
327 | import xmlrpclib |
328 | |
329 | from sqlobject import SQLObjectNotFound |
330 | + |
331 | from zope.component import getUtility |
332 | from zope.interface import implements |
333 | -from zope.security.proxy import removeSecurityProxy |
334 | +from zope.security.proxy import removeSecurityProxy, isinstance as zisinstance |
335 | |
336 | from canonical import encoding |
337 | from canonical.librarian.interfaces import ILibrarianClient |
338 | + |
339 | +from canonical.launchpad.webapp.interfaces import NotFoundError |
340 | from lp.buildmaster.interfaces.builder import CorruptBuildID |
341 | from lp.buildmaster.interfaces.buildfarmjobbehavior import ( |
342 | BuildBehaviorMismatch, IBuildFarmJobBehavior) |
343 | from lp.buildmaster.interfaces.buildqueue import IBuildQueueSet |
344 | +from lp.buildmaster.model.buildqueue import BuildQueue |
345 | from lp.services.job.interfaces.job import JobStatus |
346 | +from lp.soyuz.interfaces.build import IBuildSet |
347 | |
348 | |
349 | class BuildFarmJobBehaviorBase: |
350 | @@ -59,22 +64,101 @@ |
351 | The default behavior is that we don't add any extra values.""" |
352 | return {} |
353 | |
354 | + def _helpVerifyBuildIDComponent(self, raw_id, item_type, finder): |
355 | + """Helper for verifying parts of a `BuildFarmJob` name. |
356 | + |
357 | + Different `IBuildFarmJob` implementations can have different |
358 | + ways of constructing their identifying names. The names are |
359 | + produced by `IBuildFarmJob.getName` and verified by |
360 | + `IBuildFarmJobBehavior.verifySlaveBuildID`. |
361 | + |
362 | + This little helper makes it easier to verify an object id |
363 | + embedded in that name, check that it's a valid number, and |
364 | + retrieve the associated database object. |
365 | + |
366 | + :param raw_id: An unverified id string as extracted from the |
367 | + build name. The method will verify that it is a number, and |
368 | + try to retrieve the associated object. |
369 | + :param item_type: The type of object this id represents. Should |
370 | + be a class. |
371 | + :param finder: A function that, given an integral id, finds the |
372 | + associated object of type `item_type`. |
373 | + :raise CorruptBuildID: If `raw_id` is malformed in some way or |
374 | + the associated `item_type` object is not found. |
375 | + :return: An object that is an instance of `item_type`. |
376 | + """ |
377 | + type_name = item_type.__name__ |
378 | + try: |
379 | + numeric_id = int(raw_id) |
380 | + except ValueError: |
381 | + raise CorruptBuildID( |
382 | + "%s ID is not a number: '%s'" % (type_name, raw_id)) |
383 | + |
384 | + try: |
385 | + item = finder(numeric_id) |
386 | + except (NotFoundError, SQLObjectNotFound), reason: |
387 | + raise CorruptBuildID( |
388 | + "%s %d is not available: %s" % ( |
389 | + type_name, numeric_id, reason)) |
390 | + except Exception, reason: |
391 | + raise CorruptBuildID( |
392 | + "Error while looking up %s %d: %s" % ( |
393 | + type_name, numeric_id, reason)) |
394 | + |
395 | + if item is None: |
396 | + raise CorruptBuildID("There is no %s with id %d." % ( |
397 | + type_name, numeric_id)) |
398 | + |
399 | + assert zisinstance(item, item_type), ( |
400 | + "Looked for %s, but found %s." % (type_name, repr(item))) |
401 | + |
402 | + return item |
403 | + |
404 | + def getVerifiedBuild(self, raw_id): |
405 | + """Verify and retrieve the `Build` component of a slave build id. |
406 | + |
407 | + This does part of the verification for `verifySlaveBuildID`. |
408 | + |
409 | + By default, a `BuildFarmJob` has an identifying name of the form |
410 | + "b-q", where b is the id of its `Build` and q is the id of its |
411 | + `BuildQueue` record. |
412 | + |
413 | + Use `getVerifiedBuild` to verify the "b" part, and retrieve the |
414 | + associated `Build`. |
415 | + """ |
416 | + # Avoid circular import. |
417 | + from lp.soyuz.model.build import Build |
418 | + |
419 | + return self._helpVerifyBuildIDComponent( |
420 | + raw_id, Build, getUtility(IBuildSet).getByBuildID) |
421 | + |
422 | + def getVerifiedBuildQueue(self, raw_id): |
423 | + """Verify and retrieve the `BuildQueue` component of a slave build id. |
424 | + |
425 | + This does part of the verification for `verifySlaveBuildID`. |
426 | + |
427 | + By default, a `BuildFarmJob` has an identifying name of the form |
428 | + "b-q", where b is the id of its `Build` and q is the id of its |
429 | + `BuildQueue` record. |
430 | + |
431 | + Use `getVerifiedBuildQueue` to verify the "q" part, and retrieve |
432 | + the associated `BuildQueue` object. |
433 | + """ |
434 | + return self._helpVerifyBuildIDComponent( |
435 | + raw_id, BuildQueue, getUtility(IBuildQueueSet).get) |
436 | + |
437 | def verifySlaveBuildID(self, slave_build_id): |
438 | """See `IBuildFarmJobBehavior`.""" |
439 | # Extract information from the identifier. |
440 | try: |
441 | build_id, queue_item_id = slave_build_id.split('-') |
442 | - build_id = int(build_id) |
443 | - queue_item_id = int(queue_item_id) |
444 | except ValueError: |
445 | raise CorruptBuildID('Malformed build ID') |
446 | + |
447 | + build = self.getVerifiedBuild(build_id) |
448 | + queue_item = self.getVerifiedBuildQueue(queue_item_id) |
449 | |
450 | - try: |
451 | - queue_item = getUtility(IBuildQueueSet).get(queue_item_id) |
452 | - # Check whether build and buildqueue are properly related. |
453 | - except SQLObjectNotFound, reason: |
454 | - raise CorruptBuildID(str(reason)) |
455 | - if queue_item.specific_job.build.id != build_id: |
456 | + if build != queue_item.specific_job.build: |
457 | raise CorruptBuildID('Job build entry mismatch') |
458 | |
459 | def updateBuild(self, queueItem): |
460 | |
461 | === modified file 'lib/lp/buildmaster/tests/test_buildfarmjobbehavior.py' |
462 | --- lib/lp/buildmaster/tests/test_buildfarmjobbehavior.py 2010-03-06 04:57:40 +0000 |
463 | +++ lib/lp/buildmaster/tests/test_buildfarmjobbehavior.py 2010-03-15 21:11:07 +0000 |
464 | @@ -3,10 +3,18 @@ |
465 | |
466 | """Unit tests for BuildFarmJobBehaviorBase.""" |
467 | |
468 | -from unittest import TestCase, TestLoader |
469 | +from unittest import TestLoader |
470 | + |
471 | +from zope.component import getUtility |
472 | + |
473 | +from canonical.testing.layers import ZopelessDatabaseLayer |
474 | +from lp.testing import TestCaseWithFactory |
475 | |
476 | from lp.buildmaster.interfaces.buildbase import BuildStatus |
477 | +from lp.buildmaster.interfaces.builder import CorruptBuildID |
478 | from lp.buildmaster.model.buildfarmjobbehavior import BuildFarmJobBehaviorBase |
479 | +from lp.registry.interfaces.pocket import PackagePublishingPocket |
480 | +from lp.soyuz.interfaces.processor import IProcessorFamilySet |
481 | |
482 | |
483 | class FakeBuildFarmJob: |
484 | @@ -14,27 +22,91 @@ |
485 | pass |
486 | |
487 | |
488 | -class TestBuildFarmJobBehaviorBase(TestCase): |
489 | +class TestBuildFarmJobBehaviorBase(TestCaseWithFactory): |
490 | """Test very small, basic bits of BuildFarmJobBehaviorBase.""" |
491 | |
492 | - def setUp(self): |
493 | - self.behavior = BuildFarmJobBehaviorBase(FakeBuildFarmJob()) |
494 | + layer = ZopelessDatabaseLayer |
495 | + |
496 | + def _makeBehavior(self, buildfarmjob=None): |
497 | + """Create a `BuildFarmJobBehaviorBase`.""" |
498 | + if buildfarmjob is None: |
499 | + buildfarmjob = FakeBuildFarmJob() |
500 | + return BuildFarmJobBehaviorBase(buildfarmjob) |
501 | + |
502 | + def _makeBuild(self): |
503 | + """Create a `Build` object.""" |
504 | + x86 = getUtility(IProcessorFamilySet).getByName('x86') |
505 | + distroarchseries = self.factory.makeDistroArchSeries( |
506 | + architecturetag='x86', processorfamily=x86) |
507 | + distroseries = distroarchseries.distroseries |
508 | + archive = self.factory.makeArchive( |
509 | + distribution=distroseries.distribution) |
510 | + pocket = PackagePublishingPocket.RELEASE |
511 | + spr = self.factory.makeSourcePackageRelease( |
512 | + distroseries=distroseries, archive=archive) |
513 | + |
514 | + return spr.createBuild( |
515 | + distroarchseries=distroarchseries, pocket=pocket, archive=archive) |
516 | + |
517 | + def _makeBuildQueue(self): |
518 | + """Create a `BuildQueue` object.""" |
519 | + return self.factory.makeSourcePackageRecipeBuildJob() |
520 | |
521 | def test_extractBuildStatus_baseline(self): |
522 | # extractBuildStatus picks the name of the build status out of a |
523 | # dict describing the slave's status. |
524 | slave_status = {'build_status': 'BuildStatus.BUILDING'} |
525 | + behavior = self._makeBehavior() |
526 | self.assertEqual( |
527 | BuildStatus.BUILDING.name, |
528 | - self.behavior.extractBuildStatus(slave_status)) |
529 | + behavior.extractBuildStatus(slave_status)) |
530 | |
531 | def test_extractBuildStatus_malformed(self): |
532 | # extractBuildStatus errors out when the status string is not |
533 | # of the form it expects. |
534 | slave_status = {'build_status': 'BUILDING'} |
535 | - self.assertRaises( |
536 | - AssertionError, |
537 | - self.behavior.extractBuildStatus, slave_status) |
538 | + behavior = self._makeBehavior() |
539 | + self.assertRaises( |
540 | + AssertionError, behavior.extractBuildStatus, slave_status) |
541 | + |
542 | + def test_getVerifiedBuild_success(self): |
543 | + build = self._makeBuild() |
544 | + behavior = self._makeBehavior() |
545 | + raw_id = str(build.id) |
546 | + |
547 | + self.assertEqual(build, behavior.getVerifiedBuild(raw_id)) |
548 | + |
549 | + def test_getVerifiedBuild_malformed(self): |
550 | + behavior = self._makeBehavior() |
551 | + self.assertRaises(CorruptBuildID, behavior.getVerifiedBuild, 'hi!') |
552 | + |
553 | + def test_getVerifiedBuild_notfound(self): |
554 | + build = self._makeBuild() |
555 | + behavior = self._makeBehavior() |
556 | + nonexistent_id = str(build.id + 99) |
557 | + |
558 | + self.assertRaises( |
559 | + CorruptBuildID, behavior.getVerifiedBuild, nonexistent_id) |
560 | + |
561 | + def test_getVerifiedBuildQueue_success(self): |
562 | + buildqueue = self._makeBuildQueue() |
563 | + behavior = self._makeBehavior() |
564 | + raw_id = str(buildqueue.id) |
565 | + |
566 | + self.assertEqual(buildqueue, behavior.getVerifiedBuildQueue(raw_id)) |
567 | + |
568 | + def test_getVerifiedBuildQueue_malformed(self): |
569 | + behavior = self._makeBehavior() |
570 | + self.assertRaises( |
571 | + CorruptBuildID, behavior.getVerifiedBuildQueue, 'bye!') |
572 | + |
573 | + def test_getVerifiedBuildQueue_notfound(self): |
574 | + buildqueue = self._makeBuildQueue() |
575 | + behavior = self._makeBehavior() |
576 | + nonexistent_id = str(buildqueue.id + 99) |
577 | + |
578 | + self.assertRaises( |
579 | + CorruptBuildID, behavior.getVerifiedBuildQueue, nonexistent_id) |
580 | |
581 | |
582 | def test_suite(): |
583 | |
584 | === modified file 'lib/lp/code/model/recipebuilder.py' |
585 | --- lib/lp/code/model/recipebuilder.py 2010-02-05 15:06:28 +0000 |
586 | +++ lib/lp/code/model/recipebuilder.py 2010-03-15 21:11:07 +0000 |
587 | @@ -8,7 +8,7 @@ |
588 | 'RecipeBuildBehavior', |
589 | ] |
590 | |
591 | -from zope.component import adapts |
592 | +from zope.component import adapts, getUtility |
593 | from zope.interface import implements |
594 | |
595 | from lp.archiveuploader.permission import check_upload_to_pocket |
596 | @@ -18,7 +18,8 @@ |
597 | from lp.buildmaster.model.buildfarmjobbehavior import ( |
598 | BuildFarmJobBehaviorBase) |
599 | from lp.code.interfaces.sourcepackagerecipebuild import ( |
600 | - ISourcePackageRecipeBuildJob) |
601 | + ISourcePackageRecipeBuildJob, ISourcePackageRecipeBuildSource) |
602 | +from lp.code.model.sourcepackagerecipebuild import SourcePackageRecipeBuild |
603 | from lp.registry.interfaces.pocket import PackagePublishingPocket |
604 | from lp.soyuz.adapters.archivedependencies import ( |
605 | get_primary_current_component, get_sources_list_for_building) |
606 | @@ -168,3 +169,11 @@ |
607 | extra_info['logtail'] = raw_slave_status[2] |
608 | |
609 | return extra_info |
610 | + |
611 | + def getVerifiedBuild(self, raw_id): |
612 | + """See `IBuildFarmJobBehavior`.""" |
613 | + # This type of job has a build that is of type BuildBase but not |
614 | + # actually a Build. |
615 | + return self._helpVerifyBuildIDComponent( |
616 | + raw_id, SourcePackageRecipeBuild, |
617 | + getUtility(ISourcePackageRecipeBuildSource).getById) |
618 | |
619 | === removed file 'lib/lp/scripts/utilities/apiindex.py' |
620 | --- lib/lp/scripts/utilities/apiindex.py 2009-06-30 16:56:07 +0000 |
621 | +++ lib/lp/scripts/utilities/apiindex.py 1970-01-01 00:00:00 +0000 |
622 | @@ -1,21 +0,0 @@ |
623 | -# Copyright 2009 Canonical Ltd. This software is licensed under the |
624 | -# GNU Affero General Public License version 3 (see the file LICENSE). |
625 | - |
626 | -"""A script to build reference html from wadl. |
627 | - |
628 | -Uses an xsl from the launchpadlib package.""" |
629 | - |
630 | -__metaclass__ = type |
631 | -__all__ = ['main'] |
632 | - |
633 | - |
634 | -import pkg_resources |
635 | -import subprocess |
636 | -import sys |
637 | - |
638 | -def main(): |
639 | - source = sys.argv[1] |
640 | - stylesheet = pkg_resources.resource_filename( |
641 | - 'launchpadlib', 'wadl-to-refhtml.xsl') |
642 | - subprocess.call(['xsltproc', stylesheet, source]) |
643 | - pkg_resources.cleanup_resources() |
644 | |
645 | === modified file 'lib/lp/soyuz/doc/buildd-slavescanner.txt' |
646 | --- lib/lp/soyuz/doc/buildd-slavescanner.txt 2010-03-10 12:50:18 +0000 |
647 | +++ lib/lp/soyuz/doc/buildd-slavescanner.txt 2010-03-15 21:11:07 +0000 |
648 | @@ -95,15 +95,16 @@ |
649 | >>> lostbuilding_builder = MockBuilder( |
650 | ... 'Lost Building Slave', LostBuildingSlave()) |
651 | >>> buildergroup.rescueBuilderIfLost(lostbuilding_builder) |
652 | - WARNING:root:Builder 'Lost Building Slave' rescued from |
653 | - '1000-10000': 'Object not found' |
654 | + WARNING:root:Builder 'Lost Building Slave' rescued from '1000-10000': |
655 | + 'Build 1000 is not available: 'Object not found'' |
656 | |
657 | Then a lost slave in status 'WAITING': |
658 | |
659 | >>> lostwaiting_builder = MockBuilder( |
660 | ... 'Lost Waiting Slave', LostWaitingSlave()) |
661 | >>> buildergroup.rescueBuilderIfLost(lostwaiting_builder) |
662 | - WARNING:root:Builder 'Lost Waiting Slave' rescued from '1000-10000': 'Object not found' |
663 | + WARNING:root:Builder 'Lost Waiting Slave' rescued from '1000-10000': |
664 | + 'Build 1000 is not available: 'Object not found'' |
665 | |
666 | Both got rescued, as expected. |
667 | |
668 | |
669 | === modified file 'lib/lp/testing/factory.py' |
670 | --- lib/lp/testing/factory.py 2010-03-11 20:54:36 +0000 |
671 | +++ lib/lp/testing/factory.py 2010-03-15 21:11:07 +0000 |
672 | @@ -2008,6 +2008,81 @@ |
673 | def getAnySourcePackageUrgency(self): |
674 | return SourcePackageUrgency.MEDIUM |
675 | |
676 | + def makeSourcePackageRelease(self, archive=None, sourcepackagename=None, |
677 | + distroseries=None, maintainer=None, |
678 | + creator=None, component=None, section=None, |
679 | + urgency=None, version=None, |
680 | + builddepends=None, builddependsindep=None, |
681 | + build_conflicts=None, |
682 | + build_conflicts_indep=None, |
683 | + architecturehintlist='all', |
684 | + dsc_maintainer_rfc822=None, |
685 | + dsc_standards_version='3.6.2', |
686 | + dsc_format='1.0', dsc_binaries='foo-bin', |
687 | + date_uploaded=UTC_NOW): |
688 | + """Make a `SourcePackageRelease`.""" |
689 | + if distroseries is None: |
690 | + if archive is None: |
691 | + distribution = None |
692 | + else: |
693 | + distribution = archive.distribution |
694 | + distroseries = self.makeDistroRelease(distribution=distribution) |
695 | + |
696 | + if archive is None: |
697 | + archive = self.makeArchive( |
698 | + distribution=distroseries.distribution, |
699 | + purpose=ArchivePurpose.PRIMARY) |
700 | + |
701 | + if sourcepackagename is None: |
702 | + sourcepackagename = self.makeSourcePackageName() |
703 | + |
704 | + if component is None: |
705 | + component = self.makeComponent() |
706 | + |
707 | + if urgency is None: |
708 | + urgency = self.getAnySourcePackageUrgency() |
709 | + |
710 | + if section is None: |
711 | + section = self.getUniqueString('section') |
712 | + section = getUtility(ISectionSet).ensure(section) |
713 | + |
714 | + if maintainer is None: |
715 | + maintainer = self.makePerson() |
716 | + |
717 | + maintainer_email = '%s <%s>' % ( |
718 | + maintainer.displayname, |
719 | + maintainer.preferredemail.email) |
720 | + |
721 | + if creator is None: |
722 | + creator = self.makePerson() |
723 | + |
724 | + if version is None: |
725 | + version = self.getUniqueString('version') |
726 | + |
727 | + return distroseries.createUploadedSourcePackageRelease( |
728 | + sourcepackagename=sourcepackagename, |
729 | + maintainer=maintainer, |
730 | + creator=creator, |
731 | + component=component, |
732 | + section=section, |
733 | + urgency=urgency, |
734 | + version=version, |
735 | + builddepends=builddepends, |
736 | + builddependsindep=builddependsindep, |
737 | + build_conflicts=build_conflicts, |
738 | + build_conflicts_indep=build_conflicts_indep, |
739 | + architecturehintlist=architecturehintlist, |
740 | + changelog_entry=None, |
741 | + dsc=None, |
742 | + copyright=self.getUniqueString(), |
743 | + dscsigningkey=None, |
744 | + dsc_maintainer_rfc822=maintainer_email, |
745 | + dsc_standards_version=dsc_standards_version, |
746 | + dsc_format=dsc_format, |
747 | + dsc_binaries=dsc_binaries, |
748 | + archive=archive, |
749 | + dateuploaded=date_uploaded) |
750 | + |
751 | def makeSourcePackagePublishingHistory(self, sourcepackagename=None, |
752 | distroseries=None, maintainer=None, |
753 | creator=None, component=None, |
754 | @@ -2027,54 +2102,31 @@ |
755 | dsc_format='1.0', |
756 | dsc_binaries='foo-bin', |
757 | ): |
758 | - if sourcepackagename is None: |
759 | - sourcepackagename = self.makeSourcePackageName() |
760 | - spn = sourcepackagename |
761 | - |
762 | + """Make a `SourcePackagePublishingHistory`.""" |
763 | if distroseries is None: |
764 | - distroseries = self.makeDistroRelease() |
765 | + if archive is None: |
766 | + distribution = None |
767 | + else: |
768 | + distribution = archive.distribution |
769 | + distroseries = self.makeDistroRelease(distribution=distribution) |
770 | |
771 | if archive is None: |
772 | archive = self.makeArchive( |
773 | distribution=distroseries.distribution, |
774 | purpose=ArchivePurpose.PRIMARY) |
775 | |
776 | - if component is None: |
777 | - component = self.makeComponent() |
778 | - |
779 | if pocket is None: |
780 | pocket = self.getAnyPocket() |
781 | |
782 | if status is None: |
783 | status = PackagePublishingStatus.PENDING |
784 | |
785 | - if urgency is None: |
786 | - urgency = self.getAnySourcePackageUrgency() |
787 | - |
788 | - if section is None: |
789 | - section = self.getUniqueString('section') |
790 | - section = getUtility(ISectionSet).ensure(section) |
791 | - |
792 | - if maintainer is None: |
793 | - maintainer = self.makePerson() |
794 | - |
795 | - maintainer_email = '%s <%s>' % ( |
796 | - maintainer.displayname, |
797 | - maintainer.preferredemail.email) |
798 | - |
799 | - if creator is None: |
800 | - creator = self.makePerson() |
801 | - |
802 | - if version is None: |
803 | - version = self.getUniqueString('version') |
804 | - |
805 | - gpg_key = self.makeGPGKey(creator) |
806 | - |
807 | - spr = distroseries.createUploadedSourcePackageRelease( |
808 | - sourcepackagename=spn, |
809 | + spr = self.makeSourcePackageRelease( |
810 | + archive=archive, |
811 | + sourcepackagename=sourcepackagename, |
812 | + distroseries=distroseries, |
813 | maintainer=maintainer, |
814 | - creator=creator, |
815 | - component=component, |
816 | + creator=creator, component=component, |
817 | section=section, |
818 | urgency=urgency, |
819 | version=version, |
820 | @@ -2083,15 +2135,10 @@ |
821 | build_conflicts=build_conflicts, |
822 | build_conflicts_indep=build_conflicts_indep, |
823 | architecturehintlist=architecturehintlist, |
824 | - changelog_entry=None, |
825 | - dsc=None, |
826 | - copyright=self.getUniqueString(), |
827 | - dscsigningkey=gpg_key, |
828 | - dsc_maintainer_rfc822=maintainer_email, |
829 | dsc_standards_version=dsc_standards_version, |
830 | dsc_format=dsc_format, |
831 | dsc_binaries=dsc_binaries, |
832 | - archive=archive, dateuploaded=date_uploaded) |
833 | + date_uploaded=date_uploaded) |
834 | |
835 | sspph = SourcePackagePublishingHistory( |
836 | distroseries=distroseries, |
837 | |
838 | === modified file 'lib/lp/translations/doc/translationtemplatesbuildbehavior.txt' |
839 | --- lib/lp/translations/doc/translationtemplatesbuildbehavior.txt 2010-03-05 13:52:32 +0000 |
840 | +++ lib/lp/translations/doc/translationtemplatesbuildbehavior.txt 2010-03-15 21:11:07 +0000 |
841 | @@ -2,13 +2,21 @@ |
842 | |
843 | == Setup == |
844 | |
845 | -Set up build environment. |
846 | +Set up build environment. Clear out the build queue. |
847 | |
848 | >>> import transaction |
849 | >>> import logging |
850 | >>> logger = logging.getLogger() |
851 | >>> logger.setLevel(logging.CRITICAL) |
852 | |
853 | + >>> from canonical.database.sqlbase import quote |
854 | + >>> from canonical.launchpad.interfaces.lpstorm import IMasterStore |
855 | + >>> from lp.services.job.interfaces.job import JobStatus |
856 | + >>> from lp.services.job.model.job import Job |
857 | + >>> store = IMasterStore(Job) |
858 | + >>> query = store.execute("UPDATE Job SET status = %s" % quote( |
859 | + ... JobStatus.FAILED)) |
860 | + |
861 | >>> from lp.buildmaster.master import BuilddMaster |
862 | >>> from canonical.buildd.tests import BuilddSlaveTestSetup |
863 | >>> bm = BuilddMaster(logger, transaction) |
864 | @@ -39,7 +47,21 @@ |
865 | |
866 | Make a builder to process our build request. |
867 | |
868 | - >>> builder = factory.makeBuilder(virtualized=False, processor=processor) |
869 | + >>> builder = factory.makeBuilder( |
870 | + ... virtualized=True, processor=processor, vm_host='hostname') |
871 | + |
872 | +The builder doesn't talk to a real slave. We don't have those in our |
873 | +test suite. But we give it a fake one. |
874 | + |
875 | + >>> from lp.testing.fakemethod import FakeMethod |
876 | + >>> class FakeSlave: |
877 | + ... build = FakeMethod() |
878 | + ... cacheFile = FakeMethod() |
879 | + ... resume = FakeMethod(result=('Output here', 'Errors here', 0)) |
880 | + |
881 | + >>> from zope.security.proxy import removeSecurityProxy |
882 | + >>> slave = FakeSlave() |
883 | + >>> removeSecurityProxy(builder).slave = slave |
884 | |
885 | |
886 | == Get a job! == |
887 | @@ -60,20 +82,27 @@ |
888 | >>> print buildqueue.date_started |
889 | None |
890 | |
891 | -Dispatch the job to the build slave. The proper method to call here |
892 | -would have been Builder.findAndStartJob, but it has not been generalised |
893 | -to handle new job types yet. |
894 | - |
895 | -# XXX JeroenVermeulen bug=506617: call findAndStartJob instead. |
896 | - |
897 | - >>> from zope.security.proxy import removeSecurityProxy |
898 | - >>> removeSecurityProxy(builder)._dispatchBuildCandidate(buildqueue) |
899 | - |
900 | -The build is now marked as started. |
901 | +Our job is now first in line to be executed. |
902 | + |
903 | + >>> removeSecurityProxy(builder)._findBuildCandidate() == buildqueue |
904 | + True |
905 | + |
906 | +Dispatch the first job in line (ours!) to the build slave. |
907 | + |
908 | + >>> activated_job = builder.findAndStartJob() |
909 | + >>> activated_job == buildqueue |
910 | + True |
911 | + |
912 | +Our build is now marked as started. |
913 | |
914 | >>> buildqueue.date_started is None |
915 | False |
916 | |
917 | +The slave's build method has been called. |
918 | + |
919 | + >>> slave.build.call_count |
920 | + 1 |
921 | + |
922 | |
923 | == Teardown == |
924 | |
925 | |
926 | === modified file 'lib/lp/translations/model/translationtemplatesbuildbehavior.py' |
927 | --- lib/lp/translations/model/translationtemplatesbuildbehavior.py 2010-02-11 19:11:11 +0000 |
928 | +++ lib/lp/translations/model/translationtemplatesbuildbehavior.py 2010-03-15 21:11:07 +0000 |
929 | @@ -17,6 +17,7 @@ |
930 | |
931 | from canonical.launchpad.interfaces import ILaunchpadCelebrities |
932 | |
933 | +from lp.buildmaster.interfaces.builder import CorruptBuildID |
934 | from lp.buildmaster.interfaces.buildfarmjobbehavior import ( |
935 | IBuildFarmJobBehavior) |
936 | from lp.buildmaster.model.buildfarmjobbehavior import ( |
937 | @@ -40,12 +41,27 @@ |
938 | self._builder.slave.cacheFile(logger, chroot) |
939 | buildid = self.buildfarmjob.getName() |
940 | |
941 | - args = { 'branch_url': self.buildfarmjob.branch.url } |
942 | + args = self.buildfarmjob.metadata |
943 | filemap = {} |
944 | |
945 | self._builder.slave.build( |
946 | buildid, self.build_type, chroot_sha1, filemap, args) |
947 | |
948 | + def verifySlaveBuildID(self, slave_build_id): |
949 | + """See `IBuildFarmJobBehavior`.""" |
950 | + try: |
951 | + branch_name, queue_item_id = slave_build_id.rsplit('-', 1) |
952 | + except ValueError: |
953 | + raise CorruptBuildID( |
954 | + "Malformed translation templates build id: '%s'" % ( |
955 | + slave_build_id)) |
956 | + |
957 | + buildqueue = self.getVerifiedBuildQueue(queue_item_id) |
958 | + if buildqueue.job != self.buildfarmjob.job: |
959 | + raise CorruptBuildID( |
960 | + "ID mismatch for translation templates build '%s'" % ( |
961 | + slave_build_id)) |
962 | + |
963 | def _getChroot(self): |
964 | ubuntu = getUtility(ILaunchpadCelebrities).ubuntu |
965 | return ubuntu.currentseries.nominatedarchindep.getChroot() |
966 | |
967 | === modified file 'lib/lp/translations/tests/test_translationtemplatesbuildbehavior.py' |
968 | --- lib/lp/translations/tests/test_translationtemplatesbuildbehavior.py 2010-03-12 08:31:43 +0000 |
969 | +++ lib/lp/translations/tests/test_translationtemplatesbuildbehavior.py 2010-03-15 21:11:07 +0000 |
970 | @@ -19,6 +19,7 @@ |
971 | from lp.buildmaster.interfaces.buildbase import BuildStatus |
972 | from lp.buildmaster.interfaces.buildfarmjobbehavior import ( |
973 | IBuildFarmJobBehavior) |
974 | +from lp.buildmaster.interfaces.builder import CorruptBuildID |
975 | from lp.buildmaster.interfaces.buildqueue import IBuildQueueSet |
976 | from lp.testing import TestCaseWithFactory |
977 | from lp.testing.fakemethod import FakeMethod |
978 | @@ -123,6 +124,10 @@ |
979 | self.assertEqual( |
980 | 'translation-templates', slave_status['test_build_type']) |
981 | self.assertIn('branch_url', slave_status['test_build_args']) |
982 | + # The slave receives the public http URL for the branch. |
983 | + self.assertEqual( |
984 | + behavior.buildfarmjob.branch.composePublicURL(), |
985 | + slave_status['test_build_args']['branch_url']) |
986 | |
987 | def test_getChroot(self): |
988 | # _getChroot produces the current chroot for the current Ubuntu |
989 | @@ -170,6 +175,38 @@ |
990 | self.assertEqual(1, builder.cleanSlave.call_count) |
991 | self.assertEqual(0, behavior._uploadTarball.call_count) |
992 | |
993 | + def test_verifySlaveBuildID_success(self): |
994 | + # TranslationTemplatesBuildJob.getName generates slave build ids |
995 | + # that TranslationTemplatesBuildBehavior.verifySlaveBuildID |
996 | + # accepts. |
997 | + behavior = self._makeBehavior() |
998 | + buildfarmjob = behavior.buildfarmjob |
999 | + job = buildfarmjob.job |
1000 | + |
1001 | + # The test is that this not raise CorruptBuildID (or anything |
1002 | + # else, for that matter). |
1003 | + behavior.verifySlaveBuildID(behavior.buildfarmjob.getName()) |
1004 | + |
1005 | + def test_verifySlaveBuildID_handles_dashes(self): |
1006 | + # TranslationTemplatesBuildBehavior.verifySlaveBuildID can deal |
1007 | + # with dashes in branch names. |
1008 | + behavior = self._makeBehavior() |
1009 | + buildfarmjob = behavior.buildfarmjob |
1010 | + job = buildfarmjob.job |
1011 | + buildfarmjob.branch.name = 'x-y-z--' |
1012 | + |
1013 | + # The test is that this not raise CorruptBuildID (or anything |
1014 | + # else, for that matter). |
1015 | + behavior.verifySlaveBuildID(behavior.buildfarmjob.getName()) |
1016 | + |
1017 | + def test_verifySlaveBuildID_malformed(self): |
1018 | + behavior = self._makeBehavior() |
1019 | + self.assertRaises(CorruptBuildID, behavior.verifySlaveBuildID, 'huh?') |
1020 | + |
1021 | + def test_verifySlaveBuildID_notfound(self): |
1022 | + behavior = self._makeBehavior() |
1023 | + self.assertRaises(CorruptBuildID, behavior.verifySlaveBuildID, '1-1') |
1024 | + |
1025 | |
1026 | def test_suite(): |
1027 | return TestLoader().loadTestsFromName(__name__) |
1028 | |
1029 | === added file 'utilities/apidoc-index.pt' |
1030 | --- utilities/apidoc-index.pt 1970-01-01 00:00:00 +0000 |
1031 | +++ utilities/apidoc-index.pt 2010-03-15 21:11:07 +0000 |
1032 | @@ -0,0 +1,37 @@ |
1033 | +<html |
1034 | + xmlns="http://www.w3.org/1999/xhtml" |
1035 | + xmlns:tal="http://xml.zope.org/namespaces/tal" |
1036 | + xmlns:metal="http://xml.zope.org/namespaces/metal" |
1037 | + xmlns:i18n="http://xml.zope.org/namespaces/i18n" |
1038 | + i18n:domain="launchpad" |
1039 | +> |
1040 | + <head> |
1041 | + <title>Launchpad Web Service API</title> |
1042 | + </head> |
1043 | + |
1044 | + <body> |
1045 | + |
1046 | + <h1>Launchpad web service API documentation</h1> |
1047 | + |
1048 | + <p tal:content="structure config/service_description"> |
1049 | + Description of the Launchpad web service. |
1050 | + </p> |
1051 | + |
1052 | + <h2>Active versions</h2> |
1053 | + |
1054 | + <ul> |
1055 | + <li tal:repeat="version_and_description versions_and_descriptions"> |
1056 | + |
1057 | + <tal:version tal:define="version version_and_description/version; |
1058 | + description version_and_description/description"> |
1059 | + |
1060 | + <a tal:attributes="href string:$version.html" |
1061 | + tal:content="version">Version name</a>: |
1062 | + <tal:description replace="description" /> |
1063 | + </tal:version> |
1064 | + </li> |
1065 | + </ul> |
1066 | + |
1067 | + </body> |
1068 | + |
1069 | +</html> |
1070 | |
1071 | === renamed file 'utilities/create-lp-wadl.py' => 'utilities/create-lp-wadl-and-apidoc.py' |
1072 | --- utilities/create-lp-wadl.py 2010-03-01 21:16:06 +0000 |
1073 | +++ utilities/create-lp-wadl-and-apidoc.py 2010-03-15 21:11:07 +0000 |
1074 | @@ -12,10 +12,15 @@ |
1075 | |
1076 | import _pythonpath |
1077 | |
1078 | +from cStringIO import StringIO |
1079 | +import os |
1080 | +import pkg_resources |
1081 | +import subprocess |
1082 | import sys |
1083 | import urlparse |
1084 | |
1085 | from zope.component import getUtility |
1086 | +from zope.pagetemplate.pagetemplatefile import PageTemplateFile |
1087 | |
1088 | from canonical.launchpad.ftests import login, ANONYMOUS |
1089 | from canonical.launchpad.scripts import execute_zcml_for_scripts |
1090 | @@ -30,6 +35,9 @@ |
1091 | execute_zcml_for_scripts() |
1092 | config = getUtility(IWebServiceConfiguration) |
1093 | |
1094 | + stylesheet = pkg_resources.resource_filename( |
1095 | + 'launchpadlib', 'wadl-to-refhtml.xsl') |
1096 | + |
1097 | # Request the WADL from the root resource. |
1098 | # We do this by creating a request object asking for a WADL |
1099 | # representation. |
1100 | @@ -52,6 +60,33 @@ |
1101 | content = request.publication.getApplication(request)(request) |
1102 | f.write(content) |
1103 | f.close() |
1104 | + |
1105 | + # Now, convert the WADL into an human-readable description and |
1106 | + # put the HTML in the same directory as the WADL. |
1107 | + directory, ignore = os.path.split(path_template) |
1108 | + html_filename = os.path.join(directory, version + ".html") |
1109 | + print "Writing apidoc for version %s to %s" % ( |
1110 | + version, html_filename) |
1111 | + stdout = open(html_filename, "w") |
1112 | + subprocess.Popen(['xsltproc', stylesheet, filename], stdout=stdout) |
1113 | + stdout.close() |
1114 | + |
1115 | + # Finally, create an index.html with links to all the HTML |
1116 | + # documentation files we just generated. |
1117 | + template_file = 'apidoc-index.pt' |
1118 | + template = PageTemplateFile(template_file) |
1119 | + namespace = template.pt_getContext() |
1120 | + namespace['config'] = config |
1121 | + versions_and_descriptions = [] |
1122 | + for version in config.active_versions: |
1123 | + versions_and_descriptions.append( |
1124 | + dict(version=version, |
1125 | + description=config.version_descriptions[version])) |
1126 | + namespace['versions_and_descriptions'] = versions_and_descriptions |
1127 | + |
1128 | + f = open(os.path.join(directory, "index.html"), 'w') |
1129 | + f.write(template.pt_render(namespace)) |
1130 | + |
1131 | return 0 |
1132 | |
1133 | if __name__ == '__main__': |
1134 | |
1135 | === modified file 'versions.cfg' |
1136 | --- versions.cfg 2010-03-08 22:26:29 +0000 |
1137 | +++ versions.cfg 2010-03-15 21:11:07 +0000 |
1138 | @@ -21,15 +21,15 @@ |
1139 | grokcore.component = 1.6 |
1140 | httplib2 = 0.6.0 |
1141 | ipython = 0.9.1 |
1142 | -launchpadlib = 1.5.5 |
1143 | +launchpadlib = 1.5.6 |
1144 | lazr.authentication = 0.1.1 |
1145 | lazr.batchnavigator = 1.1 |
1146 | lazr.config = 1.1.3 |
1147 | lazr.delegates = 1.1.0 |
1148 | lazr.enum = 1.1.2 |
1149 | lazr.lifecycle = 1.1 |
1150 | -lazr.restful = 0.9.21 |
1151 | -lazr.restfulclient = 0.9.11 |
1152 | +lazr.restful = 0.9.23 |
1153 | +lazr.restfulclient = 0.9.12 |
1154 | lazr.smtptest = 1.1 |
1155 | lazr.testing = 0.1.1 |
1156 | lazr.uri = 1.0.2 |
The diff is screwed up due to the precursor branch being out-of-date, but here's a paste: http:// pastebin. ubuntu. com/393568/