Merge ~andrey-fedoseev/launchpad:jira-bug-watch into launchpad:master
- Git
- lp:~andrey-fedoseev/launchpad
- jira-bug-watch
- Merge into master
Proposed by
Andrey Fedoseev
Status: | Needs review |
---|---|
Proposed branch: | ~andrey-fedoseev/launchpad:jira-bug-watch |
Merge into: | launchpad:master |
Diff against target: |
873 lines (+742/-5) 9 files modified
lib/lp/bugs/externalbugtracker/__init__.py (+2/-0) lib/lp/bugs/externalbugtracker/base.py (+8/-3) lib/lp/bugs/externalbugtracker/jira.py (+268/-0) lib/lp/bugs/externalbugtracker/tests/test_externalbugtracker.py (+1/-1) lib/lp/bugs/externalbugtracker/tests/test_jira.py (+432/-0) lib/lp/bugs/interfaces/bugtracker.py (+9/-0) lib/lp/bugs/model/bugwatch.py (+11/-0) lib/lp/bugs/tests/test_bugwatch.py (+9/-0) lib/lp/services/config/schema-lazr.conf (+2/-1) |
Related bugs: |
Reviewer | Review Type | Date Requested | Status |
---|---|---|---|
Colin Watson (community) | Approve | ||
Review via email: mp+433103@code.launchpad.net |
Commit message
Add external bug tracker for JIRA
Description of the change
To post a comment you must log in.
Unmerged commits
- 9f871dc... by Andrey Fedoseev
-
Add external bug tracker for JIRA
-
docs:0 (build) lint:0 (build) mypy:0 (build) 1 → 3 of 3 results First • Previous • Next • Last
Preview Diff
[H/L] Next/Prev Comment, [J/K] Next/Prev File, [N/P] Next/Prev Hunk
1 | diff --git a/lib/lp/bugs/externalbugtracker/__init__.py b/lib/lp/bugs/externalbugtracker/__init__.py |
2 | index 65c5333..d4b887d 100644 |
3 | --- a/lib/lp/bugs/externalbugtracker/__init__.py |
4 | +++ b/lib/lp/bugs/externalbugtracker/__init__.py |
5 | @@ -51,6 +51,7 @@ from lp.bugs.externalbugtracker.bugzilla import Bugzilla |
6 | from lp.bugs.externalbugtracker.debbugs import DebBugs, DebBugsDatabaseNotFound |
7 | from lp.bugs.externalbugtracker.github import GitHub |
8 | from lp.bugs.externalbugtracker.gitlab import GitLab |
9 | +from lp.bugs.externalbugtracker.jira import Jira |
10 | from lp.bugs.externalbugtracker.mantis import Mantis |
11 | from lp.bugs.externalbugtracker.roundup import Roundup |
12 | from lp.bugs.externalbugtracker.rt import RequestTracker |
13 | @@ -68,6 +69,7 @@ BUG_TRACKER_CLASSES = { |
14 | BugTrackerType.ROUNDUP: Roundup, |
15 | BugTrackerType.RT: RequestTracker, |
16 | BugTrackerType.SOURCEFORGE: SourceForge, |
17 | + BugTrackerType.JIRA: Jira, |
18 | } |
19 | |
20 | |
21 | diff --git a/lib/lp/bugs/externalbugtracker/base.py b/lib/lp/bugs/externalbugtracker/base.py |
22 | index ee7be38..0d2efd4 100644 |
23 | --- a/lib/lp/bugs/externalbugtracker/base.py |
24 | +++ b/lib/lp/bugs/externalbugtracker/base.py |
25 | @@ -279,16 +279,17 @@ class ExternalBugTracker: |
26 | except requests.RequestException as e: |
27 | raise BugTrackerConnectError(self.baseurl, e) |
28 | |
29 | - def _postPage(self, page, form, repost_on_redirect=False): |
30 | + def _postPage(self, page, data, repost_on_redirect=False, json=False): |
31 | """POST to the specified page and form. |
32 | |
33 | - :param form: is a dict of form variables being POSTed. |
34 | + :param data: is a dict of form variables being POSTed. |
35 | :param repost_on_redirect: override RFC-compliant redirect handling. |
36 | By default, if the POST receives a redirect response, the |
37 | request to the redirection's target URL will be a GET. If |
38 | `repost_on_redirect` is True, this method will do a second POST |
39 | instead. Do this only if you are sure that repeated POST to |
40 | this page is safe, as is usually the case with search forms. |
41 | + :param json: if True, the data will be JSON encoded. |
42 | :return: A `requests.Response` object. |
43 | """ |
44 | hooks = ( |
45 | @@ -301,8 +302,12 @@ class ExternalBugTracker: |
46 | if not url.endswith("/"): |
47 | url += "/" |
48 | url = urljoin(url, page) |
49 | + if json: |
50 | + kwargs = {"json": data} |
51 | + else: |
52 | + kwargs = {"data": data} |
53 | response = self.makeRequest( |
54 | - "POST", url, headers=self._getHeaders(), data=form, hooks=hooks |
55 | + "POST", url, headers=self._getHeaders(), hooks=hooks, **kwargs |
56 | ) |
57 | raise_for_status_redacted(response) |
58 | return response |
59 | diff --git a/lib/lp/bugs/externalbugtracker/jira.py b/lib/lp/bugs/externalbugtracker/jira.py |
60 | new file mode 100644 |
61 | index 0000000..b529c5b |
62 | --- /dev/null |
63 | +++ b/lib/lp/bugs/externalbugtracker/jira.py |
64 | @@ -0,0 +1,268 @@ |
65 | +# Copyright 2022 Canonical Ltd. This software is licensed under the |
66 | +# GNU Affero General Public License version 3 (see the file LICENSE). |
67 | + |
68 | +"""Jira ExternalBugTracker utility.""" |
69 | + |
70 | +__all__ = [ |
71 | + "Jira", |
72 | + "JiraCredentials", |
73 | + "JiraBug", |
74 | + "JiraStatus", |
75 | + "JiraPriority", |
76 | +] |
77 | + |
78 | +import base64 |
79 | +import datetime |
80 | +from enum import Enum |
81 | +from typing import Dict, Iterable, NamedTuple, Optional, Tuple |
82 | +from urllib.parse import urlunsplit |
83 | + |
84 | +import dateutil.parser |
85 | + |
86 | +from lp.bugs.externalbugtracker import ( |
87 | + BugTrackerConnectError, |
88 | + ExternalBugTracker, |
89 | +) |
90 | +from lp.bugs.interfaces.bugtask import BugTaskImportance, BugTaskStatus |
91 | +from lp.services.config import config |
92 | +from lp.services.webapp.url import urlsplit |
93 | + |
94 | +JiraCredentials = NamedTuple( |
95 | + "JiraCredentials", |
96 | + ( |
97 | + ("username", str), |
98 | + ("password", str), |
99 | + ), |
100 | +) |
101 | + |
102 | + |
103 | +class JiraStatus(Enum): |
104 | + |
105 | + UNDEFINED = "undefined" |
106 | + NEW = "new" |
107 | + INDETERMINATE = "indeterminate" |
108 | + DONE = "done" |
109 | + |
110 | + @property |
111 | + def launchpad_status(self): |
112 | + if self == JiraStatus.UNDEFINED: |
113 | + return BugTaskStatus.UNKNOWN |
114 | + elif self == JiraStatus.NEW: |
115 | + return BugTaskStatus.NEW |
116 | + elif self == JiraStatus.INDETERMINATE: |
117 | + return BugTaskStatus.INPROGRESS |
118 | + elif self == JiraStatus.DONE: |
119 | + return BugTaskStatus.FIXRELEASED |
120 | + else: |
121 | + raise AssertionError() |
122 | + |
123 | + |
124 | +class JiraPriority(Enum): |
125 | + |
126 | + UNDEFINED = "undefined" |
127 | + LOWEST = "Lowest" |
128 | + LOW = "Low" |
129 | + MEDIUM = "Medium" |
130 | + HIGH = "High" |
131 | + HIGHEST = "Highest" |
132 | + |
133 | + @property |
134 | + def launchpad_importance(self): |
135 | + if self == JiraPriority.UNDEFINED: |
136 | + return BugTaskImportance.UNKNOWN |
137 | + elif self == JiraPriority.LOWEST: |
138 | + return BugTaskImportance.WISHLIST |
139 | + elif self == JiraPriority.LOW: |
140 | + return BugTaskImportance.LOW |
141 | + elif self == JiraPriority.MEDIUM: |
142 | + return BugTaskImportance.MEDIUM |
143 | + elif self == JiraPriority.HIGH: |
144 | + return BugTaskImportance.HIGH |
145 | + elif self == JiraPriority.HIGHEST: |
146 | + return BugTaskImportance.CRITICAL |
147 | + else: |
148 | + raise AssertionError() |
149 | + |
150 | + |
151 | +class JiraBug: |
152 | + def __init__(self, key: str, status: JiraStatus, priority: JiraPriority): |
153 | + self.key = key |
154 | + self.status = status |
155 | + self.priority = priority |
156 | + |
157 | + @classmethod |
158 | + def from_api_data(cls, bug_data) -> "JiraBug": |
159 | + try: |
160 | + status = JiraStatus( |
161 | + bug_data["fields"]["status"]["statusCategory"]["key"] |
162 | + ) |
163 | + except ValueError: |
164 | + status = JiraStatus.UNDEFINED |
165 | + |
166 | + try: |
167 | + priority = JiraPriority(bug_data["fields"]["priority"]["name"]) |
168 | + except ValueError: |
169 | + priority = JiraPriority.UNDEFINED |
170 | + |
171 | + return cls( |
172 | + key=bug_data["key"], |
173 | + status=status, |
174 | + priority=priority, |
175 | + ) |
176 | + |
177 | + def __eq__(self, other): |
178 | + if not isinstance(other, JiraBug): |
179 | + raise ValueError() |
180 | + return ( |
181 | + self.key == other.key |
182 | + and self.status == other.status |
183 | + and self.priority == other.priority |
184 | + ) |
185 | + |
186 | + |
187 | +class Jira(ExternalBugTracker): |
188 | + """An `ExternalBugTracker` for dealing with Jira issues.""" |
189 | + |
190 | + batch_query_threshold = 0 # Always use the batch method. |
191 | + |
192 | + def __init__(self, baseurl): |
193 | + _, host, path, query, fragment = urlsplit(baseurl) |
194 | + path = "/rest/api/2/" |
195 | + baseurl = urlunsplit(("https", host, path, "", "")) |
196 | + super().__init__(baseurl) |
197 | + self.cached_bugs = {} # type: Dict[str, Optional[JiraBug]] |
198 | + |
199 | + @property |
200 | + def credentials(self) -> Optional[JiraCredentials]: |
201 | + credentials_config = config["checkwatches.credentials"] |
202 | + # lazr.config.Section doesn't support get(). |
203 | + try: |
204 | + username = credentials_config["{}.username".format(self.basehost)] |
205 | + password = credentials_config["{}.password".format(self.basehost)] |
206 | + return JiraCredentials( |
207 | + username=username, |
208 | + password=password, |
209 | + ) |
210 | + except KeyError: |
211 | + return |
212 | + |
213 | + def getCurrentDBTime(self): |
214 | + # See https://docs.atlassian.com/software/jira/docs/api/REST/9.3.1/#api/2/serverInfo-getServerInfo # noqa |
215 | + response_data = self._getPage("serverInfo").json() |
216 | + return dateutil.parser.parse(response_data["serverTime"]).astimezone( |
217 | + datetime.timezone.utc |
218 | + ) |
219 | + |
220 | + def getModifiedRemoteBugs(self, bug_ids, last_accessed): |
221 | + """See `IExternalBugTracker`.""" |
222 | + modified_bugs = self.getRemoteBugBatch( |
223 | + bug_ids, last_accessed=last_accessed |
224 | + ) |
225 | + self.cached_bugs.update(modified_bugs) |
226 | + return list(modified_bugs) |
227 | + |
228 | + def getRemoteBug(self, bug_id: str) -> Tuple[str, Optional[JiraBug]]: |
229 | + """See `ExternalBugTracker`.""" |
230 | + if bug_id not in self.cached_bugs: |
231 | + self.cached_bugs[bug_id] = self._loadJiraBug(bug_id) |
232 | + return bug_id, self.cached_bugs[bug_id] |
233 | + |
234 | + def getRemoteBugBatch( |
235 | + self, bug_ids, last_accessed=None |
236 | + ) -> Dict[str, Optional[JiraBug]]: |
237 | + """See `ExternalBugTracker`.""" |
238 | + bugs = { |
239 | + bug_id: self.cached_bugs[bug_id] |
240 | + for bug_id in bug_ids |
241 | + if bug_id in self.cached_bugs |
242 | + } |
243 | + if set(bugs) == set(bug_ids): |
244 | + return bugs |
245 | + |
246 | + for jira_bug in self._loadJiraBugs( |
247 | + bug_ids, last_accessed=last_accessed |
248 | + ): |
249 | + if jira_bug.key not in bug_ids: |
250 | + continue |
251 | + bugs[jira_bug.key] = self.cached_bugs[jira_bug.key] = jira_bug |
252 | + |
253 | + return bugs |
254 | + |
255 | + def getRemoteImportance(self, bug_id) -> str: |
256 | + """See `ExternalBugTracker`.""" |
257 | + remote_bug = self.bugs[bug_id] # type: JiraBug |
258 | + return remote_bug.priority.value |
259 | + |
260 | + def getRemoteStatus(self, bug_id) -> str: |
261 | + """See `ExternalBugTracker`.""" |
262 | + remote_bug = self.bugs[bug_id] # type: JiraBug |
263 | + return remote_bug.status.value |
264 | + |
265 | + def convertRemoteImportance( |
266 | + self, remote_importance: str |
267 | + ) -> BugTaskImportance: |
268 | + """See `IExternalBugTracker`.""" |
269 | + return JiraPriority(remote_importance).launchpad_importance |
270 | + |
271 | + def convertRemoteStatus(self, remote_status: str) -> BugTaskStatus: |
272 | + """See `IExternalBugTracker`.""" |
273 | + return JiraStatus(remote_status).launchpad_status |
274 | + |
275 | + def _getHeaders(self): |
276 | + headers = super()._getHeaders() |
277 | + credentials = self.credentials |
278 | + if credentials: |
279 | + headers["Authorization"] = "Basic {}".format( |
280 | + base64.b64encode( |
281 | + "{}:{}".format( |
282 | + credentials.username, credentials.password |
283 | + ).encode() |
284 | + ).decode() |
285 | + ) |
286 | + return headers |
287 | + |
288 | + def _loadJiraBug(self, bug_id: str) -> Optional[JiraBug]: |
289 | + # See https://docs.atlassian.com/software/jira/docs/api/REST/9.3.1/#api/2/issue-getIssue # noqa |
290 | + try: |
291 | + response = self._getPage( |
292 | + "issue/{}".format(bug_id), |
293 | + params={ |
294 | + "fields": "status,priority", |
295 | + }, |
296 | + ) |
297 | + except BugTrackerConnectError: |
298 | + return |
299 | + |
300 | + return JiraBug.from_api_data(response.json()) |
301 | + |
302 | + def _loadJiraBugs( |
303 | + self, bug_ids, last_accessed=None, start_at=0 |
304 | + ) -> Iterable[JiraBug]: |
305 | + # See https://docs.atlassian.com/software/jira/docs/api/REST/9.3.1/#api/2/search-searchUsingSearchRequest # noqa |
306 | + |
307 | + jql_query = "id in ({})".format(",".join(bug_ids)) |
308 | + if last_accessed is not None: |
309 | + jql_query = "{} AND updated >= {}".format( |
310 | + jql_query, last_accessed.strftime("%Y-%m-%d %H:%M") |
311 | + ) |
312 | + |
313 | + params = { |
314 | + "jql": jql_query, |
315 | + "fields": ["status", "priority"], |
316 | + "startAt": start_at, |
317 | + } |
318 | + |
319 | + response_data = self._postPage("search", data=params, json=True).json() |
320 | + |
321 | + max_results = response_data["maxResults"] |
322 | + total = response_data["total"] |
323 | + |
324 | + for bug_data in response_data["issues"]: |
325 | + yield JiraBug.from_api_data(bug_data) |
326 | + |
327 | + if total > (start_at + max_results): |
328 | + yield from self._loadJiraBugs( |
329 | + bug_ids, |
330 | + last_accessed=last_accessed, |
331 | + start_at=start_at + max_results, |
332 | + ) |
333 | diff --git a/lib/lp/bugs/externalbugtracker/tests/test_externalbugtracker.py b/lib/lp/bugs/externalbugtracker/tests/test_externalbugtracker.py |
334 | index 03b3197..2da0866 100644 |
335 | --- a/lib/lp/bugs/externalbugtracker/tests/test_externalbugtracker.py |
336 | +++ b/lib/lp/bugs/externalbugtracker/tests/test_externalbugtracker.py |
337 | @@ -167,7 +167,7 @@ class TestCheckwatchesConfig(TestCase): |
338 | ) |
339 | responses.add("POST", base_url + target, body=fake_form) |
340 | |
341 | - bugtracker._postPage(form, form={}, repost_on_redirect=True) |
342 | + bugtracker._postPage(form, {}, repost_on_redirect=True) |
343 | |
344 | requests = [call.request for call in responses.calls] |
345 | self.assertThat( |
346 | diff --git a/lib/lp/bugs/externalbugtracker/tests/test_jira.py b/lib/lp/bugs/externalbugtracker/tests/test_jira.py |
347 | new file mode 100644 |
348 | index 0000000..1301dce |
349 | --- /dev/null |
350 | +++ b/lib/lp/bugs/externalbugtracker/tests/test_jira.py |
351 | @@ -0,0 +1,432 @@ |
352 | +# Copyright 2022 Canonical Ltd. This software is licensed under the |
353 | +# GNU Affero General Public License version 3 (see the file LICENSE). |
354 | +import datetime |
355 | +import json |
356 | + |
357 | +import responses |
358 | +import transaction |
359 | +from testtools.matchers import ( |
360 | + ContainsDict, |
361 | + Equals, |
362 | + MatchesListwise, |
363 | + MatchesStructure, |
364 | + StartsWith, |
365 | +) |
366 | +from zope.component import getUtility |
367 | + |
368 | +from lp.app.interfaces.launchpad import ILaunchpadCelebrities |
369 | +from lp.bugs.externalbugtracker import get_external_bugtracker |
370 | +from lp.bugs.externalbugtracker.jira import ( |
371 | + Jira, |
372 | + JiraBug, |
373 | + JiraCredentials, |
374 | + JiraPriority, |
375 | + JiraStatus, |
376 | +) |
377 | +from lp.bugs.interfaces.bugtask import BugTaskStatus |
378 | +from lp.bugs.interfaces.bugtracker import BugTrackerType |
379 | +from lp.bugs.interfaces.externalbugtracker import IExternalBugTracker |
380 | +from lp.bugs.scripts.checkwatches import CheckwatchesMaster |
381 | +from lp.services.log.logger import BufferLogger |
382 | +from lp.testing import TestCase, TestCaseWithFactory, verifyObject |
383 | +from lp.testing.layers import ZopelessDatabaseLayer, ZopelessLayer |
384 | + |
385 | + |
386 | +class TestJira(TestCase): |
387 | + |
388 | + layer = ZopelessLayer |
389 | + |
390 | + def setUp(self): |
391 | + super().setUp() |
392 | + self.jira = Jira("https://warthogs.atlassian.net") |
393 | + self.pushConfig( |
394 | + "checkwatches.credentials", |
395 | + **{ |
396 | + "warthogs.atlassian.net.username": "launchpad", |
397 | + "warthogs.atlassian.net.password": "launchpad", |
398 | + }, |
399 | + ) |
400 | + |
401 | + def test_implements_interface(self): |
402 | + self.assertTrue(verifyObject(IExternalBugTracker, self.jira)) |
403 | + |
404 | + def test_convert_jira_url_to_api_endpoint(self): |
405 | + self.assertEqual( |
406 | + "https://warthogs.atlassian.net/rest/api/2", self.jira.baseurl |
407 | + ) |
408 | + |
409 | + def test_credentials(self): |
410 | + self.assertEqual( |
411 | + JiraCredentials( |
412 | + username="launchpad", |
413 | + password="launchpad", |
414 | + ), |
415 | + self.jira.credentials, |
416 | + ) |
417 | + |
418 | + def test_getHeaders(self): |
419 | + headers = self.jira._getHeaders() |
420 | + self.assertThat( |
421 | + headers, |
422 | + ContainsDict( |
423 | + {"Authorization": Equals("Basic bGF1bmNocGFkOmxhdW5jaHBhZA==")} |
424 | + ), |
425 | + ) |
426 | + |
427 | + @responses.activate |
428 | + def test_getCurrentDBTime(self): |
429 | + responses.add( |
430 | + "GET", |
431 | + self.jira.baseurl + "/serverInfo", |
432 | + json={ |
433 | + "baseUrl": "https://warthogs.atlassian.net", |
434 | + "buildDate": "2022-11-15T06:27:18.000+0800", |
435 | + "buildNumber": 100210, |
436 | + "defaultLocale": {"locale": "en_US"}, |
437 | + "deploymentType": "Cloud", |
438 | + "scmInfo": "28a36363a81be3fec088cc03de57ea0d3b868a26", |
439 | + "serverTime": "2022-11-15T14:11:11.818+0800", |
440 | + "serverTitle": "Jira", |
441 | + "version": "1001.0.0-SNAPSHOT", |
442 | + "versionNumbers": [1001, 0, 0], |
443 | + }, |
444 | + ) |
445 | + self.assertEqual( |
446 | + self.jira.getCurrentDBTime(), |
447 | + datetime.datetime( |
448 | + 2022, 11, 15, 6, 11, 11, 818000, tzinfo=datetime.timezone.utc |
449 | + ), |
450 | + ) |
451 | + requests = [call.request for call in responses.calls] |
452 | + self.assertThat( |
453 | + requests, |
454 | + MatchesListwise( |
455 | + [ |
456 | + MatchesStructure( |
457 | + method=Equals("GET"), |
458 | + path_url=Equals("/rest/api/2/serverInfo"), |
459 | + headers=ContainsDict( |
460 | + {"Authorization": StartsWith("Basic ")} |
461 | + ), |
462 | + ), |
463 | + ] |
464 | + ), |
465 | + ) |
466 | + |
467 | + @responses.activate |
468 | + def test_getRemoteBug(self): |
469 | + responses.add( |
470 | + "GET", |
471 | + self.jira.baseurl + "/issue/LP-984", |
472 | + json={ |
473 | + "fields": { |
474 | + "priority": {"name": "Medium"}, |
475 | + "status": {"statusCategory": {"key": "indeterminate"}}, |
476 | + }, |
477 | + "key": "LP-984", |
478 | + }, |
479 | + ) |
480 | + responses.add("GET", self.jira.baseurl + "/issue/LP-123", status=404) |
481 | + self.assertEqual( |
482 | + ( |
483 | + "LP-984", |
484 | + JiraBug( |
485 | + key="LP-984", |
486 | + status=JiraStatus.INDETERMINATE, |
487 | + priority=JiraPriority.MEDIUM, |
488 | + ), |
489 | + ), |
490 | + self.jira.getRemoteBug("LP-984"), |
491 | + ) |
492 | + self.assertEqual(("LP-123", None), self.jira.getRemoteBug("LP-123")) |
493 | + |
494 | + requests = [call.request for call in responses.calls] |
495 | + self.assertThat( |
496 | + requests, |
497 | + MatchesListwise( |
498 | + [ |
499 | + MatchesStructure( |
500 | + method=Equals("GET"), |
501 | + path_url=Equals( |
502 | + "/rest/api/2/issue/LP-984?fields=status%2Cpriority" |
503 | + ), |
504 | + ), |
505 | + MatchesStructure( |
506 | + method=Equals("GET"), |
507 | + path_url=Equals( |
508 | + "/rest/api/2/issue/LP-123?fields=status%2Cpriority" |
509 | + ), |
510 | + ), |
511 | + ] |
512 | + ), |
513 | + ) |
514 | + |
515 | + # Getting the same bug the second time should fetch it from the cache |
516 | + # without making another request to JIRA API |
517 | + self.assertEqual( |
518 | + ( |
519 | + "LP-984", |
520 | + JiraBug( |
521 | + key="LP-984", |
522 | + status=JiraStatus.INDETERMINATE, |
523 | + priority=JiraPriority.MEDIUM, |
524 | + ), |
525 | + ), |
526 | + self.jira.getRemoteBug("LP-984"), |
527 | + ) |
528 | + self.assertEqual(("LP-123", None), self.jira.getRemoteBug("LP-123")) |
529 | + self.assertEqual(2, len(responses.calls)) |
530 | + |
531 | + @responses.activate |
532 | + def test_getRemoteBugBatch(self): |
533 | + |
534 | + existing_bugs = [ |
535 | + { |
536 | + "fields": { |
537 | + "priority": {"name": "High"}, |
538 | + "status": {"statusCategory": {"key": "indeterminate"}}, |
539 | + }, |
540 | + "key": "1", |
541 | + }, |
542 | + { |
543 | + "fields": { |
544 | + "priority": {"name": "Medium"}, |
545 | + "status": {"statusCategory": {"key": "done"}}, |
546 | + }, |
547 | + "key": "2", |
548 | + }, |
549 | + ] |
550 | + |
551 | + def search_callback(request): |
552 | + payload = json.loads(request.body.decode()) |
553 | + start_at = payload["startAt"] |
554 | + |
555 | + if start_at >= len(existing_bugs): |
556 | + return 404, {}, "" |
557 | + |
558 | + return ( |
559 | + 200, |
560 | + {}, |
561 | + json.dumps( |
562 | + { |
563 | + "issues": existing_bugs[start_at : start_at + 1], |
564 | + "total": len(existing_bugs), |
565 | + "startAt": start_at, |
566 | + "maxResults": 1, |
567 | + } |
568 | + ), |
569 | + ) |
570 | + |
571 | + responses.add_callback( |
572 | + "POST", |
573 | + self.jira.baseurl + "/search", |
574 | + callback=search_callback, |
575 | + content_type="application/json", |
576 | + ) |
577 | + |
578 | + self.assertDictEqual( |
579 | + { |
580 | + "1": JiraBug( |
581 | + key="1", |
582 | + status=JiraStatus.INDETERMINATE, |
583 | + priority=JiraPriority.HIGH, |
584 | + ), |
585 | + "2": JiraBug( |
586 | + key="2", |
587 | + status=JiraStatus.DONE, |
588 | + priority=JiraPriority.MEDIUM, |
589 | + ), |
590 | + }, |
591 | + self.jira.getRemoteBugBatch(["1", "2"]), |
592 | + ) |
593 | + |
594 | + requests = [call.request for call in responses.calls] |
595 | + self.assertThat( |
596 | + requests, |
597 | + MatchesListwise( |
598 | + [ |
599 | + MatchesStructure( |
600 | + method=Equals("POST"), |
601 | + path_url=Equals("/rest/api/2/search"), |
602 | + ), |
603 | + MatchesStructure( |
604 | + method=Equals("POST"), |
605 | + path_url=Equals("/rest/api/2/search"), |
606 | + ), |
607 | + ] |
608 | + ), |
609 | + ) |
610 | + |
611 | + for i, call in enumerate(responses.calls): |
612 | + payload = json.loads(call.request.body.decode()) |
613 | + self.assertEqual("id in (1,2)", payload["jql"]) |
614 | + self.assertEqual(["status", "priority"], payload["fields"]) |
615 | + self.assertEqual(i, payload["startAt"]) |
616 | + |
617 | + # Getting the same bugs the second time should fetch it from the cache |
618 | + # without making another request to JIRA API |
619 | + self.assertDictEqual( |
620 | + { |
621 | + "1": JiraBug( |
622 | + key="1", |
623 | + status=JiraStatus.INDETERMINATE, |
624 | + priority=JiraPriority.HIGH, |
625 | + ), |
626 | + "2": JiraBug( |
627 | + key="2", |
628 | + status=JiraStatus.DONE, |
629 | + priority=JiraPriority.MEDIUM, |
630 | + ), |
631 | + }, |
632 | + self.jira.getRemoteBugBatch(["1", "2"]), |
633 | + ) |
634 | + self.assertEqual(2, len(responses.calls)) |
635 | + |
636 | + # Verify JQL query when `last_accessed` is specified |
637 | + self.jira.getRemoteBugBatch( |
638 | + ["3"], last_accessed=datetime.datetime(2000, 1, 1, 1, 2, 3) |
639 | + ) |
640 | + payload = json.loads(responses.calls[-1].request.body.decode()) |
641 | + self.assertEqual( |
642 | + "id in (3) AND updated >= 2000-01-01 01:02", payload["jql"] |
643 | + ) |
644 | + |
645 | + |
646 | +class TestJiraUpdateBugWatches(TestCaseWithFactory): |
647 | + |
648 | + layer = ZopelessDatabaseLayer |
649 | + |
650 | + @responses.activate |
651 | + def test_process_one(self): |
652 | + responses.add( |
653 | + "GET", |
654 | + "https://warthogs.atlassian.net/rest/api/2/issue/LP-984", |
655 | + json={ |
656 | + "fields": { |
657 | + "priority": {"name": "Medium"}, |
658 | + "status": {"statusCategory": {"key": "indeterminate"}}, |
659 | + }, |
660 | + "key": "LP-984", |
661 | + }, |
662 | + ) |
663 | + responses.add( |
664 | + "GET", |
665 | + "https://warthogs.atlassian.net/rest/api/2/serverInfo", |
666 | + json={ |
667 | + "serverTime": datetime.datetime.now( |
668 | + tz=datetime.timezone.utc |
669 | + ).isoformat() |
670 | + }, |
671 | + ) |
672 | + bug_tracker = self.factory.makeBugTracker( |
673 | + base_url="https://warthogs.atlassian.net", |
674 | + bugtrackertype=BugTrackerType.JIRA, |
675 | + ) |
676 | + bug = self.factory.makeBug() |
677 | + bug.addWatch( |
678 | + bug_tracker, "LP-984", getUtility(ILaunchpadCelebrities).janitor |
679 | + ) |
680 | + self.assertEqual( |
681 | + [("LP-984", None)], |
682 | + [ |
683 | + (watch.remotebug, watch.remotestatus) |
684 | + for watch in bug_tracker.watches |
685 | + ], |
686 | + ) |
687 | + transaction.commit() |
688 | + logger = BufferLogger() |
689 | + bug_watch_updater = CheckwatchesMaster(transaction, logger=logger) |
690 | + jira = get_external_bugtracker(bug_tracker) |
691 | + jira.batch_query_threshold = 1 |
692 | + bug_watch_updater.updateBugWatches(jira, bug_tracker.watches) |
693 | + self.assertEqual( |
694 | + "INFO Updating 1 watches for 1 bugs on " |
695 | + "https://warthogs.atlassian.net/rest/api/2\n", |
696 | + logger.getLogBuffer(), |
697 | + ) |
698 | + self.assertEqual( |
699 | + [("LP-984", BugTaskStatus.INPROGRESS)], |
700 | + [ |
701 | + ( |
702 | + watch.remotebug, |
703 | + jira.convertRemoteStatus(watch.remotestatus), |
704 | + ) |
705 | + for watch in bug_tracker.watches |
706 | + ], |
707 | + ) |
708 | + |
709 | + @responses.activate |
710 | + def test_process_many(self): |
711 | + remote_bugs = [ |
712 | + { |
713 | + "fields": { |
714 | + "priority": {"name": "Medium"}, |
715 | + "status": { |
716 | + "statusCategory": { |
717 | + "key": "indeterminate" |
718 | + if (bug_id % 2) == 0 |
719 | + else "done" |
720 | + } |
721 | + }, |
722 | + }, |
723 | + "key": str(bug_id), |
724 | + } |
725 | + for bug_id in range(1000, 1010) |
726 | + ] |
727 | + responses.add( |
728 | + "POST", |
729 | + "https://warthogs.atlassian.net/rest/api/2/search", |
730 | + json={ |
731 | + "startAt": 0, |
732 | + "maxResults": 100, |
733 | + "total": len(remote_bugs), |
734 | + "issues": remote_bugs, |
735 | + }, |
736 | + ) |
737 | + responses.add( |
738 | + "GET", |
739 | + "https://warthogs.atlassian.net/rest/api/2/serverInfo", |
740 | + json={ |
741 | + "serverTime": datetime.datetime.now( |
742 | + tz=datetime.timezone.utc |
743 | + ).isoformat() |
744 | + }, |
745 | + ) |
746 | + bug = self.factory.makeBug() |
747 | + bug_tracker = self.factory.makeBugTracker( |
748 | + base_url="https://warthogs.atlassian.net", |
749 | + bugtrackertype=BugTrackerType.JIRA, |
750 | + ) |
751 | + for remote_bug in remote_bugs: |
752 | + bug.addWatch( |
753 | + bug_tracker, |
754 | + remote_bug["key"], |
755 | + getUtility(ILaunchpadCelebrities).janitor, |
756 | + ) |
757 | + transaction.commit() |
758 | + logger = BufferLogger() |
759 | + bug_watch_updater = CheckwatchesMaster(transaction, logger=logger) |
760 | + jira = get_external_bugtracker(bug_tracker) |
761 | + bug_watch_updater.updateBugWatches(jira, bug_tracker.watches) |
762 | + self.assertEqual( |
763 | + "INFO Updating 10 watches for 10 bugs on " |
764 | + "https://warthogs.atlassian.net/rest/api/2\n", |
765 | + logger.getLogBuffer(), |
766 | + ) |
767 | + self.assertContentEqual( |
768 | + [ |
769 | + (str(bug_id), BugTaskStatus.INPROGRESS) |
770 | + for bug_id in (1000, 1002, 1004, 1006, 1008) |
771 | + ] |
772 | + + [ |
773 | + (str(bug_id), BugTaskStatus.FIXRELEASED) |
774 | + for bug_id in (1001, 1003, 1005, 1007, 1009) |
775 | + ], |
776 | + [ |
777 | + ( |
778 | + watch.remotebug, |
779 | + jira.convertRemoteStatus(watch.remotestatus), |
780 | + ) |
781 | + for watch in bug_tracker.watches |
782 | + ], |
783 | + ) |
784 | diff --git a/lib/lp/bugs/interfaces/bugtracker.py b/lib/lp/bugs/interfaces/bugtracker.py |
785 | index 3f383e3..a44c99b 100644 |
786 | --- a/lib/lp/bugs/interfaces/bugtracker.py |
787 | +++ b/lib/lp/bugs/interfaces/bugtracker.py |
788 | @@ -219,6 +219,15 @@ class BugTrackerType(DBEnumeratedType): |
789 | """, |
790 | ) |
791 | |
792 | + JIRA = DBItem( |
793 | + 14, |
794 | + """ |
795 | + JIRA Issues |
796 | + |
797 | + The issue tracker for JIRA-based projects. |
798 | + """, |
799 | + ) |
800 | + |
801 | |
802 | # A list of the BugTrackerTypes that don't need a remote product to be |
803 | # able to return a bug filing URL. We use a whitelist rather than a |
804 | diff --git a/lib/lp/bugs/model/bugwatch.py b/lib/lp/bugs/model/bugwatch.py |
805 | index 2bbef55..83fe3cd 100644 |
806 | --- a/lib/lp/bugs/model/bugwatch.py |
807 | +++ b/lib/lp/bugs/model/bugwatch.py |
808 | @@ -70,6 +70,7 @@ BUG_TRACKER_URL_FORMATS = { |
809 | BugTrackerType.TRAC: "ticket/%s", |
810 | BugTrackerType.SAVANE: "bugs/?%s", |
811 | BugTrackerType.PHPPROJECT: "bug.php?id=%s", |
812 | + BugTrackerType.JIRA: "%s", |
813 | } |
814 | |
815 | |
816 | @@ -418,6 +419,7 @@ class BugWatchSet: |
817 | BugTrackerType.SAVANE: self.parseSavaneURL, |
818 | BugTrackerType.SOURCEFORGE: self.parseSourceForgeLikeURL, |
819 | BugTrackerType.TRAC: self.parseTracURL, |
820 | + BugTrackerType.JIRA: self.parseJiraURL, |
821 | } |
822 | |
823 | def get(self, watch_id): |
824 | @@ -745,6 +747,15 @@ class BugWatchSet: |
825 | base_url = urlunsplit((scheme, host, base_path, "", "")) |
826 | return base_url, remote_bug |
827 | |
828 | + def parseJiraURL(self, scheme, host, path, query): |
829 | + """Extract a JIRA issue base URL and bug ID.""" |
830 | + match = re.match(r"^/browse/([A-Z]{1,10}-\d+)$", path) |
831 | + if not match: |
832 | + return None |
833 | + remote_bug = match.group(1) |
834 | + base_url = urlunsplit((scheme, host, "/", "", "")) |
835 | + return base_url, remote_bug |
836 | + |
837 | def extractBugTrackerAndBug(self, url): |
838 | """See `IBugWatchSet`.""" |
839 | for trackertype, parse_func in self.bugtracker_parse_functions.items(): |
840 | diff --git a/lib/lp/bugs/tests/test_bugwatch.py b/lib/lp/bugs/tests/test_bugwatch.py |
841 | index a71a7d0..275761b 100644 |
842 | --- a/lib/lp/bugs/tests/test_bugwatch.py |
843 | +++ b/lib/lp/bugs/tests/test_bugwatch.py |
844 | @@ -224,6 +224,15 @@ class ExtractBugTrackerAndBugTest(WithScenarios, TestCase): |
845 | "bug_id": "12345", |
846 | }, |
847 | ), |
848 | + ( |
849 | + "JIRA", |
850 | + { |
851 | + "bugtracker_type": BugTrackerType.JIRA, |
852 | + "bug_url": "https://warthogs.atlassian.net/browse/LP-984", |
853 | + "base_url": "https://warthogs.atlassian.net/", |
854 | + "bug_id": "LP-984", |
855 | + }, |
856 | + ), |
857 | ] |
858 | |
859 | layer = LaunchpadFunctionalLayer |
860 | diff --git a/lib/lp/services/config/schema-lazr.conf b/lib/lp/services/config/schema-lazr.conf |
861 | index 640865d..da9477e 100644 |
862 | --- a/lib/lp/services/config/schema-lazr.conf |
863 | +++ b/lib/lp/services/config/schema-lazr.conf |
864 | @@ -237,7 +237,8 @@ api.github.com.token: none |
865 | gitlab.com.token: none |
866 | gitlab.gnome.org.token: none |
867 | salsa.debian.org.token: none |
868 | - |
869 | +warthogs.atlassian.net.username: none |
870 | +warthogs.atlassian.net.password: none |
871 | |
872 | [cibuild.soss] |
873 | # value is a JSON Object |
This looks like a perfectly reasonable first-pass implementation, and I'm fine with landing it as far as it goes.
We'll need to be careful about the credentials handling. The existing `checkwatches. credentials` stuff was intended for cases where the bug tracker itself is essentially public but we need some kind of credentials for Launchpad to connect to it anyway, either out of politeness (credentials allow our sync script to be identified unambiguously), or to gain access to higher rate limits (as in the GitHub/GitLab cases), or because we need to push comments (for Bugzilla). In this case, though, the credentials are partly also being used because the remote bug tracker is private, which is a very different matter: if we were to configure these credentials on production, it would mean that anyone could discover information about the status of a given Jira issue by guessing its URL and adding a bug watch for it. Not a very serious information leak since it only tells you the remote status and importance, but nevertheless probably not something we should leave designed into the system in case somebody wants to extend it in future to gather more information. I think it's fine to leave an XXX comment about this for now, though, as it doesn't become a problem until we configure credentials; perhaps we could change some other part of the system to restrict who can add such bug watches, or restrict them to certain projects, or something like that.
Having gathered requirements for Launchpad/Jira integration, I also think this will probably not address those requirements on its own (though it may be a component of the eventual solution). I've belatedly written down what I know so far here: https:/ /docs.google. com/document/ d/1CiEgo- CHX8Go0lTAdryqK CeFxaVnBq49QVkf BshX28M