Merge ~pelpsi/launchpad:typeerror-unmarshall-non-string-objects into launchpad:master

Proposed by Simone Pelosi
Status: Merged
Approved by: Simone Pelosi
Approved revision: 02462e1fffc54c4816b9373f0b9be8f682b88507
Merge reported by: Otto Co-Pilot
Merged at revision: not available
Proposed branch: ~pelpsi/launchpad:typeerror-unmarshall-non-string-objects
Merge into: launchpad:master
Diff against target: 124 lines (+85/-1)
2 files modified
lib/lp/services/fields/__init__.py (+4/-1)
lib/lp/snappy/browser/tests/test_snap.py (+81/-0)
Reviewer Review Type Date Requested Status
Ines Almeida Approve
Colin Watson (community) Approve
Review via email: mp+450364@code.launchpad.net

Commit message

Drop `value_type` to leave dict values unvalidated

Weakening `value_type` validation we can support `_byarch` member
in `auto_build_channels` variable. This should be converted later in
a custom validation for `_byarch`.

LP: #2033382

To post a comment you must log in.
Revision history for this message
Colin Watson (cjwatson) wrote :

I'm not sure about this. Do you know why we're returning an int in an allegedly text field in the first place? That seems like a type error that we should fix.

review: Needs Information
Revision history for this message
Ines Almeida (ines-almeida) wrote :

What type of value were we passing to the `obfuscated_email` function that caused the TypeError? I'm wondering since this is a `TextFieldMarshaller`...

It would be good to create a test that replicated the bug, i.e., a test that makes a request that would have led to the TypeError.

review: Needs Information
Revision history for this message
Simone Pelosi (pelpsi) wrote (last edit ):

> I'm not sure about this. Do you know why we're returning an int in an
> allegedly text field in the first place? That seems like a type error that we
> should fix.

The problem for core20 is caused by `'auto_build_channels':{'_byarch': {'riscv64': {'snapcraft': 'beta'}}}` because we are trying to unmarshall `{'riscv64': {'snapcraft': 'beta'}}` that is not a string.

For core22 we have instead `"auto_build_channels": {"snapcraft": "stable/launchpad-buildd"}` and we can successfully unmarshall "stable/launchpad-buildd"

I'm not sure if it is correct to have a nested dictionary in "auto_build_channels"

Revision history for this message
Simone Pelosi (pelpsi) wrote :

> What type of value were we passing to the `obfuscated_email` function that
> caused the TypeError? I'm wondering since this is a `TextFieldMarshaller`...
>
> It would be good to create a test that replicated the bug, i.e., a test that
> makes a request that would have led to the TypeError.

Sure, I can push the test I created during the investigation.
I will put it in `lp.snappy.browser.tests.test_snap`

Revision history for this message
Simone Pelosi (pelpsi) wrote (last edit ):

Moreover you will get a similar error ( https://oops.canonical.com/oops/?oopsid=OOPS-f72a3fae0ab34f1607dfbd203f5b274a ) navigating `https://launchpad.net/~ubuntu-core-service/+snap/core20` in incognito mode, that's because we call the obfuscation in that case.

In this specific case is a little bit different because we are using json.dump on the cache but the error is pretty much the same, we are not recursively iterating on the dictionary and the obfuscation method fails

Revision history for this message
Colin Watson (cjwatson) wrote :

In that case, I think the correct fix would be to weaken `value_type` in the constructor for `SnapBuildChannelsField` instead. It's legitimate for the channels dict to contain a `_byarch` member which is then used to select different behaviour depending on the architecture (see `Snap.requestBuildsFromJob`).

I think the reasonable choices are (1) to just drop `value_type` entirely and leave the dict values unvalidated, or (2) to figure out how to implement custom validation here that validates `_byarch` values as dictionaries mapping architecture names to the current definition of `SnapBuildChannelsField`, and any other values to `TextLine`. The stricter typing in (2) is probably more desirable, but I'm not sure whether it will turn out to be an unreasonable amount of work.

Revision history for this message
Simone Pelosi (pelpsi) wrote :

I think that we can choose the (1) to push a quick fix and then we can open a ticket to implement the stricter typing (2) to have a more elegant solution. What do you think ?

Revision history for this message
Colin Watson (cjwatson) wrote :

I'm fine with that.

Revision history for this message
Simone Pelosi (pelpsi) wrote :

Implemented agreed changes and added new test cases

Revision history for this message
Colin Watson (cjwatson) wrote :

Before you land this, please use `git rebase -i` or similar to amend the history to remove the commit that you reverted.

review: Approve
Revision history for this message
Colin Watson (cjwatson) wrote :

Oh, also please update the overall commit message of this MP.

Revision history for this message
Simone Pelosi (pelpsi) wrote :

Thank you Colin for the review, I squashed commits into one and I updated the commit message

Revision history for this message
Ines Almeida (ines-almeida) :
review: Approve

Preview Diff

[H/L] Next/Prev Comment, [J/K] Next/Prev File, [N/P] Next/Prev Hunk
1diff --git a/lib/lp/services/fields/__init__.py b/lib/lp/services/fields/__init__.py
2index a4a919a..dff799d 100644
3--- a/lib/lp/services/fields/__init__.py
4+++ b/lib/lp/services/fields/__init__.py
5@@ -1035,9 +1035,12 @@ class SnapBuildChannelsField(Dict):
6 for snap_name in sorted(snap_names)
7 )
8 )
9+ # TODO: custom validation that validates `_byarch` values
10+ # as dictionaries mapping architecture names to the current
11+ # definition of `SnapBuildChannelsField`,
12+ # and any other values to `TextLine`.
13 super().__init__(
14 key_type=Choice(values=snap_names),
15- value_type=TextLine(),
16 description=description,
17 **kwargs,
18 )
19diff --git a/lib/lp/snappy/browser/tests/test_snap.py b/lib/lp/snappy/browser/tests/test_snap.py
20index f5535cc..1004c4c 100644
21--- a/lib/lp/snappy/browser/tests/test_snap.py
22+++ b/lib/lp/snappy/browser/tests/test_snap.py
23@@ -72,6 +72,7 @@ from lp.testing import (
24 BrowserTestCase,
25 TestCaseWithFactory,
26 admin_logged_in,
27+ api_url,
28 login,
29 login_admin,
30 login_person,
31@@ -88,6 +89,7 @@ from lp.testing.pages import (
32 find_tag_by_id,
33 find_tags_by_class,
34 get_feedback_messages,
35+ webservice_for_person,
36 )
37 from lp.testing.publication import test_traverse
38 from lp.testing.views import create_initialized_view, create_view
39@@ -2514,6 +2516,85 @@ class TestSnapView(BaseTestSnapView):
40 authorize_link = browser.getLink("Reauthorize store uploads")
41 self.assertEqual(authorize_url, authorize_link.url)
42
43+ def test_snap_anonymous_view(self):
44+ registrant = self.factory.makePerson(name="snap-registrant")
45+ team_owner = self.factory.makeTeam(
46+ name="snap-creator", members=[registrant]
47+ )
48+ date_created = datetime(2000, 1, 1, tzinfo=timezone.utc)
49+ snap = self.factory.makeSnap(
50+ name="core20",
51+ registrant=registrant,
52+ owner=team_owner,
53+ date_created=date_created,
54+ auto_build_channels={"snapcraft": "stable/launchpad-buildd"},
55+ )
56+ browser = self.getViewBrowser(
57+ context=snap, no_login=True, user=registrant
58+ )
59+ content = find_main_content(browser.contents)
60+ self.assertThat(
61+ "Source snap channels for automatic builds:\n"
62+ "snapcraft\nstable/launchpad-buildd",
63+ MatchesTagText(content, "auto_build_channels"),
64+ )
65+
66+ # Bug #2033382
67+ def test_snap_anonymous_view_byarch(self):
68+ registrant = self.factory.makePerson(name="snap-registrant")
69+ team_owner = self.factory.makeTeam(
70+ name="snap-creator", members=[registrant]
71+ )
72+ date_created = datetime(2000, 1, 1, tzinfo=timezone.utc)
73+ snap = self.factory.makeSnap(
74+ name="core20",
75+ registrant=registrant,
76+ owner=team_owner,
77+ date_created=date_created,
78+ auto_build_channels={
79+ "_byarch": {"riscv64": {"snapcraft": "beta"}}
80+ },
81+ )
82+ browser = self.getViewBrowser(
83+ context=snap, no_login=True, user=registrant
84+ )
85+ content = find_main_content(browser.contents)
86+ self.assertThat(
87+ "Source snap channels for automatic builds:\n"
88+ "_byarch\n{'riscv64': {'snapcraft': 'beta'}}",
89+ MatchesTagText(content, "auto_build_channels"),
90+ )
91+
92+ # Bug #2033382
93+ def test_snap_anonymous_api_byarch(self):
94+ registrant = self.factory.makePerson(name="snap-registrant")
95+ team_owner = self.factory.makeTeam(
96+ name="snap-creator", members=[registrant]
97+ )
98+ date_created = datetime(2000, 1, 1, tzinfo=timezone.utc)
99+ snap = self.factory.makeSnap(
100+ name="core20",
101+ registrant=registrant,
102+ owner=team_owner,
103+ date_created=date_created,
104+ auto_build_channels={
105+ "_byarch": {"riscv64": {"snapcraft": "beta"}}
106+ },
107+ )
108+ api_base = "http://api.launchpad.test/devel"
109+ snap_url = api_base + api_url(snap)
110+
111+ webservice = webservice_for_person(
112+ None,
113+ default_api_version="devel",
114+ )
115+ response = webservice.get(snap_url)
116+ self.assertEqual(200, response.status)
117+ self.assertEqual(
118+ {"_byarch": {"riscv64": {"snapcraft": "beta"}}},
119+ response.jsonBody()["auto_build_channels"],
120+ )
121+
122
123 class TestHintedSnapBuildChannelsWidget(TestSnapBuildChannelsWidget):
124 def setUp(self):

Subscribers

People subscribed via source and target branches

to status/vote changes: