Launchpad itself

Merge ~cjwatson/launchpad:librarian-new-container-workaround into launchpad:master

  1. Git
  2. lp:~cjwatson/launchpad
  3. librarian-new-container-workaround
  4. Merge into master
Proposed by Colin Watson
Status: Rejected
Rejected by: Colin Watson
Proposed branch: ~cjwatson/launchpad:librarian-new-container-workaround
Merge into: launchpad:master
Diff against target: 70 lines (+32/-1)
3 files modified
lib/lp/services/config/schema-lazr.conf (+5/-0)
lib/lp/services/librarianserver/swift.py (+9/-0)
lib/lp/services/librarianserver/tests/test_swift.py (+18/-1)
Reviewer Review Type Date Requested Status
Ioana Lasc (community) Approve
Review via email: mp+405744@code.launchpad.net

Commit message

Allow librarian-feed-swift to sleep after creating container

Description of the change

radosgw apparently has a bug where newly-created containers need to be left idle for a short time before they can be used reliably. For now, allow working around this by configuring `librarian_server.new_container_delay` to a number of seconds to sleep after creating a new container.

See https://portal.admin.canonical.com/C131836.

To post a comment you must log in.
Revision history for this message
Ioana Lasc (ilasc) :
review: Approve
Revision history for this message
Colin Watson (cjwatson) wrote :

For the record: I'm keeping this in my back pocket until we find out whether just upgrading ceph on PS5 will work. That should pick up the fix for https://tracker.ceph.com/issues/48632, which looks like a promising candidate.

Revision history for this message
Colin Watson (cjwatson) wrote :

Withdrawing this, because it turned out that a ceph upgrade was good enough.

Unmerged commits

85603c6... by Colin Watson

Allow librarian-feed-swift to sleep after creating container

radosgw apparently has a bug where newly-created containers need to be
left idle for a short time before they can be used reliably. For now,
allow working around this by configuring
`librarian_server.new_container_delay` to a number of seconds to sleep
after creating a new container.

See https://portal.admin.canonical.com/C131836.

Preview Diff

[H/L] Next/Prev Comment, [J/K] Next/Prev File, [N/P] Next/Prev Hunk
1diff --git a/lib/lp/services/config/schema-lazr.conf b/lib/lp/services/config/schema-lazr.conf
2index 749ecaa..894fac0 100644
3--- a/lib/lp/services/config/schema-lazr.conf
4+++ b/lib/lp/services/config/schema-lazr.conf
5@@ -1247,6 +1247,11 @@ restricted_logfile: -
6 # datatype: string
7 root: none
8
9+# Allow working around a radosgw bug by sleeping for a short time (measured
10+# in seconds) after creating new containers.
11+# datatype: integer
12+new_container_delay: 0
13+
14 # Swift connection information and secret.
15 #
16 # datatype: urlbase
17diff --git a/lib/lp/services/librarianserver/swift.py b/lib/lp/services/librarianserver/swift.py
18index 662b2fa..81df074 100644
19--- a/lib/lp/services/librarianserver/swift.py
20+++ b/lib/lp/services/librarianserver/swift.py
21@@ -180,6 +180,15 @@ def _to_swift_file(log, swift_connection, lfc_id, fs_path):
22 raise
23 log.info('Creating {0} container'.format(container))
24 swift_connection.put_container(container)
25+ # radosgw apparently has a bug where newly-created containers need
26+ # to be left idle for a short time before they can be used reliably.
27+ # If configured to do so, work around this by sleeping for a while
28+ # before continuing.
29+ if config.librarian_server.new_container_delay:
30+ log.info(
31+ 'Sleeping for %d seconds after creating container',
32+ config.librarian_server.new_container_delay)
33+ time.sleep(config.librarian_server.new_container_delay)
34
35 try:
36 headers = quiet_swiftclient(
37diff --git a/lib/lp/services/librarianserver/tests/test_swift.py b/lib/lp/services/librarianserver/tests/test_swift.py
38index 61a4a3d..5555f01 100644
39--- a/lib/lp/services/librarianserver/tests/test_swift.py
40+++ b/lib/lp/services/librarianserver/tests/test_swift.py
41@@ -1,4 +1,4 @@
42-# Copyright 2013-2017 Canonical Ltd. This software is licensed under the
43+# Copyright 2013-2021 Canonical Ltd. This software is licensed under the
44 # GNU Affero General Public License version 3 (see the file LICENSE).
45
46 """Librarian disk to Swift storage tests."""
47@@ -367,6 +367,23 @@ class TestFeedSwift(TestCase):
48 finally:
49 swift_client.close()
50
51+ def test_new_container_delay_unconfigured(self):
52+ # If not explicitly configured to do so, _to_swift_file does not
53+ # sleep after creating a new container.
54+ log = BufferLogger()
55+ with patch('time.sleep') as mock_sleep:
56+ swift.to_swift(log)
57+ mock_sleep.assert_not_called()
58+
59+ def test_new_container_delay(self):
60+ # _to_swift_file sleeps for a configurable number of seconds after
61+ # creating a new container, to work around a radosgw bug.
62+ log = BufferLogger()
63+ self.pushConfig('librarian_server', new_container_delay=60)
64+ with patch('time.sleep') as mock_sleep:
65+ swift.to_swift(log)
66+ mock_sleep.assert_called_once_with(60)
67+
68
69 class TestHashStream(TestCase):
70 layer = BaseLayer

Subscribers

People subscribed via source and target branches

to status/vote changes: