Launchpad itself

Merge lp:~stub/launchpad/update-storm into lp:launchpad

  1. update-storm
  2. Merge into devel
Proposed by Stuart Bishop
Status: Merged
Merge reported by: Stuart Bishop
Merged at revision: not available
Proposed branch: lp:~stub/launchpad/update-storm
Merge into: lp:launchpad
Diff against target: None lines
To merge this branch: bzr merge lp:~stub/launchpad/update-storm
Reviewer Review Type Date Requested Status
Jonathan Lange (community) Approve
Christian Reis release-critical Pending
Review via email: mp+9130@code.launchpad.net
To post a comment you must log in.
Revision history for this message
Stuart Bishop (stub) wrote :

Bug #393625 was raised because one of our scripts was using an unacceptable amount of RAM. Testing showed reducing the Storm cache size fixed the problem. Most likely, this script is dealing with objects containing long text columns.

This branch changes the default Storm cache size to be much more conservative. It also adds a load of boilerplate so we can easily override the conservative default for individual components where we detect slowness due to cache thrashing and can afford to use more RAM.

The appserver cache size settings remain unchanged from the value we have been running on edge for the last few weeks.

I would like to get this branch landed release critical so we don't have to cherry pick or disable update-pkgcache.

Revision history for this message
Jonathan Lange (jml) wrote :

Hey Stuart,

Thanks for debugging this problem. The changes look good, but I've got a couple of questions:

  - is there a way to avoid repeating the settings so many times over? is it desirable to do so?

  - why is one of the items changed to a cache size of 10000?

  - can you please update the storm_cache_size comment in the schema to say what units are being used?

Thanks,
jml

review: Needs Fixing
Revision history for this message
Stuart Bishop (stub) wrote :

On Wed, Jul 22, 2009 at 2:11 PM, Jonathan Lange<email address hidden> wrote:
> Review: Needs Fixing
> Hey Stuart,
>
> Thanks for debugging this problem. The changes look good, but I've got a couple of questions:
>
>  - is there a way to avoid repeating the settings so many times over? is it desirable to do so?
>

It doesn't look like it can be avoided. It is not desirable. I'm not sure if the new config machinery was implemented this way by design or by necessity. Perhaps I'm doing something wrong.

>  - why is one of the items changed to a cache size of 10000?

That is the appserver. Its not being changed - it is just being moved up a bit to be consistent.

>  - can you please update the storm_cache_size comment in the schema to say what units are being used?

Done.

I think I need to discuss on the Storm list how we should implement a cache that is bounded by RAM usage rather than object count.

--
Stuart Bishop <email address hidden>
http://www.stuartbishop.net/

Revision history for this message
Jonathan Lange (jml) wrote :

Thanks for that.

I agree, raising that on the storm list is definitely a good plan. Raising the config issue on the launchpad-dev list might be a good idea as well.

Oh, I forgot -- do you need to change the production config branch also?

jml

review: Approve
Revision history for this message
Stuart Bishop (stub) wrote :

> >  - is there a way to avoid repeating the settings so many times over? is it
> desirable to do so?
> >
>
> It doesn't look like it can be avoided. It is not desirable. I'm not sure if
> the new config machinery was implemented this way by design or by necessity.
> Perhaps I'm doing something wrong.

I should explain this more. Without this boilerplate, the values are inherited from the [database] section but they cannot be overridden in a launchpad-lazr.conf file. I believe this is because the config parser knows nothing about how these sections inherit from the [database] section causing unknown key errors to be raised. The [database] fallback is done in lib/canonical/config and is launchpad specific.

Revision history for this message
Stuart Bishop (stub) wrote :

On Wed, Jul 22, 2009 at 2:24 PM, Jonathan Lange<email address hidden> wrote:

> Oh, I forgot -- do you need to change the production config branch also?

No - I'm changing the defaults. Production configs don't override them yet.

--
Stuart Bishop <email address hidden>
http://www.stuartbishop.net/

Revision history for this message
Christian Reis (kiko) wrote :

On Wed, Jul 22, 2009 at 06:55:23AM -0000, Stuart Bishop wrote:
> -# datatype: string
> +# datatype: string.

Don't add the period here. If this isn't going to risk keeping us all up
at 2am tonight, then rc=kiko ;-)
--
Christian Robottom Reis | [+55 16] 3376 0125 | http://launchpad.net/~kiko
                        | [+55 16] 9112 6430 | http://async.com.br/~kiko

Preview Diff

[H/L] Next/Prev Comment, [J/K] Next/Prev File, [N/P] Next/Prev Hunk
=== modified file 'lib/canonical/config/schema-lazr.conf'
--- lib/canonical/config/schema-lazr.conf 2009-07-16 21:12:16 +0000
+++ lib/canonical/config/schema-lazr.conf 2009-07-22 06:43:05 +0000
@@ -2,6 +2,8 @@
2# The database user which will be used to expire questions.2# The database user which will be used to expire questions.
3# datatype: string3# datatype: string
4dbuser: answertracker4dbuser: answertracker
5storm_cache: generational
6storm_cache_size: 500
57
6# The number of days of inactivity required before a question in8# The number of days of inactivity required before a question in
7# the open or needs information state is expired.9# the open or needs information state is expired.
@@ -31,6 +33,8 @@
31# The database user which will be used by this process.33# The database user which will be used by this process.
32# datatype: string34# datatype: string
33dbuser: branchscanner35dbuser: branchscanner
36storm_cache: generational
37storm_cache_size: 500
3438
35# See [error_reports].39# See [error_reports].
36error_dir: none40error_dir: none
@@ -46,6 +50,8 @@
46# The database user which will be used by this process.50# The database user which will be used by this process.
47# datatype: string51# datatype: string
48dbuser: fiera52dbuser: fiera
53storm_cache: generational
54storm_cache_size: 500
4955
50# datatype: string56# datatype: string
51default_sender_address: noreply@launchpad.net57default_sender_address: noreply@launchpad.net
@@ -178,6 +184,8 @@
178# The database user to run this process as.184# The database user to run this process as.
179# datatype: string185# datatype: string
180dbuser: checkwatches186dbuser: checkwatches
187storm_cache: generational
188storm_cache_size: 500
181189
182# See [error_reports].190# See [error_reports].
183error_dir: none191error_dir: none
@@ -477,6 +485,8 @@
477# The database user which will be used by this process.485# The database user which will be used by this process.
478# datatype: string486# datatype: string
479dbuser: create-merge-proposals487dbuser: create-merge-proposals
488storm_cache: generational
489storm_cache_size: 500
480490
481# See [error_reports].491# See [error_reports].
482error_dir: none492error_dir: none
@@ -540,10 +550,16 @@
540soft_request_timeout: None550soft_request_timeout: None
541551
542# The Storm cache type to use. May be 'default', 'generational' or 'stupid'552# The Storm cache type to use. May be 'default', 'generational' or 'stupid'
553# datatype: string
543storm_cache: generational554storm_cache: generational
544555
545# The size of the Storm cache.556# The size of the Storm cache. We start small, because this is the
546storm_cache_size: 5000557# default used by everything and we have no idea if we are dealing
558# with tiny objects or huge objects. Individual scripts can easily
559# increase the cache size if database performance is an issue and RAM
560# usage is not. See Bug #393625 for the issue that prompted this change.
561# datatype: integer
562storm_cache_size: 500
547563
548564
549[debug]565[debug]
@@ -609,13 +625,16 @@
609# The database user which will be used by this process.625# The database user which will be used by this process.
610# datatype: string626# datatype: string
611dbuser: teammembership627dbuser: teammembership
628storm_cache: generational
629storm_cache_size: 500
612630
613631
614[gina]632[gina]
615# The database user which will be used by this process.633# The database user which will be used by this process.
616# datatype: string634# datatype: string
617dbuser: gina635dbuser: gina
618636storm_cache: generational
637storm_cache_size: 500
619638
620[gina_target.template]639[gina_target.template]
621# The distribution name (e.g. ubuntu) from where the packages640# The distribution name (e.g. ubuntu) from where the packages
@@ -803,6 +822,8 @@
803# The database user which will be used by this process.822# The database user which will be used by this process.
804# datatype: string823# datatype: string
805dbuser: karma824dbuser: karma
825storm_cache: generational
826storm_cache_size: 500
806827
807# When calculating karma, if a categories scaling factor is828# When calculating karma, if a categories scaling factor is
808# larger than this it is reduced down to this maximum. This is829# larger than this it is reduced down to this maximum. This is
@@ -821,6 +842,8 @@
821# datatype: string842# datatype: string
822dbuser: launchpad_main843dbuser: launchpad_main
823auth_dbuser: launchpad_auth844auth_dbuser: launchpad_auth
845storm_cache: generational
846storm_cache_size: 10000
824847
825# If true, Launchpad is running in read-only mode. Attempts to848# If true, Launchpad is running in read-only mode. Attempts to
826# write to the Launchpad database will be denied, and an explanatory849# write to the Launchpad database will be denied, and an explanatory
@@ -1019,10 +1042,6 @@
1019# ba-ws.geonames.net.1042# ba-ws.geonames.net.
1020geonames_identity:1043geonames_identity:
10211044
1022storm_cache: generational
1023storm_cache_size: 10000
1024
1025
1026[launchpad_session]1045[launchpad_session]
1027# The hostname where the session database is located.1046# The hostname where the session database is located.
1028# If the value is empty or None, localhost via UNIX sockets1047# If the value is empty or None, localhost via UNIX sockets
@@ -1051,6 +1070,8 @@
1051# The database user which will be used by this process.1070# The database user which will be used by this process.
1052# datatype: string1071# datatype: string
1053dbuser: librarian1072dbuser: librarian
1073storm_cache: generational
1074storm_cache_size: 500
10541075
1055isolation_level: read_committed1076isolation_level: read_committed
10561077
@@ -1104,6 +1125,8 @@
1104# The database user which will be used by this process.1125# The database user which will be used by this process.
1105# datatype: string1126# datatype: string
1106dbuser: librariangc1127dbuser: librariangc
1128storm_cache: generational
1129storm_cache_size: 500
11071130
11081131
1109[librarian_server]1132[librarian_server]
@@ -1304,6 +1327,8 @@
1304# The database user which will be used by this process.1327# The database user which will be used by this process.
1305# datatype: string1328# datatype: string
1306dbuser: mp-creation-job1329dbuser: mp-creation-job
1330storm_cache: generational
1331storm_cache_size: 500
13071332
1308# See [error_reports].1333# See [error_reports].
1309error_dir: none1334error_dir: none
@@ -1323,7 +1348,6 @@
1323# datatype: integer1348# datatype: integer
1324retained_days: 3661349retained_days: 366
13251350
1326
1327[personalpackagearchive]1351[personalpackagearchive]
1328# Directory to be created to store PPAs.1352# Directory to be created to store PPAs.
1329# datatype: string1353# datatype: string
@@ -1359,18 +1383,22 @@
1359# The database user which will be used by this process.1383# The database user which will be used by this process.
1360# datatype: string1384# datatype: string
1361dbuser: poimport1385dbuser: poimport
13621386storm_cache: generational
1387storm_cache_size: 500
13631388
1364[processmail]1389[processmail]
1365# The database user which will be used by this process.1390# The database user which will be used by this process.
1366# datatype: string1391# datatype: string
1367dbuser: processmail1392dbuser: processmail
13681393storm_cache: generational
1394storm_cache_size: 500
13691395
1370[productreleasefinder]1396[productreleasefinder]
1371# The database user which will be used by this process.1397# The database user which will be used by this process.
1372# datatype: string1398# datatype: string
1373dbuser: productreleasefinder1399dbuser: productreleasefinder
1400storm_cache: generational
1401storm_cache_size: 500
13741402
1375[profiling]1403[profiling]
1376# When set to True, each requests will be profiled and the resulting data1404# When set to True, each requests will be profiled and the resulting data
@@ -1392,6 +1420,8 @@
1392# The database user which will be used by this process.1420# The database user which will be used by this process.
1393# datatype: string1421# datatype: string
1394dbuser: reclaim-branch-space1422dbuser: reclaim-branch-space
1423storm_cache: generational
1424storm_cache_size: 500
13951425
1396# See [error_reports].1426# See [error_reports].
1397error_dir: none1427error_dir: none
@@ -1434,9 +1464,14 @@
1434translate_pages_max_batch_size: 501464translate_pages_max_batch_size: 50
14351465
1436[rosettabranches]1466[rosettabranches]
1467# XXX stub 20090722 bug=402891: Scripts need to use unique
1468# database users but the rosetta branch scanner is incorrectly reusing an
1469# existing database user.
1437# The database user which will be used by the rosetta-branches cronscript.1470# The database user which will be used by the rosetta-branches cronscript.
1438# datatype: string1471# datatype: string.
1439dbuser: branchscanner1472dbuser: branchscanner
1473storm_cache: generational
1474storm_cache_size: 500
14401475
1441# See [error_reports].1476# See [error_reports].
1442error_dir: none1477error_dir: none
@@ -1452,6 +1487,8 @@
1452# The database user which will be used by this process.1487# The database user which will be used by this process.
1453# datatype: string1488# datatype: string
1454dbuser: send-branch-mail1489dbuser: send-branch-mail
1490storm_cache: generational
1491storm_cache_size: 500
14551492
1456# See [error_reports].1493# See [error_reports].
1457error_dir: none1494error_dir: none
@@ -1467,6 +1504,8 @@
1467# The database user which will be used by this process.1504# The database user which will be used by this process.
1468# datatype: string1505# datatype: string
1469dbuser: shipit1506dbuser: shipit
1507storm_cache: generational
1508storm_cache_size: 500
14701509
1471# datatype: string1510# datatype: string
1472admins_email_address: info@shipit.ubuntu.com1511admins_email_address: info@shipit.ubuntu.com
@@ -1501,7 +1540,8 @@
1501# The database user which will be used by this process.1540# The database user which will be used by this process.
1502# datatype: string1541# datatype: string
1503dbuser: statistician1542dbuser: statistician
15041543storm_cache: generational
1544storm_cache_size: 500
15051545
1506[supermirror]1546[supermirror]
1507# The longest period of time, in seconds, that the scheduler will1547# The longest period of time, in seconds, that the scheduler will
@@ -1550,22 +1590,30 @@
1550# The database user which will be used by this process.1590# The database user which will be used by this process.
1551# datatype: string1591# datatype: string
1552dbuser: targetnamecacheupdater1592dbuser: targetnamecacheupdater
1593storm_cache: generational
1594storm_cache_size: 500
15531595
15541596
1555[updateremoteproduct]1597[updateremoteproduct]
1556# The database user to run this process as.1598# The database user to run this process as.
1557# datatype: string1599# datatype: string
1558dbuser: updateremoteproduct1600dbuser: updateremoteproduct
1601storm_cache: generational
1602storm_cache_size: 500
15591603
1560[updatesourceforgeremoteproduct]1604[updatesourceforgeremoteproduct]
1561# The database user to run this process as.1605# The database user to run this process as.
1562# datatype: string1606# datatype: string
1563dbuser: updatesourceforgeremoteproduct1607dbuser: updatesourceforgeremoteproduct
1608storm_cache: generational
1609storm_cache_size: 500
15641610
1565[uploader]1611[uploader]
1566# The database user which will be used by this process.1612# The database user which will be used by this process.
1567# datatype: string1613# datatype: string
1568dbuser: uploader1614dbuser: uploader
1615storm_cache: generational
1616storm_cache_size: 500
15691617
1570# datatype: string1618# datatype: string
1571default_recipient_name: none1619default_recipient_name: none
@@ -1584,13 +1632,19 @@
1584# The database user which will be used by this process.1632# The database user which will be used by this process.
1585# datatype: string1633# datatype: string
1586dbuser: queued1634dbuser: queued
1635storm_cache: generational
1636storm_cache_size: 500
15871637
15881638
1589[binaryfile_expire]1639[binaryfile_expire]
1590dbuser: binaryfile-expire1640dbuser: binaryfile-expire
1641storm_cache: generational
1642storm_cache_size: 500
15911643
1592[generateppahtaccess]1644[generateppahtaccess]
1593dbuser: generateppahtaccess1645dbuser: generateppahtaccess
1646storm_cache: generational
1647storm_cache_size: 500
15941648
1595[vhosts]1649[vhosts]
1596# When true, use https URLs unless explicitly overridden.1650# When true, use https URLs unless explicitly overridden.