Regression: syncing packages with UTF-8 changelogs fails

Bug #635591 reported by Colin Watson
16
This bug affects 2 people
Affects Status Importance Assigned to Milestone
Launchpad itself
Fix Released
High
Jelmer Vernooij

Bug Description

lp_archive@cocoplum:~/syncs$ sync-source.py -b cjwatson -S experimental dctrl-tools
Getting binaries for maverick...
[Updating] dctrl-tools (2.14 [Ubuntu] < 2.15 [Debian])
 * Trying to add dctrl-tools...
2010-09-11 07:53:51 INFO - <dctrl-tools_2.15.tar.gz: downloading from http://ftp.debian.org/debian/>
2010-09-11 07:53:51 INFO - <dctrl-tools_2.15.dsc: downloading from http://ftp.debian.org/debian/>
I: dctrl-tools [main] -> dctrl-tools_2.14 [main].
Traceback (most recent call last):
  File "/srv/launchpad.net/codelines/current/scripts/ftpmaster-tools/sync-source.py", line 940, in <module>
    main()
  File "/srv/launchpad.net/codelines/current/scripts/ftpmaster-tools/sync-source.py", line 936, in main
    do_diff(Sources, Suite, origin, arguments, current_binaries)
  File "/srv/launchpad.net/codelines/current/scripts/ftpmaster-tools/sync-source.py", line 725, in do_diff
    Options.requestor, origin, Suite, current_binaries)
  File "/srv/launchpad.net/codelines/current/scripts/ftpmaster-tools/sync-source.py", line 673, in add_source
    current_sources, current_binaries)
  File "/srv/launchpad.net/codelines/current/scripts/ftpmaster-tools/sync-source.py", line 495, in import_dsc
    origin)
  File "/srv/launchpad.net/codelines/current/scripts/ftpmaster-tools/sync-source.py", line 175, in generate_changes
    changes += changelog
UnicodeDecodeError: 'ascii' codec can't decode byte 0xc3 in position 921: ordinal not in range(128)

The first time I tried this, my locale was C. I tried setting it to en_GB.UTF-8 (which exists on cocoplum), but that made no difference.

The relevant line from the changelog is this:

  * French translation updated by David Prévot (closes: #593990)

I've checked that this is correctly encoded in UTF-8 as 0xC3 0xA9:

lp_archive@cocoplum:~/syncs$ tar xzOf dctrl-tools_2.15.tar.gz dctrl-tools-2.15/debian/changelog | head -n35 | grep 'French translation' | hd
00000000 20 20 2a 20 46 72 65 6e 63 68 20 74 72 61 6e 73 | * French trans|
00000010 6c 61 74 69 6f 6e 20 75 70 64 61 74 65 64 20 62 |lation updated b|
00000020 79 20 44 61 76 69 64 20 50 72 c3 a9 76 6f 74 20 |y David Pr..vot |
00000030 28 63 6c 6f 73 65 73 3a 20 23 35 39 33 39 39 30 |(closes: #593990|
00000040 29 0a |).|
00000042

I've marked this as a regression in the bug summary. While I'm not absolutely certain about this, Debian changelogs have been specified as UTF-8 for several years now and UTF-8 is often used in translation credits in changelogs, so I find it implausible that we wouldn't have run across this before now. However, none of the recent changes to sync-source.py seem to be likely candidates. Perhaps this is something to do with the recent upgrade of cocoplum to lucid?

Tags: lp-soyuz qa-ok

Related branches

Jelmer Vernooij (jelmer)
Changed in soyuz:
assignee: nobody → Jelmer Vernooij (jelmer)
importance: Undecided → High
status: New → Triaged
Revision history for this message
Colin Watson (cjwatson) wrote :

This is also blocking bug 637640.

Revision history for this message
Colin Watson (cjwatson) wrote :
Jelmer Vernooij (jelmer)
tags: added: qa-bad
Revision history for this message
Jelmer Vernooij (jelmer) wrote :

A fix for the QA issue is in ec2 at the moment. I've already re-QA'ed the branch with that fix on dogfood, and put up a branch for RC review:

https://code.edge.launchpad.net/~jelmer/launchpad/production-cp-635591/+merge/35748

Jelmer Vernooij (jelmer)
tags: added: qa-ok
removed: qa-bad
Revision history for this message
Jelmer Vernooij (jelmer) wrote :

A quick update.

The QA issue fix has landed on devel, the cherrypick is now in PQM waiting to land on production-devel.

Revision history for this message
Launchpad QA Bot (lpqabot) wrote : Bug fixed by a commit
Changed in soyuz:
milestone: none → 10.10
tags: added: qa-needstesting
removed: qa-ok
Changed in soyuz:
status: Triaged → Fix Committed
Ursula Junque (ursinha)
tags: added: qa-ok
removed: qa-needstesting
Changed in soyuz:
status: Fix Committed → Fix Released
Revision history for this message
Colin Watson (cjwatson) wrote :

Fix confirmed here. Thanks!

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.