edge rollouts broke CSS

This is a one-time thing, per my email to the list.

If there's a bug, then it's me not informing the LOSAs clearly enough, for you all to be prepared and to give me feedback before you encountered it.

I think that's a policy problem, rather than a bug. I'm going to close this bug, and invite your feedback on the policy problem.

I'd like to disagree (for the moment) that this is purely a policy bug. We sync from a clean tree, run "make bin/buildout && bin/buildout" and then rsync the code to the edge servers. On the frontend servers we then run "make build", while on the appservers we run "make compile". Things worked fine on the appservers, it was only on the frontend servers that we ran into problems.

I tried a simple experiment (make a new branch, ``make bin/buildout && bin/buildout``, then ``make build``) and it worked, so it means that the basic mechanism is fine. Do you have the original files and the post ``make clean && make build`` files? If so, I'd like to compare _pythonpath.py and parts/scripts/site.py before and after (paths are from root of LP tree).

On Fri, 2010-04-09 at 13:41 +0000, Gary Poster wrote:
> I tried a simple experiment (make a new branch, ``make bin/buildout &&
> bin/buildout``, then ``make build``) and it worked, so it means that the
> basic mechanism is fine. Do you have the original files and the post
> ``make clean && make build`` files? If so, I'd like to compare
> _pythonpath.py and parts/scripts/site.py before and after (paths are
> from root of LP tree).

Ok, so these are the files before we run bin/buildout:

https://pastebin.canonical.com/30371/

And here they are on the deployment server after we've done that:

https://pastebin.canonical.com/30372/

And here they are on one of the frontends that failed after running make
clean; make build:

https://pastebin.canonical.com/30370/

Thanks, Tom

OK, that clarifies the issue. The reason devs don't see this is because we don't build on one path and deploy on another. The heart of the problem is that the original build on the deployment machine has this line

site_dir = "/home/pqm/for_rollouts/edge/parts/scripts"

and the build on the front end (make clean && make build) has this line:

site_dir = "/srv/edge.launchpad.net/edge/launchpad-rev-10662/parts/scripts"

I'd like to transfer this discussion to a faster medium like IRC or phone, but the thoughts on my mind are these:

- Do you already rsync /home/pqm/for_rollouts/edge/lib and /home/pqm/for_rollouts/edge/eggs, or symlink them somehow on the ?
- Can we build on the deployment box using the same path that will be used on the frontend box?
- Buildout has an option to use relative paths which would solve this, I think. Right now, we can't use it because of some technical reasons. I can fix that, and I think it would solve the problem. I suspect that's your winning option, if I can get it to work :-)
- Other solutions are possible but the ones I can think of would involve significantly more effort, so they are not compelling.

I'll ask on IRC if we can get on the phone to nail this down. I want to make sure I understand the current process before I proceed.

Tom and I talked on the phone, and I agreed I would make the buildout change.

As a quick hack fix, replacing make build with the following works locally and could be temporarily put into the deployment scripts:

rm bin/buildout && rm -rf bin && rm _pythonpath.py && make bin/buildout && bin/buildout && make build

I suspect that make compile is now working by chance rather than design, so the same kind of thing (rm bin/buildout && rm -rf bin && rm _pythonpath.py && make bin/buildout && bin/buildout && make compile) might be appropriate there.

I plan to have the proper fix for this done today. I leave it to Tom as to whether he wants to incorporate the quick hack fix so edge gets updates over the weekend.

Gary

On Fri, 2010-04-09 at 15:37 +0000, Gary Poster wrote:
[...]
> I plan to have the proper fix for this done today. I leave it to Tom as
> to whether he wants to incorporate the quick hack fix so edge gets
> updates over the weekend.

This does seems a bit convoluted. I've done a manual update today, but I
think we'll keep these disabled until we get the proper fix, especially
if it could affect make compile as well.

Thanks, Tom

I didn't make it today. I'm lowering this to High. I'll return to this Monday.

I modified the deployment configs per https://pastebin.canonical.com/30676/ and then the deployment failed with the following error: https://pastebin.canonical.com/30675/

First thought, from before I saw the error: I think we need to have a mid-term priority of making a basic deployment pattern that can be repeated easily by devs. It's not fair to you all to have to do things that break. It's not fair to devs like me who have to touch things like this that I can't test them myself. It's not fair to all devs that the problem holds up critical roll-outs.

More importantly for the moment, I think the subunit error is a completely separate problem, and related to jml's recent work. I suspect that the machine needs to update its launchpad-dependencies, or launchpad-dependencies has not been properly maintained. I will investigate.

Confirmed: https://pastebin.canonical.com/30675/ is a separate problem. subunit should be in launchpad-dependencies, not launchpad-developer-dependencies. I'm working on that.

lp.testing should not be imported in production code. That way we don't need subunit in launchpad-dependancies. I'm fixing this.

Fixed in stable r10734 <http://bazaar.launchpad.net/~launchpad-pqm/launchpad/stable/revision/10734>

Fixed in stable r10738 <http://bazaar.launchpad.net/~launchpad-pqm/launchpad/stable/revision/10738>

I've added in a "make clean" before "make build/compile" on the edge rollouts and that seems to work. So I've re-enabled edge updates. Does the "fix committed" status mean we no longer need to do this, and everything should work fine with our normal deployment steps?

No, that was an automatic update because I linked to this bug on my branch, which only fixed the "other" problem discovered once the work-around is in place. Gary is finishing off the proper fix which will allow us to remove the work-around.

I landed a branch yesterday that, to the best of my ability to QA locally, should fix the problem in a good way. Please remove the ``make clean`` before the ``make build`` and let's see if it works.

On Thu, 2010-04-22 at 15:43 +0000, Gary Poster wrote:
> I landed a branch yesterday that, to the best of my ability to QA
> locally, should fix the problem in a good way. Please remove the ``make
> clean`` before the ``make build`` and let's see if it works.

I've done this and forced an edge rollout that seemed to work fine, so I
think we're in good shape.

Thanks, Tom

Fixed in stable r10761 <http://bazaar.launchpad.net/~launchpad-pqm/launchpad/stable/revision/10761>