html5browser js test suites are too sensitive to test length

Bug #875697 reported by Gary Poster
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
HTML5-browser
Fix Released
Undecided
Gary Poster
Launchpad itself
Fix Released
High
Gary Poster

Bug Description

We can only specify a per-suite timeout limit for html5browser tests. This means that test suites must be artificially divided, and it means that differences in machine speed aggregate per test making timeouts harder to predict. A real-world example is a set of tests that pass within the timeout on a developer box and on ec2, but take too long on buildbot.

As agreed with Curtis, we will make html5browser support incremental timeouts, and change our javascript testrunner to communicate when each test passes or fails. Then we can say, for instance, that a timeout for a yuixhr suite is 5 minutes (a limit we never expect to hit), but timeout for a single yuixhr test is somewhere between 6 and 10 seconds. We can do smaller or identical timeouts for pure-JS tests.

A related problem is that html5browser takes a long time to spin up on some boxes (64 bit, in Curtis' experience), adding 15 to 20 seconds per suite. Further work might be to solve this problem, and/or to make html5browser a service that LP starts with a layer and communicates with for each test. We won't try to solve that for this bug.

Related branches

Revision history for this message
Gary Poster (gary) wrote :

This blocks landing the work for high bug 872089 which itself blocks landing the work for critical bug 724609.

Changed in html5-browser:
assignee: nobody → Gary Poster (gary)
Changed in launchpad:
assignee: nobody → Gary Poster (gary)
status: New → In Progress
Changed in html5-browser:
status: New → In Progress
Changed in launchpad:
importance: Undecided → High
summary: - html5browser js tests are too sensitive to test length
+ html5browser js test suites are too sensitive to test length
Revision history for this message
Launchpad QA Bot (lpqabot) wrote :
tags: added: qa-needstesting
Changed in launchpad:
status: In Progress → Fix Committed
William Grant (wgrant)
tags: added: qa-untestable
removed: qa-needstesting
Changed in launchpad:
status: Fix Committed → Fix Released
Gary Poster (gary)
Changed in html5-browser:
status: In Progress → Fix Released
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.