[odf-discuss] Three kinds of tests

[robert_weir at us.ibm.com]

odf-discuss-bounces at opendocumentfellowship.com wrote on 05/06/2008 
05:02:52 PM:


> 
> Something like Acid tests for ODF sounds really good to me. That is, 
> having a single document that tests many features. However, I note that 
> Rob Weir seems to prefer a different approach, where there are many 
> documents and each document tests one feature and one feature only. I 
> don't know enough about conformance testing to have an opinion on which 
> is better. I would love to hear from Rob as to why he prefers the 
> approach he prefers.
> 

I see three kinds of tests:

1) Atomic tests that exercise just a single features or small set of 
related features, like text alignment.  These are good because they allow 
you to measure coverage, both of the standard and the implementation.  It 
allows you to say "The 48 text attribute tests in the suite cover 85% of 
the ODF features in this area" or "OpenOffice Write passes 79% of the text 
attribute test cases".  In other words, if you aim for 100% specification 
coverage (which I hope a test suite would target) then you can be 
quantitative about conformance.  This approach also lends itself to 
incremental work, to prioritization, and to updating with new ODF 
releases.

2) The ACID type tests are great for PR.  They give you an immediate, 
visual feel for how conformance an application is in a way that a number 
does not always do.  They are great for groups like the Web Standards 
Group who want to push vendors to increase their conformance level.  The 
public and the press can interpret a test like this easily.  But it lacks 
some of the advantages of the atomic test approach. 

3) Interoperability tests.  These are based on real world documents that 
use a combination of features to solve a real business problem.  The goal 
here is to deal with real world complexity and see how multiple vendors 
handle it.  This is typically an event-based kind of test, sometimes 
called a "plugfest".  This is what we did in Barcelona at the 
interoperability workshop.  The goal is to really "find the unexpected". I 
don't think the other two types of tests really find the complex cases. 
Customers certainly do.  So you need to test like customers use the 
product, which means doing dumb things as well.

I think all three types of tests are important for interoperability. 
Strictly speaking, only the first is a conformance test, since it is the 
only one that is aiming to test all requirements of the standard.  But I 
think there is a good argument that we should be advancing on all three 
fronts.

Regards,

-Rob

-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://lists.opendocumentfellowship.com/pipermail/odf-discuss/attachments/20080506/ae9b2729/attachment.htm