comp.software.testing Frequently Asked Questions (FAQ)@import 'http://faqs.org/abstracts/css/default.css';@import 'http://faqs.org/search.css';[ Usenet FAQs | Search | Web FAQs | Documents | RFC Index ] Search the FAQ Archives MultiPagecomp.software.testing Frequently Asked Questions (FAQ)There are reader questions on this topic!Helpothers by sharing your knowledgeFrom: raymond.rivest@crim.ca (Raymond Rivest)Newsgroups: comp.software.testing, comp.software-engSubject: comp.software.testing Frequently Asked Questions (FAQ)Summary: read me before posting to comp.software.testingKeywords: FAQ, software testing, software engineering, archives, conferences, books, periodicals, products, organizationsMessage-ID: <70Q_9.29065$LR3.1246006@charlie.risq.qc.ca>Date: Sat, 01 Feb 2003 13:32:51 GMTArchive-name: software-eng/testing-faqPosting-Frequency: monthlyLast-modified: 2002/08/06Version: $Revision: 3.9 $Maintainer: cst-faq@crim.caURL: http://www.faqs.org/faqs/software-eng/testing-faq/ http://www.crim.ca/ctl/cst.FAQ.html http://www.cigital.com/c.s.t.faq.html comp.software.testing FAQ and information resourceThis is the Frequently Asked Questions list for comp.software.testing.This FAQ is cross-posted to comp.software-eng. Please don't hesitate to send corrections, additions, and other comments to cst-faq@crim.ca.Be sure to use the tag of FAQ: in the subject line, for example,"Subject: FAQ: Modification request...."The latest version of this FAQ may be retrieved using Usenet, the WorldWide Web, ftp, email, gopher, or even telnet. See the FAQ "Introductionto the *.answers newsgroups" in news.announce.newusers or<ftp://rtfm.mit.edu/pub/usenet/news.answers/news-answers/introduction>for details. Please check the date above - if this file is more than amonth old, it is obsolete. The main ftp site for the FAQ is<ftp://rtfm.mit.edu/pub/usenet/news.answers/software-eng/testing-faq>.The folks at CRIM Software Test Center sponsor a web sitefor the FAQ at http://www.crim.ca/ctl/cst.FAQ.html.The folks at Cigital sponsor a web site for the FAQ, at<http://www.cigital.com/c.s.t.faq.html>.This file is Copyright 2001, 2002 by Raymond Rivest. Danny R. Faught is theoriginal author. Permission is granted to distribute this file uneditedand in its entirety, provided the "Date" header is no more than one monthearlier than the date of distribution. This information is providedwithout any express or implied warranty.Subject: 1. Table of Contents 1. Table of Contents 2. Change Log 3. What is this newsgroup about? 4. Netiquette 5. Beta testing 6. What other FAQs are relevant? 7. What other resources are available? 8. World Wide Web resources 9. What's the difference between QA and testing? 10. I'm looking for a test tool... 11. How do I find information about testing object-oriented programs? 12. How do I test web-related applications? 13. What is black box/white box testing? 14. What are unit, component and integration testing? 15. What's the difference between load and stress testing ? 16. Should we discuss bug tracking tools in this newsgroup? 17. What kind of salary should a tester make? 18. Where can I find sample test plans? 19. What is the best tester to developer ratio? 20. Conferences 21. Periodicals 22. Books 23. Bibliographic resources 24. Organizations 25. ContributorsSubject: 2. Change LogThis section will highlight notable additions, deletions, and changes tothe FAQ.Changed in version 3.:o Some minor cosmetic changes.o Added CRIM STC in section 24.Changed in version 3.8:o Added Extreme Programming mini-FAQ link in section 6.Changed in version 3.7:o Finally cleaned up Danny's e-mail to tejasconsulting :)o Removed "The Outlook" is no longer available from McCabe? (subject 22).o Added a commercial newsletter to subject 22: The Payne Report, http://www.cigital.com/paynereport/.o Changed the comp.software.config-mgmt url to http://www.daveeaton.com/scm/PMTools.html.o Added http://www.QALinks.com and http://www.TestingFAQs.org in subject 8.o Added Die Software Testing Website http://www.softwaretesting.de in suject 8.o Added LogiGear web page at http://www.qacity.com/ in subject 8.o Added Tejas Software Consulting Newsletter in subject 21.o Added a new subject tag in section 4 (netiquette) for job searcher (CV:).o Added a link to http://www.xprogramming.com/software.htm in subject 7 about eXtreme programming and testing.o Added two references about security testing in subject 7.o Added Quality Web Systems by Elfriede A. Dustin/Douglas McDiarmid in subject 12.o Updated a broken link for http://www.asq.org/pub/ in subject 21o Added a quick reference defining what is a "smoke test" in subject 7.Changed in version 3.6:o Changed reference from www.dejanews.com to groups.google.com. Google acquired dejanews !Changed in version 3.5:o Inserted a new Section 15 and moved the rest downward. What's the difference between load and stress testing ?Changed in version 3.4:o Reordered the Table of Content. Section 7 to 4, 10 to 5 and moved the rest downward to explain the purpose of comp.software.testing from the start. o Changed references of Danny Faught's FAQs from his home page to http://www.testingfaqs.org/ in Section 10 (a forgotten link). o Added TESTCOM conference to Section 19.o Added reference to the defect tracking tool FAQ in Section 15.o Added StickyMinds.com in Section 8.o Added a book reference in Section 12.o Added the StickyMinds.com template section in Section 17o Added a reference to comp.software.measurement newsgroup in Section 7.Changed in version 3.3:o Added the mention of subject tag for cst-faq alias because of spams.o Changed references of Danny Faught's FAQs from his home page to http://www.testingfaqs.org/o Added reference to BetaSoft's QAForums in section 5.Changed in version 3.2:o Changed references of Danny Faught's FAQs to his home page.o Changed Danny Faught's email.o Added LatinStar 2001 conference.o Changed STQE booklist to stickyminds.o Added a new URL for the html FAQ.Changed in version 3.1:o Changed references of cst-faq@cigital.com to cst-faq@crim.ca.Changed in version 3.0:o Head FAQ maintainer responsibilities transferred from Danny Faught to Raymond Rivest. Have fun with it, Raymond!Changed in version 2.7:o Delegated some items to Cigital's SRM Hotlist, so I don't have to maintain them in two different places. This includes the list of mailing lists (subject 5), web pointers and dates for conferences (subject 19), and the entire book list (subject 21). This will also help to make the FAQ somewhat less bulky.o Removed the Triangle Quality Council, which is being phased out. Also removed Association for Software Testing Australia, which no longer seems to exist. (subject 23)Changed in version 2.6:o Reliable Software Technologies is now Cigital, Inc. Changed all RST references to Cigital.o Reflected the change from American Programmer to Cutter IT Journal. (subject 20)o Removed information about the cst-improve list, which is no longer in use. (subject 5)o Updated contact information for Software Practitioner. (subject 20)Subject: 3. What is this newsgroup about?If you're new to Usenet, please read through the FAQs innews.announce.newusers and hang out in news.newusers.questions for awhile before you consider posting.If you do not know how to subscribe to comp.software.testing, youfirst need to get Usenet access from an Internet service provider oryour company, and you need newsreader software. Different serviceproviders support different newsreaders, so there is no generalanswer. Ask your service provider or local system administrator tohelp. If all else fails, use a web browser to access Usenet at<http://groups.google.com/groups?hl=en&lr=&safe=off&group=comp.software.testing>.The original charter for comp.software.testing can be found at<http://tsunami.jpl.nasa.gov/TEL/docs/cst-charter.html>. Here's anexcerpt (with corrections): CONTENTS: For the purpose of this discussion, we will assume that a system is composed of hardware, software, people, and procedures. The proposed group should be chartered to include discussions characterized by the following list of subjects, compiled from preliminary discussions: system test automation system test planning system test optimization (e.g. minimize regression testing) test tools test metrics measurement technology nature of testing under various development models (e.g. object-oriented, real time, etc...) testing in a rapid prototyping environment (i.e. sans spec) relationship of various classes/types of tests to requirements, ... conference and symposium announcements the definition of "software testing" and its relationship to SQA and debugging. requirements traceability risk-based testing the most useful form of specifications and functional requirements from the tester's point of view. testing techniques, e.g., structured testing using control flowgraphs and basis path testing, equivalence class partitioning, boundary value analysis, cause-effect graphing, path predicate testing, data flow testing, program slices, data object state and usage analysis, data flow anomaly analysis, and sensitivity analysis. test coverage criteria, e.g., statement (C0), path, branch (C1), module (S0), and call-pair (S1) coverage criteria. All levels of testing for both hardware and software should be considered appropriate subject matter. While it is likely that the group will take on a software flavor early on, we should try to generate interest and input from people with hardware and system testing experience and perspective. It is especially important that we foster discussion of *system* level testing issues, since this is the weakest area of our collective knowledge.Subject: 4. NetiquetteDid I mention that you should read the FAQs in news.announce.newusers ifyou haven't done so lately? Also, you can reduce the chances ofembarrassing yourself by reading comp.software.testing for a whilebefore posting. At the very least, scan all the articles in the groupcurrently on your server to make sure your question hasn't already beenasked and answered.Please help the newsgroup's participants find articles in the categoriesthey're interested in. If your posting falls into the categories below,please put the three-letter tag at the beginning of your subject line.This will also increase the likelihood that those wanting to read yourpost will find it. For example, your subject line might read -"Subject: ADV: Widget Tester 6.0 released". JOB: Help wanted ads ANN: Announcements for new books, publications CFP: Call for papers or participants CNF: Conference announcements ADV: Other advertisements and commercial product announcements CV: Curriculum Vitae, looking for a job Job postings that are cross-posted to the various job-relatednewsgroups should not be posted to comp.software.testing, since thesubject conventions for those groups do not allow them to be easilyidentified as job postings, and the misc.jobs FAQ(<ftp://rtfm.mit.edu/pub/usenet/news.answers/jobs/welcome-faq>)specifies that job postings should not be cross-posted between thejob-related newsgroups and non job-related newsgroups.Some recruiters refuse to follow the job posting policy. If you don'twant to read job postings, consider using a kill file tosystematically kill articles from these recruiters.Many questions about a particular test tool only interest those readerswho use that tool. Please make sure to put the name of the test toolin the subject line so readers can select the articles they read bylooking at the subject. In general, please use a descriptive subjectline.If you post an article and later decide you shouldn't have posted it,please cancel it. It is much better to correct the problem than tosend *another* post to apologize for the first one. Situations wherethis would be appropriate include posting empty articles, multiplecopies of the same article, and any goof where you'd like to correctsomething you posted shortly after it was sent. Look in yournewsreader documentation for help with cancelling an article; some alsohave a "supercede" feature which makes it easy to correct a posting.While the cancel feature has been common for quite some time, somenewer newsreaders unfortunately do not offer it. If this is the case,complain to the author, and consider using a more reasonablenewsreader. Here's a manual method to cancel an article - start tofollow up to the errant posting. Change the subject to"cmsg cancel <message-id>", where message-id is taken from theimportant. Post it, and the news system should interpret it as acancel request rather than a real posting. You can only do this foran article you posted yourself.People have also expressed concern about postings that are totallyoff-topic. These postings are typically "spam" postings that go to mostof the thousands of newsgroups that exist. There is very little that anunmoderated group can do proactively to prevent them. Seenews.admin.net-abuse.usenet and<http://www.tezcat.com/~gbyshenk/ive.been.spammed.html> for moreinformation.Please don't post test messages to comp.software.testing. This groupis for discussing software testing, not for testing your newssoftware. If you want several automatic confirmations that your postworked, post a test message to misc.test. There are other test groupsat various levels of the news hierarchy that all end in ".test". It'sbest to try a local one first.Subject: 5. Beta testingDiscussions about managing a beta test program are within the scope ofthis newsgroup. However, companies looking for beta testers, andaspiring beta testers looking for something to test, should go tocomp.sources.testers instead. A possible exception is test toolvendors looking for beta testers.If you are looking for a career in software testing,comp.software.testing will welcome you. Just don't call it "betatesting". Beta testing is typically conducted by end users of asoftware product who are not paid a salary for their efforts.Subject: 6. What other FAQs are relevant?Danny Faught maintains the following FAQs, originally created by BrianMarick: Testing Contractors and Consultants List Testing Courses List Testing Tool Supplier ListThese FAQs are available at <http://www.testingfaqs.org/>.Please consult the appropriate FAQ before asking questions about theinformation already contained therein. The Testing Tool Suppliers Listalso contains a nice list of the available software test tools. If yousee anything that needs to be changed or added to these FAQs, pleasecontact Danny Faught at faught@tejasconsulting.com so that they may continue to be useful to everyone.Since software testing is a subdiscipline of software engineering,you will probably find the FAQs posted to comp.software-eng useful.They can be found at <ftp://rtfm.mit.edu/pub/usenet/comp.software-eng/> or<http://www.qucis.queensu.ca/Software-Engineering/>The Problem Management Tools (bug tracking) FAQ is posted tocomp.software.config-mgmt, and is also available on news.answers andthe rtfm archives. The URL is<http://www.daveeaton.com/scm/PMTools.html>.Rick Hower maintains the "Software QA/Test Resource Center" at<http://www.softwareqatest.com>. It covers a broad range ofquestions about software testing.Related to new trends, eXtreme Programming, here's the mini-FAQ:<http://www.jera.com/techinfo/xpfaq.html>.Subject: 7. What other resources are available?The public comp.software.testing archives and the mailing list gateway are no longer available. As an alternative, try Google groups at <http://groups.google.com/groups?hl=en&lr=&safe=off&group=comp.software.testing>Comp.software.testing articles are archived back to March, 1995.If you're interested in quality measures (metrics), than <news:comp.software.measurement>will interest you. However, I didn'tfind any FAQ for this newsgroup.If eXtreme testing is of interest, than<http://www.xprogramming.com/software.htm> will interest you.See also Ronald E. Jeffries's article at <http://www.xprogramming.com/publications/SP99%20Extreme%20for%20Web.pdf>http://www.cigital.com/hotlist/publications-ml.html>.BetaSoft hosts Web forums on most popular testing tools and techniquesat <http://www.qaforums.com/>.A hot topic is network security. Here are two pretty good references for testing.<http://www.ideahamster.org/> <http://www.itprc.com/security.htm>A quick reference explaining what is a "smoke test" can be found at<http://www.construx.com/stevemcc/bp04.htm>.Subject: 8. World Wide Web resourcesHere are a few web sites that each try to connect you to much of thesoftware testing and quality information that is available on the web. The Cigital Software Risk Management Hotlist <http://www.cigital.com/hotlist/>http://www.mtsu.edu/~storm>http://www.soft.com/Institute/HotList/> Kerry Zallar's software testing page <http://www.testingstuff.com/> Bret Pettichord's Software Testing Hotlist <http://www.io.com/~wazmo/qa/> StickyMinds.com Software Testing Area <http://www.StickyMinds.com/> Betasoft QA Links page <http://www.QALinks.com> Danny Faught's testing faqs page <http://www.TestingFAQs.org> Die Software Testing Website <http://www.softwaretesting.de> This website has lots of articles, book reviews, software testing related software for download and a discussion forum. Most of the website is in german language. All contents can be accessed free of charge. LogiGear's QACity.Com : Resources for Busy Testers <http://www.qacity.com/>Subject: 9. What's the difference between QA and testing?Sharon Codrington asks: Please enlighten this young QA person as to the difference between QA & Testing. Am I right in thinking that QA is more a preventive thing, ensuring quality in the company and therefore the product rather than just testing the product for software bugs?And Bret Pettichord answers: DING. DING. DING. You are correct. Most testing jobs I see are nevertheless advertised as "QA". Some people suggest that QC is a better set of initials to describe testing.Don Mills writes: In my courses and my practice, I stick to the ANSI/IEEE definitions, which agree with what is generally held *outside* the software arena. The definitions boil down to: * TESTING means "quality control" * QUALITY CONTROL measures the quality of a product * QUALITY ASSURANCE measures the quality of processes used to create a quality product.Rick Hower addresses this question at<http://www.softwareqatest.com/qatfaq1.html>.Subject: 10. I'm looking for a test tool...Did you read the Testing Tool Suppliers FAQ at<http://www.testingfaqs.org/>? It has a list of many of theavailable tools, plus a list of the commercial tools listings. Whileevery question about test tools isn't answered there, it's a good start,and it'll at least allow you to ask a more directed question if you dopost a question about test tools.When you do post a tools question, you should expect a few "check outmy company's tool" responses from a few vendors. This rarely givesyou any more information than is available in the tools FAQ, and whichvendors respond at any given time varies widely. Often you'll find avendor trying to drum up interest by posting the same thing inresponse to every current article that mentions tools. Keep in mindthat your choices are much broader than those that happen to getposted.Subject: 11. How do I find information about testing object-orientedprograms?Based on information originally provided by Brian Marick:Books that include some discussion of testing OO programs: Binder, _Testing Object-Oriented Systems_ Jacobson, _Object-Oriented Software Engineering_ Marick, _The Craft of Software Testing_ McGregor, _Object-Oriented Software Development_ Siegel, _Object Oriented Software Testing_There are a vast number of articles about testing OO programs. Isuggest starting with the September 1994 Communications of the ACM,which is devoted to this topic. Chase references. The February 1996issue of Object has a piece on system testing with use cases.Conferences like STAR and Quality Week and Pacific Northwest SoftwareQuality Conference seem to always have papers on this topic. Isuggest going to a conference and buttonholing people.Web sites with discussion: <http://www.cs.washington.edu/homes/gmurphy/testSTApp.html> <http://www.testing.com/> <http://www.toa.com>Courses on testing object-oriented code (one by Robert Binder,one by Ed Berard) are described in the Testing Courses FAQ. Also,a new addition is an offering from Software Quality Engineering.Subject: 12. How do I test web-related applications?Rick Hower maintains a web page which includes a nice list of webtesting tools, and some guidelines for testing web sites.<http://www.softwareqatest.com/qatweb1.html> Please contact him atrickhower@earthlink.net if you can offer additions or corrections to thelist.One book that can be of help, "Testing Applicationson the Web" by Hung Q. Nguyen and "The Web Testing Handbook" by Steven Splaine and Stefan P. Jaskiel.One recent addition is Quality Web Systems by Elfriede A. Dustin/Douglas McDiarmid.<http://www.StickyMinds.com/books.asp>It also contains a Tool Evaluation appendix, comparing themajor tools (Mercury, Rational, Empirix, Compuware, Segue).This matrix consists of approx. 30 pages of detailedevaluation criteria.Subject: 13. What is black box/white box testing?Black-box and white-box are test design methods. Black-box test designtreats the system as a "black-box", so it doesn't explicitly useknowledge of the internal structure. Black-box test design is usuallydescribed as focusing on testing functional requirements. Synonyms forblack-box include: behavioral, functional, opaque-box, andclosed-box. White-box test design allows one to peek inside the "box",and it focuses specifically on using internal knowledge of the softwareto guide the selection of test data. Synonyms for white-box include:structural, glass-box and clear-box.While black-box and white-box are terms that are still in popular use,many people prefer the terms "behavioral" and "structural". Behavioraltest design is slightly different from black-box test design becausethe use of internal knowledge isn't strictly forbidden, but it's stilldiscouraged. In practice, it hasn't proven useful to use a single testdesign method. One has to use a mixture of different methods so thatthey aren't hindered by the limitations of a particular one. Some callthis "gray-box" or "translucent-box" test design, but others wish we'dstop talking about boxes altogether.It is important to understand that these methods are used during thetest design phase, and their influence is hard to see in the tests oncethey're implemented. Note that any level of testing (unit testing,system testing, etc.) can use any test design methods. Unit testing isusually associated with structural test design, but this is becausetesters usually don't have well-defined requirements at the unit levelto validate.Definitions of these terms can be found in these references:Beizer, _Black Box Testing_, p. 8.Beizer, _Software Testing Techniques_, pp. 10-11.Daich, et al., _STSC Software Test Technologies Report_, pp. 198, 206.Kaner, et.al., _Testing Computer Software_, Second Edition, pp. 41-43.Mosley, _The Handbook of MIS Application Software Testing_, pp. 68-76.Myers, _The Art of Software Testing_, pp. 8-11.Wilson, _Unix Test Tools and Benchmarks_, pp. 298, 306Subject: 14. What are unit, component and integration testing?The following definitions are from a posting by Boris Beizer onthe topic of "integration testing" in the c.s.t. newsgroup. The definitions of integration tests are after Leung and White. Note that the definitions of unit, component, integration, and integration testing are recursive: Unit. The smallest compilable component. A unit typically is the work of one programmer (At least in principle). As defined, it does not include any called sub-components (for procedural languages) or communicating components in general. Unit Testing: in unit testing called components (or communicating components) are replaced with stubs, simulators, or trusted components. Calling components are replaced with drivers or trusted super-components. The unit is tested in isolation. component: a unit is a component. The integration of one or more components is a component. Note: The reason for "one or more" as contrasted to "Two or more" is to allow for components that call themselves recursively. component testing: the same as unit testing except that all stubs and simulators are replaced with the real thing. Two components (actually one or more) are said to be integrated when: a. They have been compiled, linked, and loaded together. b. They have successfully passed the integration tests at the interface between them. Thus, components A and B are integrated to create a new, larger, component (A,B). Note that this does not conflict with the idea of incremental integration -- it just means that A is a big component and B, the component added, is a small one. Integration testing: carrying out integration tests. Integration tests (After Leung and White) for procedural languages. This is easily generalized for OO languages by using the equivalent constructs for message passing. In the following, the word "call" is to be understood in the most general sense of a data flow and is not restricted to just formal subroutine calls and returns -- for example, passage of data through global data structures and/or the use of pointers. Let A and B be two components in which A calls B. Let Ta be the component level tests of A Let Tb be the component level tests of B Tab The tests in A's suite that cause A to call B. Tbsa The tests in B's suite for which it is possible to sensitize A -- the inputs are to A, not B. Tbsa + Tab == the integration test suite (+ = union). Note: Sensitize is a technical term. It means inputs that will cause a routine to go down a specified path. The inputs are to A. Not every input to A will cause A to traverse a path in which B is called. Tbsa is the set of tests which do cause A to follow a path in which B is called. The outcome of the test of B may or may not be affected. There have been variations on these definitions, but the key point is that it is pretty darn formal and there's a goodly hunk of testing theory, especially as concerns integration testing, OO testing, and regression testing, based on them. As to the difference between integration testing and system testing. System testing specifically goes after behaviors and bugs that are properties of the entire system as distinct from properties attributable to components (unless, of course, the component in question is the entire system). Examples of system testing issues: resource loss bugs, throughput bugs, performance, security, recovery, transaction synchronization bugs (often misnamed "timing bugs").Subject: 15. What's the difference between load and stress testing ?Boris Beizer says:> One of the most common, but unfortunate misuse of terminology>is treating "load testing" and "stress testing" as synonymous. The >consequence of this ignorant semantic abuse is usually that the system>is neither properly "load tested" nor subjected to a meaningful stress>test.>>1. Stress testing is subjecting a system to an unreasonable load >while denying it the resources (e.g., RAM, disc, mips, interrupts, >etc.) needed to process that load. The idea is to stress a system to>the breaking point in order to find bugs that will make that break >potentially harmful. The system is not expected to process the >overload without adequate resources, but to behave (e.g., fail) in a >decent manner (e.g., not corrupting or losing data). Bugs and failure>modes discovered under stress testing may or may not be repaired>depending on the application, the failure mode, consequences, etc. >The load (incoming transaction stream) in stress testing is often >deliberately distorted so as to force the system into resource >depletion.>>2. Load testing is subjecting a system to a statistically >representative (usually) load. The two main reasons for using such >loads is in support of software reliability testing and in >performance testing. The term "load testing" by itself is too vague>and imprecise to warrant use. For example, do you mean representative>load," "overload," "high load," etc. In performance testing, load is>varied from a minimum (zero) to the maximum level the system can >sustain without running out of resources or having, transactions >suffer (application-specific) excessive delay.>>3. A third use of the term is as a test whose objective is to >determine the maximum sustainable load the system can handle. >In this usage, "load testing" is merely testing at the highest >transaction arrival rate in performance testing.Subject: 16. Should we discuss bug tracking tools in this newsgroup?Discussing bug tracking tools is not within the original charter ofcomp.software.testing. The comp.software.config-mgmt newsgroup seemsto have taken this topic under its wing. That group has a FAQ forproblem management tools. See the "What other FAQs are relevant?"section for details.Danny Faught maintains a FAQ under is wing for defect-tracking toolsat http://www.testingfaqs.org/t-track.htm.Subject: 17. What kind of salary should a tester make?Salary information is available at <http://www.pencom.com/industry.html>and <http://www.computerjobs.com/salary_survey_search.cfm>. A nicesummary of the various surveys that are available is at<http://www.ondaweb.com/sti/salary.htm>.John Tyson writes:> You could check the May '95 issue of Application Development Trends> magazine. The article "Testing moves from purgatory to profession" by> Linda Hayes [lghayes@metronet.com] has an excellent all-around> (non-technical) article on testing and does include some regional> salaries (albeit very general).John also encourages testing consultants to report their rates atJanet Ruhl's Computer Consultant's Resource Page -<http://www.realrates.com>.http://qualityprogress.asq.org/qp/nov97.html>> to order a back issue.Mark McWhinney writes:> You might not want to use the survey results. While they are helpful> for defining an organization's general pay scales, they are not very> useful on a case by case basis. There are too many factors involved> that may push a particular employee's base salary significantly above> or below the average or outside the range.>> For QA and test people, these surveys are even more a problem. QA and> test people are second-class citizens. They are paid less than their> counterparts in development. In part this is due to the fact that QA> and test people tend to be less senior and less experienced, therefore> deserving of less pay. I have no problem with that. However,> developers and testers with equivalent skill, education, and experience> are not paid the same. This is wrong....Subject: 18. Where can I find sample test plans?See the following IEEE standards: 829-1983 IEEE Standard for Software Test Documentation 1008-1987 IEEE Standard for Software Unit Testing 1012-1986 IEEE Standard for Software Verification and Validation Plans 1059-1993 IEEE Guide for Software Verification and Validation PlansYou can contact the IEEE in the US at 800-678-4333. Also, Requisite(800-732-4047) sells a Standards Pack with Microsoft Word Templates for829, 830, and 1012 (part number 3100-05879). The Software ProductivityCenter (<http://www.spc.ca>, 604-662-8181) sells Word templates,including a test plan standard and a test plan template.Several people recommend Kaner's book, _Testing Computer Software_, fortest plan samples. You'll find the ETET test plan in<ftp://ftp.ecs.soton.ac.uk/pub/etet/> under the path"doc/testplan". See <http://www.acomtech.com/testplan.html> for anadaptation of IEEE 829. And visit<http://members.tripod.com/~bazman/frame.html> for a sample system testplan. There is an outline of an IEEE based test plan on<http://www.itc.nrcs.usda.gov/sqa/tst_outline.html>. StickyMinds.com also have a template section for test documents at<http://www.stickyminds.com/swtest.asp?btntopic=tem&Function=Search&topcat=SWTST>.Information in this section was gleaned from posts by: Burt Gearhart,Wayne Woodruff, Brent Parsons, Castor Fu, Michele Mercer, BarryDorgan, and others.Subject: 19. What is the best tester to developer ratio?Reported tester:developer ratios range from 10:1 to 1:10.Jeremy L. Mordkoff writes:> There's no simple answer. It depends on so many things, I can't even> list them all. Amount of reused code, number and type of interfaces,> platform, quality goals, etc.>> It also can depend on the development model. The more specs, the less> testers. The roles can play a big part also. Does QA own beta?> Do you include process auditors or planning activities?Boris Beizer adds:> These figures can all vary very widely depending on how you define> "tester" and "developer". In some organizations, a "tester" is anyone> who happens to be testing software at the time -- such as their own. In> other organizations, a "tester" is only a member of an independent test> group.>> It is far, far, better to ask about the test labor content than it is> to ask about the tester/developer ratio. The test labor content, across> most applications is generally accepted as 50%, when people do honest> accounting. For life-critical software, this can go up to 80%.Subject: 20. ConferencesFor further information about the conferences listed here and others, see the Conferences section of the SRM Hotlist at <http://www.cigital.com/hotlist/conferences.html>.Opinions in this section are by Boris Beizer.All of these conferences have published proceedings. Generally,materials presented at these conferences and in their proceedings leadthe publication in regular journals (above) by about 2-3 years.1. International Symposium on Software Testing and Analysis (ISSTA) theory, academic, leading edge practitioners (see 3b in the periodicals section.) Now transitioning to a summer conference that will alternate with ICSE. Sponsored by ACM and ACM's SIGSOFT.2. International Conference on Software Engineering (ICSE). Spring, world-wide. Technical. Primary source after ISSTA for leading edge results. 3. Quality Week (QW). Annual, San Francisco Bay Area. Biggest Conference on Testing and QA. Typically 700+. Many vendors. Good balance between technical/theoretical and practitioners. Very broad base. Workshops. Sponsored by Software Research Institute, qw@soft.com, 800-942-SOFT, 415-957-1441. Quality Week Europe (QWE) is held in Brussels, Belgium in November.4. Pacific Northwest Software Quality Conference (PNSQC). Annual, Portland Oregon, October. Definitely not a regional conference, despite the name. Very broad from technical/research to practitioners. A shade more academic and leading edge participation than quality week. 5. International Conference and Exposition on Testing Computer Software (TCS). Less technical and more practitioner/management than Quality Week. Smaller, 400+. Workshops. 17th annual conference was June 12-15, 2000, in Washington, D.C. Presented in cooperation with the ACM SIGSoft, ASQ Software Division, IEEE Reliability Society, and Software Technology Support Center. 6. Software Testing, Analysis, and Review (STAR). Software Quality Engineering. STAR East is in Orlando in May, and STAR West is on the east coast in the fall. About the same target as the International Conference (5) above. Comparable level and interest. Workshops. SQE 1-800-423-8378. 7. QAI International Software Testing Conference. More of a tutorial/workshop than a conference. Newbie orientation. Fall, Orlando, Florida. Quality Assurance Institute, 407-363-1111.8. IEEE Compsac, world-wide, various locations. Fall. Typical IEEE technical conference with a substantial number of papers of direct interest to testing and QA. Not as many as IEEE Software Engineering, though. There are about a dozen other annual conferences with a substantialtesting content: Korea, Japan, Germany, to name a few. In addition,there are about two-dozen "conferences", privately sponsored by variousindividuals and organizations. These are not real conferences in thatthere is no formal review process. Speakers are invited by theconference organizer. Often, a heavy tutorial content. These areusually small (under 100 participants, very few vendors). They range,depending on the organizer, from superb to fair-value, to outrightrip-off. Sorry. I won't play 20 questions on this one.(end of Beizer's comments)Also of note: International Conference on Software Quality (ICSQ), sponsored by the Software Division of the ASQ. International Symposium on Software Reliability Engineering (ISSRE), sponsored by IEEE. International Software Assurance Certification Conference (ISACC), "The Software Risk Management Conference." ISACC is an international conference in an annual series to be devoted exclusively to the topic of software certification. International Conferences on Practical Software Quality / Testing Techniques (PSQT and PSTT). Two conferences a year, one in Minnesota and one further south. Managed by Software Dimensions. Software Testing Analysis & Review Europe (EuroSTAR). Software Testing Analysis & Review Latin American (LatinSTAR) in March 20-23, 2001. The IFIP 14th International Conference on Testing of Communicating Systems Berlin, Germany, March, 19th - 22nd, 2002 http://www.fokus.gmd.de/events/testcom2002Subject: 21. PeriodicalsComments in this section are by Boris Beizer1. IEEE TSE (Transactions on Software Engineering). Monthly. The most prestigious journal for testing stuff. Volume 1 number 1 published the landmark article by Goodenough and Gerhart. Almost every issue since has had papers on testing and quality assurance. I have all the issues back to V1 #1. Much of that stuff is still relevant.2. ACM TOSEM (Transactions on Software Engineering Methodology). Quarterly. Relatively new journal (1992). Has prestigious editorial board. Somewhat more theoretical than IEEE TSE.3a. ACM SIGSOFT Software Engineering Notes. (from the Special Interest Group on Software Engineering). Monthly. Goes back to May 1976. This is the place where people publish stuff to stake out claims. It is not as strictly refereed as IEEE TSE or TOSEM, but some overview is provided to keep out the obvious crap. Important because there is a two or three year waiting list for publication in IEEE SE or ACM TOSEM. People who know this field read ACM SIGSOFT for advanced information on what will be published "officially" in IEEE SE and ACM TOSEM. You read an article and contact the author directly for more information.3b. ISSTA conference proceedings. Annual. (Known as TAV in the past -- and possibly new name changes in the offing.) ISSTA is the theoretical/academic conference devoted to testing. This is the most prestigious place to publish new results in testing theory and software reliability theory.4. IEEE Software. Six times a year. Rarely the latest stuff. More like surveys and overviews once a subfield has become established. Refereed, generally high standards. Mostly overviews, but occasionally new stuff. Additional comments from Scott Killops: You might mention in your blurb about 'IEEE Software' the regular "Quality Time" feature. This is wide ranging and generally quite useful. I've often thought that a collection of all of the "Quality Time" articles published to date would make a worthwhile book.5. Software Testing, Verification, and Reliability. Quarterly. International journal published by John Wiley and Sons. Prestigious international editorial board. High quality original articles and excellent surveys. Comparable in scope to IEEE TSE and ACM TOSEM, but sharply focused on testing, verification, and reliability. ISSN 0960-0833. <http://www.csc.liv.ac.uk/users/mrw/stvr.main.html> <http://www.wiley.com/> info@qm.jwiley.com, cs-journals@wiley.co.uk John Wiley & Sons, Inc., Professional, Reference and Trade Group 605 Third Avenue, New York, NY 10158-00126. ACM Computing Surveys. Quarterly. Not specific to testing and QA, but contains the most prestigious survey articles in the field, typically only when a subfield is well established. The authors are usually authorities. Articles are long and comprehensive. When an ACM Survey on a topic appears, it usually means that the field has matured to the point where it is possible to write meaningful books. Volume 29, #4, December 1997, contains the most comprehensive survey of testing research ever published--"Software Unit Test Coverage and Adequacy", Hong Zhu, Patrick A. V. Hall, and John H. R. May, pages 366-427. The bibliography contains 221 entries. Anyone interested in understanding the technical/research literature of software testing should start with this survey.7. Communications of the ACM. Monthly. Survey articles and overviews. Sometimes (rarely) original stuff. More academic and foundational/theory oriented than IEEE Software, but generally the same level. Doesn't publish too much on testing.8. Cutter IT Journal (formerly American Programmer). Monthly. Cutter Information Corp., 37 Broadway, Suite 1, Arlingon MA 02474, phone 781-641-5118. Private journal with prestigious editorial board. Non-technical, philosophical and overviews. Management orientation. Essentially a good newsletter. <http://www.cutter.com/itjournal>9. Software Testing & Quality Engineering. Bi-monthly. Software Quality Engineering, sqeinfo@sqe.com, 800-423-8378, fax 904-268-0733, phone 503-829-6806, <http://www.stqemagazine.com/>. Formerly called Software QA magazine and Software QA Quarterly.10. The Software Practitioner. Newsletter, 6 times a year. Computing Trends, 1416 Sare Rd., Bloomington IN 47401. Robert L. Glass - Editor-in-Chief, rlglass@acm.org. Non technical, management, overviews. Not too much on testing. http://SoftwarePractitioner.com/11. Crosstalk: The Journal of Defense Software Engineering. 6 times a year, free. Software Technology Support Center, Ogden AFB UT 84056. 801-777-8057. Heavy DoD orientation. Practitioner/Management/ Philosophical. Broad based on software engineering, including IV&V.http://www.stsc.hill.af.mil/CrossTalk/crostalk.html>custserv@software.hill.af.mil.12. Other journals. I'm sure I've offended several editors. There are journals concerned with testing, QA, and software engineering published in Japan, Korea, Germany, and Australia. Other journals, such as IEEE Transactions on Computers, IEEE Transactions on Reliability, ACM SIGPLAN, ACM SIGARCH, ACM SIGSIM, ACM SIGMETRICS, ACM OOPSLA, all publish material of interest to testing from time to time, especially insofar as hardware testing results and theory apply to software testing; also language issues vis-a-vis testing.(end of Beizer's comments)The American Society for Quality has begun publication of SoftwareQuality Professional. This journal will span thebody of knowledge for the Certified Software Quality Engineer.<http://www.asq.org/pub/>Commercial newsletters - the newsletters below are available for free topotential customers and some provide very good technical content. Quality Techniques Newsletter (QTN), formerly Testing Techniques Newsletter (TTN). Monthly, in electronic form only. Software Research, Inc., 415-957-1441, 800-942-SOFT. Send "subscribe" in the body of a message to qtn@soft.com. <http://www.soft.com/News/QTN-Online/> The Payne Report, http://www.cigital.com/paynereport/ Tejas Software Consulting Newsletter, emailed monthly. See <http://tejasconsulting.com/#news> for archives and to get a free subscription.Subject: 22. BooksBooks relevant to software testers are listed in the Books section ofthe SRM Hotlist at <http://www.cigital.com/hotlist/publications-books.html>.See also <http://www.StickyMinds.com/> under "Books" for Software Testing and Software Quality Engineering's book list, formerly known as the SingleSource Guide. See the misc.technical.books FAQ <http://www.faqs.org/faqs/books/technical/> for contact information for technical book publishers. Comp.software.testing readers have recommended Reiter's Scientific and Professional Books <http://www.reiters.com> and Fatbrain.com <http://www.fatbrain.com/>.Subject: 23. Bibliographic resourcesComments by Boris Beizer.1. ACM Computing Reviews. Monthly. The primary review journal. Most good papers on testing and quality assurance are eventually reviewed here. Extensive evaluations and criticisms by excellent reviewer pool. <http://www.acm.org/reviews/>http://www.acm.org/catalog/journals/107.html>Also, some books have large bibliographies that are very useful, such asthose in Software Testing Techniques and Testing Computer Software.Subject: 24. OrganizationsThe organizations below are devoted to software testing or quality, orhave special interest groups with such a focus. National andInternational organziations are listed first, followed by localorganizations.National/International----------------------American Society for Quality 611 E. Wisconsin Ave. P.O. Box 3005 Milwaukee, WI 53201-3005 800-248-1946, 414-272-8575 <http://www.asq.org> The ASQ (formerly ASQC) administers the CQE (Certified Quality Engineer) and CSQE (Certifified Software Quality Engineer) designations. The ASQ press has many titles that may be of interest to software testers. Stolen from a conference announcement: > Founded in 1946, the American Society for Quality (ASQ) > provides a variety of professional, educational, and informational > programs reflecting the changing needs of business and industry. > Headquartered in Milwaukee, WI, ASQ has been the leading quality > improvement organization in the United States for almost 50 years. > The Society is composed of more than 135,000 individual members and > 1,000 sustaining members worldwide. ASQ's vision is to be the > world's recognized champion and leading authority on all issues > related to quality. The mission of ASQ is to facilitate > continuous improvement and increased customer satisfaction by > identifying, communicating, and promoting the use of quality > principles, concepts, and technologies. ASQ's Software Division > ASQ's Software Division is comprised of [sic] more than 5,000 members > including software quality professionals and software engineers > interested in applying quality principles to the field of software > development. The Division develops a software engineer > certification program, publishes a quarterly newsletter, works with > the Software Quality System Registration Committee on establishing > an ISO 9000 Software Registration Program in the United States, > interacts with other professional software organizations such as > the IEEE and the ACM (Association for Computing Machinery), and is > involved in many other activities. <http://www.asq.org/about/divtech/softdiv/swqweb.html> October is National Quality Month (in the U.S.), as decreed by Ronald Reagan in 1984. Quality Assurance Institute 7575 Dr. Phillips Blvd., Suite 350 Orlando Florida 32819 Phone: 407-363-1111 Fax: 407-363-1112 <http://www.qaiusa.com/> The QAI publishes The Journal of the Quality Assurance Institute monthly. They provide CQA certification. Steve Driscoll wrote: > The CQA is someone who has demonstrated a mastery of the Common > Body of Knowledge(CBOK) for ensuring the quality of systems (which > often include software). While ISO-9000 is considered part of the > CBOK, it's actually independent of the CQA designation (there is a > certification program for ISO auditors). > ... while my life is still pretty much the same as it was prior to > obtaining the CQA designation, I found that the reading and > studying to prepare for the exam extremely beneficial in my duties > as a Quality Analyst. I recommend it to anyone who is interested > in producing quality software (which includes developers as well > as quality analysts).Society for Software Quality > Vision - To be recognized as the Society for those interested > in promoting "quality" as a universal goal for software. > > Mission - Promote "quality" as a desirable attribute of software. PO Box 86958 San Diego, CA 92138-6958 619-297-1544 <http://www.ssq.org/>Software Process Improvement Network > The Software Process Improvement Network is comprised of > individuals who want to improve software engineering practice. The > individuals are organized into regional groups called "SPINS" that > meet and share their experiences initiating and sustaining software > process improvement programs. They meet annually at the SEPG > (Software Engineering Process Group) Conference, which is > co-sponsored by the SEI and a regional SPIN. (from the web page) <http://www.sei.cmu.edu/collaborating/spins/spins.html>British Computer Society Software Testing BCS Specialist Group > The Group has a number of full-day meetings per annum and is host > to the international conference on software testing - EuroSTAR. It > is producing a standard on component testing and has a sub-group on > Statistical Testing Methods. A full library of testing-related > material is available at meetings. > > It promotes awareness of testing, competence in the field, and > encourages research. (from the web page) <http://www.bcs.org.uk/siggroup/sg55.htm>Chinese Association for Software Quality (CASQ) This association is formed to share information and experiences amongst its members and people on software quality engineering. <http://www.casq.org> casq@newcastle.ac.ukSwedish Association for Software Testing (SAST) SAST is a non-profit organisation with members from all different companies that are interested in Software Testing within Sweden. The purpose of SAST to improve status of Software Testers and Software Testing, through informal education and exchange of information. Since this is a Swedish organization most activities are in Swedish. If you are interested you are welcome to contact Lise-Lotte Karlsson Boman (lise-lotte.karlsson-boman@posten.se) or Jan Warg (jan@warg.se). <http://public-www.pi.se/~sast>TestNet TestNet is a Special Interest Group in Software Testing with over 400 members of over 150 companies and organisations mainly in the Netherlands (although there are a few members from Belgium). The meetings and news-items are in Dutch. > The purpose of TestNet is the professionalization of testing IT products, > and an increase in the awareness and the importance of testing as a > profession in its own right. TestNet stimulates the exchange of > professional knowledge and practical experience amongst testers, and > stimulates research, from a scientific standpoint as well as from a > practical perspective. (from the web page) If interested or for information contact at secretaris@testnet.org <http://www.testnet.org> testnet@testnet.orgCRIM Software Test Centre CRIM is a transfer and liaison centre with fifteen years' experience in leading-edge information technologies and computer application sectors. CRIM Software Test Centre (STC) helps big or small businesses and other kinds of organizations boost product quality, cut risk and optimize investments. Technologically, CRIM STC pinpoints the strengths and weaknesses of software applications based on pre-established criteria. CRIM STC may also recommend various ways of improving and optimizing the product testing and development processes. <http://www.crim.ca/en/index.epl?selec=a000&href=/en/ctl/index.htm> Local organizations-------------------Most of the organizations above sponsor local organizations. Contact themto find local organizations near you. In this section you'll find localorganizations that don't appear to have a national affiliation.The Software Quality Institute At UT (stolen from a conference announcement) > The Software Quality Institute (SQI) at The University of Texas at > Austin is a multidisciplinary partnership between UT-Austin and the > software and information systems industries in Texas. It is > recognized as a leading authority on and champion for software > quality. SQI was founded in 1993 for the purpose of strengthening > organizations to compete more successfully in global markets > through sponsorship of seminars and conferences. An advisory group > of 24 industry and academic representatives guides the Institute. > Its popular programs include: a 13 week long Software Project > Management Certificate Program, one- and two-day seminars offered > to the public and as in-company contract courses, our "Software > Quality Matters" quarterly newsletter that addresses issues of > concern relative to quality issues, sponsorship of monthly forums > for discussion of software issues (including the Software Process > Improvement Network (A-SPIN), the Austin Software Executives' Group > (ASEG), and the Austin Forum for Object-Oriented Technology > (AFOOT), and on-line services which include a Worldwide Web > Home Page and topical newsgroups. The web page is <http://www.utexas.edu/coe/sqi/>. The newsgroups seem to be reachable only at UT. For information on newsletter subscriptions, contact info@sqi.utexas.edu.Central Ohio Quality Assurance Association Mission Statement: To provide and promote continuous quality improvement information to software organizations in our communities. COQAA is organized to share state-of-the-art quality assurance methods, tools, and techniques among its members. P.O. Box 14191, Columbus, Ohio 43214-0191 contact: Jerry Fehribach, 614-447-3600, jfehribach@cas.org <http://www.coqaa.org>Australian Software Quality Research InstituteGriffith University, Brisbane, Australia<http://www-sqi.cit.gu.edu.au/>Triangle Information Systems Quality Association (North Carolina)http://www.rtpnet.org/tisqa/contacts - Dennis Brandel 72652.1146@compuserve.com David Wood DAVDWOOD@VNET.IBM.COMNorth Carolina Quality Leadership Foundation 800-207-5685 http://www.rtpnet.org/ncqlf/New England Software Quality Assurance Forum (NESQAF) NESQAF is a regional non-profit society created by software quality professionals for software quality professionals. Each month, professionals from software companies and MIS organizations along with other industry experts exchange ideas, principles, and experiences on the latest quality assurance techniques and technologies. An email distribution list is available. Monthly meetings in Cambridge, MA. Contact - Alan Titelbaum, alan.titelbaum@tfn.com, 617-856-2467. <http://www.nesqaf.org/>Software Testing Institute Offers industry research, surveys, publications, online services and seminars to software development and testing professionals. A list of software testing product vendors, including web site and e-mail links, is available from our web site: <http://www.ondaweb.com/sti> contact - Susan Archer, Director (sarcher@metronet.com) 726 Loganwood Ave., Richardson, TX 75080 972-680-8507, fax 972-680-8905Centre for Software Reliability The Centre for Software Reliability (CSR) is a research centre within the Department of Computing Science at the University of Newcastle upon Tyne; it conducts research on how to achieve improved levels of dependability from computing systems. csr@newcastle.ac.uk <http://www.csr.ncl.ac.uk/>Seattle Area Software Quality Assurance Group (SASQAG) Purpose: To Promote Professional Software Quality Practices <http://www.sasqag.org/>Subject: 25. ContributorsThanks go to the people who have contributed to the FAQ in variousways, including the team of assistant FAQ maintainers - Chris Petrov,and Richard de Graaf, and Danny Faught. Email addresses for some ofthose mentioned herein are listed below.Boris Beizer - bbeizer@acm.orgBob Binder - rbinder@rbsc.comRichard de Graaf - r.dgraaf1@chello.nlBarry Dorgan - bazman@bigfoot.comSteve Driscoll - sdriscol@oclc.orgSigrid Eldh - Sigrid.Eldh@ehpt.comDanny Faught - faught@tejasconsulting.comRick Hower - rickhower@earthlink.netScott Killops - Scott_B_Killops@ccm.jf.intel.comJennifer Larsen - Jennifer_Larsen@datatel.comBrian Marick - marick@testing.comMark McWhinney - msm@powertest.comDon Mills - donmil@voyager.co.nzJeremy L. Mordkoff - mordkoff@acm.orgCharles Nichols - nichols@cse.ogi.eduChris Petrov - petrovchris@usa.net Bret Pettichord - bret@pettichord.comRaymond Rivest - rrivest@crim.caLaura Rose - laura@PACorp.comJohn Tyson - jmt@acm.orgMultiPage
Rate this FAQ
N/A
Worst
Weak
OK
Good
Great
Related questions and answers [ Usenet FAQs | Search | Web FAQs | Documents | RFC Index ] Send corrections/additions to the FAQ Maintainer: raymond.rivest@crim.ca (Raymond Rivest)Last Update September 08 2008 @ 00:13 AM© 2008 FAQS.ORG. All rights reserved. |
|
