View source for Software Testing/Meeting Notes 2019-04-08

{{:Software Testing}}


__TOC__


==== Resources ====

* [https://www.techsoupcanada.ca/en/directory/334 TechSoup Canada catalogue: Developer Software]
* [https://obsproject.com/ OBS Project] - software that is good for capturing videos of tests including drop menus


One of James Bach's [https://www.youtube.com/watch?v=ILkT_HV9DVU talks on YouTube]
{{#evu:https://www.youtube.com/watch?v=ILkT_HV9DVU  | dimensions=640 | alignment=center}}


==== Meeting Notes ====
* Introductions


* Nicholas Collins
** Software tester for a few years, knowledge of how ''his'' company works
** But two years isn't a long time compared to some software testers
** Nick has prepared notes, will be presenting slightly differently from other KWNPSA sessions
** SysAdmin in insurance industry; laid off (as are many of us); back to school to upgrade IT skills
** Uses Visual Studio, C#, other languages
** People he's met were developers, or business-specific skills; when software testers are needed these people are thrust into the role
** This might change as more universities offer software testing as a major
** There are very few courses or certificates in software testing, more prevalent in the US
*** but Fanshawe college in London has a certificate program
** Some institutions have a couple of courses in tech writing, project management, quality management; maybe a night course in software quality testing
** Without academic rigour, different people use different terminology, nomenclature
*** "Should I know what all these different terms mean?" But it's fairly common with other software testers Nick has spoken to
** At Microsoft, developers use their development skills to write tests.  Needs more skills than just coding
*** Microsoft has internal courses to train testers how to test software
*** Get promoted to full developer once you've proven you can write tests
** people use Terminology like "Post-mortem" (although nobody dies), mix up "Milestone" and "Benchmark", &c.
** Software testing is the start for a developer's career, then to DevOps
*** Does this mean the most junior, inexperienced programmers are responsible for testing software?
** Nick: large companies use junior testers to run tests, senior testers to supervise
** During an upgrade Nick (a programmer at the time) did testing for the Database Analyst
*** But a junior intern was assigned to that role as well, just to gain experience.
*** Worked out details at a high level, then applied tests to get results


* Project Managers take different approaches


* '''You can always think of more tests'''
** It's a fine balance between staying on schedule and being thorough
** Walkthroughs and working in a team can be helpful


* Some testing instructors do not like teaching from texts
** eg. "Software Testing" by Yogesh Singh
** But Nicholas gets good ideas from texts, doesn't agree with those testing instructors
** THe problem is that the authors suffer from "Perfect Worldism"
*** A world where there is unlimited time and money, and the perfect tests can be developed
** Nicholas has experience with sticky problems, gets ideas from texts to adapt to his problem


* Party talk: Software tester does not lead to stimulating conversation
** YouTube presenter on software testing is not dull!  (**********Need link!***********)


* Even the simplest test "is A < 70 ?" can have seven or eight tests
** Test results, but also overflows, boundaries, different data types
** Input validation can require many tests


* Working with other people, eg. technical writers
** For them to understand the software they'll play with the software, and may create unanticipated conditions
** Everyone can be a software tester to some degree: Project manager, developer, writer. Even sales?
** Sometimes testers find problems with usability as they're running tests; not part of the test suite


* How effective are some of these ad-hoc testers?
** Is there a bias? Do they have some incentive to pass tests even when there are problems?
*** Sometimes a QA will hold back tests that would have been better to give to the developer in the first place
** Accessibility testing is a new skill for QA, may become a testing requirement
** Business Analyst (BA), developer and tester make a good team
*** Sometimes the process of testing will identify the need for more testing


* Reporting bugs
** Requires consideration, tact


* Test plans may need to be developed quickly
** But near the end of a project when time is tight there may not be time to develop tests
** So quality of code may suffer near the end of the project
*** Breaking things during testing that no-one has time to fix


* Automated testing?
** Nick has experience with automated regression testing
** Automated regression testing reduces the introduction of new bugs


* Open Broadcaster Software
** Used to catch all activity during user testing
** Also use Virtual Box recorder uses host to capture all the output on the VM screen


* "Monkey Testing"
** Also "fuzz testing" or "fuzzing"
** Fill all fields, try to overflow, pound on the keyboard, click as fast as possible
*** But this this does not lead to reproducible errors (fine timing errors)
*** Although some testers claim they can reproduce


* Pride in finding bugs?
** Nick finds that the "high five" time should occur only after the entire team has identified, reported, documented, and fixed the error, and re-tested


* Load testing
** Hitting a system with a large number of transactions, &c.
** But a bogged down system may not be writing to logs, making analysis difficult
** A benefit in load testing is adding assertions, find issues with threads
*** Assertions and Singletons...
** Be sure to validate the output even when just testing for capacity


* Nick has written a test for XML testing
** But the code Nick wrote was not well tested at all! Oh, the irony!


* Q: Do you use debugger software like GDB to examine the flow of code?
* A: Not common, but becoming more prevalent.
** Certainly having a debugger to throw at the code is nice to have
** But much testing is done with the software under test as a black box, just examine the input and the expected output


* Nick speaks of the complexity of software testing.
** One thing works fine by itself, and other thing does too, but do they work together?
** Different software on different platform needs to interoperate, but sometimes differences in date formats causes problems
*** although each platform by itself passed all tests
** Dealing with currencies, eg. USD and CAD, and GBP
** Dealing with leap years and 29 February
** General rule: Anything date sensitive needs to test for leap years
*** and time zones! Anything dealing with calendars needs to worry about time zones


* What happens internationally when different countries need to interoperate?
** Companies have service contracts that define how the service is implemented
*** If the system is changed, the contract defines who is responsible for continued interoperation
*** If I make a change and it breaks your system, it's your fault for not defining the contract accurately
*** called "spring contracts"


* Nick gives an example from James ---- YouTube video ("nominal input voltage is 100VAC to 250VAC")
** "Test the nominal range" is an incomplete answer
** Also need to test outside the range
** The user manual may give advice not to go outside the nominal range, but users don't necessarily read the manual
** So, does the system fail gracefully outside the nominal range?
** This is the function of the software tester, to design the test to ensure that software or equipment is failsafe
*** eg. for medical equipment
*** How much money is available to fry the device under test?  Some prototypes may be '''really''' expensive
*** Many examples of people damaging electronics with incorrect application of voltage!
** It's good for testers to think outside the parameters of the system


* Testing to ensure system has a consistent look and feel
** eg. fonts on some menus were different
*** Is that a software testers responsibility? Sometimes as an additional task
*** There are tools (overlays, templates) to find these issues
** Window resizing can make the application fail, but there need to be limits for those tests
** Testing for "greyed out" functions can be time consuming
*** When a function is available when it shouldn't be can result in errors
** These are general things for a tester to keep in mind


* Systems that have features which have little to do with each other
** Easy to test they're not contending for resources, &c.
** But still important to run these features simultaneous to shake loose bugs, eg. memory allocation, concurrent DB access
** Perhaps a simple monitor with limited functions: But what if something goes wrong, does the device report an error?


* Client-side data validation: All testing needs to be duplicated at the server to ensure malusers don't bypass client-side validation
** But that increases load on the server


* Logging
** Logs may indicate problems with the way the code executes, eg. repeated log entries indicate an invalid loop
** Circular reasoning: How can the logs from software under test be considered 
*** Logs are only one step, begin the process of analysis
*** NewRelic will test user experience (surveillance software)


* Nick has found bugs because the test suites are well designed
** But at least half the time the bugs discovered were found in spite of the test, which was not designed to find that kind of bug


* Q&A
** Is the developer + tester model usable?
*** May be a bit scary for shops not set up for that collaborative arrangement
** Nick says to just forge ahead.
*** Having experience is good, but can also develop that experience in-house
** Worries about the coming requirements for accessibility for software
*** May take changes in coding practices (use POSH: Plain Ol' Semantic HTML instead of Javascripted forms)
*** Jurisdictional differences may be difficult to deal with


Back to: [[Software Testing]]

[[Category:KWNPSA Meeting Notes]]