Testing Software at DrChrono

Testing Software @ DrChrono


🤖 How familiar are you with testing software?

I was born in it, molded by it

I've written one before

I know some of these words

What?

Go to app.deckdeckgo.com/poll and use the code {0}
Awaiting votes

🧪 Test (noun)

  • A procedure intended to establish the quality, performance, or reliability of something, especially before it is taken into widespread use.

😪 The Problem

  • Add new features without breaking our existing features.
  • Fix existing bugs without introducing more problems.
  • A reliable product will make less customers churn.

💖 Advantages

  • Have confidence when you make changes
  • Introduce less defects
  • Catch defects sooner, when the cost of fixing them is the lowest                                                                                      

  

🛑 Limitations

  • Opportunity cost. Writing tests is a investment of time and money that could have been used to create new features or fix bugs.         
  • Other tools, like mypy and code review, can catch many of the same problems at a lower cost.
  • Tests can make changes harder.

🏭 Testing is quality control for our software factory

PyCharm's Test Runner tab

  CircleCI runs our tests every time we create a Pull Request, and our test are required to pass to merge our change

📉 Regression Testing

Using the iOS app, a provider tried to filter out appointments for their Sacramento office, but the visits from other offices were showing too.

1️⃣ The actual fix was only a single line!

📼 Betamax: Record your test suite's HTTP interactions and replay them

🖼 Front-end tests

We don't have tests for our JavaScript, Angular, or JQuery code.

Elm has some tests, but we're stuck on an older version and have decided to replace it.

TypeScript, React, GraphQL have good test coverage.

🛤 Test Rail

🛤 Test Rail

🤖 Selenium

📐 Measuring the quality of quality control

“If you cannot measure it, you cannot improve it.” - Lord Kelvin

Test coverage is a measure (in percent) of the degree to which the source code of the program is executed by a test suite.

The first published reference was by Miller and Maloney in the February 1963 issue of Communications of the ACM in an article called "Systematic mistake analysis of digital computer programs"

💵 How much testing is economical?

  • The problem is not that we don’t know how to write correct software. The problem is that correct software is too expensive.                        
  • The amount of testing required is different for every business, it depends on the impact of failure
  • https://hypothesis.works/articles/economics-of-software-correctness/

💯 Should we try to achieve 100% test coverage?

🚀 The SpaceX Dragon's crew displays use JavaScript,   has a "high bar" for test coverage

🤨 How do I achieve 100% test coverage?

An easy way is to write tests before you write the code.

📚 Recommended Reading! This is free online at www.obeythetestinggoat.com

🧬 Mutation Testing

Another measure of test quality. Add "mutations" to you code and run your test suite. A test suite that does not detect and reject the mutated code is considered defective.

Like coverage analysis, mutation testing helps ensure that you’re testing all of the code.

Coverage only tells you if a line of code is executed, mutation testing will determine if your tests actually check the behavior of your code.

🔮 Opportunities for the future @ DrChrono

  1. ​Write Selenium code to automate our Test Rail scenarios.
  2. Write tests for untested Javascript, or prioritize to rewrite in Typescript/React.
  3. Test/lint our infrastructure code (i.e. Terraform, Salt)
  4. Run our test suite with mutations and see how many tests still pass

👋 That's all folks!

Citations/References

GDC. (2021). Automated Testing of Gameplay Features in 'Sea of Thieves'. YouTube. Retrieved December 29, 2021, from https://www.youtube.com/watch?v=X673tOi8pU8.

Percival, H. (2017). Test-driven development with Python: Obey the testing goat. O'Reilly.

Prause C.R., Werner J., Hornig K., Bosecker S., Kuhrmann M. (2017) Is 100% Test Coverage a Reasonable Requirement? Lessons Learned from a Space Software Project. In: Felderer M., Méndez Fernández D., Turhan B., Kalinowski M., Sarro F., Winkler D. (eds) Product-Focused Software Process Improvement. PROFES 2017. Lecture Notes in Computer Science, vol 10611. Springer, Cham. https://doi.org/10.1007/978-3-319-69926-4_25

MacIver, D. R. (2016, April 15). The Economics of Software Correctness. Hypothesis. Retrieved December 29, 2021, from https://hypothesis.works/articles/economics-of-software-correctness/

R/spacex - we are the spacex software team, ask us anything! reddit. (n.d.). Retrieved December 29, 2021, from https://www.reddit.com/r/spacex/comments/gxb7j1/

Quality control employee at a gun factory with thought balloon that reads,? - New Yorker Cartoon. Tom Cheney Published May 16, 2005