50 suggestions on correctness and testing for scientific software for PDEs

I think testing and debugging is one of the harder aspects of scientific and numerical software. It’s easy to get buried in giant pile of code and have no idea where to look for bugs. It’s even harder when actually the bug isn’t in the code but is in the concepts, math or data.

The fundamental problem with testing and debugging scientific software is that we don’t know correct intermediate values or even the correct final output. I helped write a previous post on similar topics but I wanted to write more on the topic. So, here are various suggestions that probably would’ve helped me at some point. I’d like to expand quite a bit more on individual points here in the future.

  1. Don’t trust your code. Seriously, never trust your code.
  2. Treasure correct code and use version control. Don’t lose your treasure.
  3. Use existing correct code to maximum value because it means you can check intermediate values that would otherwise be untestable.
  4. Hunt the internet for existing correct code. But don’t trust it.
  5. Add one feature at a time.1
  6. Define “one feature” as narrowly as possible.2
  7. Edges cases shouldn’t be an after thought. Sometimes solving an edge case can be informative about the common case.
  8. It’s okay to spend a couple hours just thinking about how to design a test problem that tests only one new feature.
  9. Any time you can, compare with an analytical solution.
  10. The method of manufactured solutions (MMS) is amazing.
  11. Use MMS even if it requires implementing some extra features. It will be worth it.
  12. Check the order of accuracy/convergence rate!
  13. Even on problems where the true solution is unknown, it’s possible to check for convergence rate by comparing with a very high accuracy solution.
  14. Be a more careful programmer. Sometimes, It’s okay to spend thirty minutes just thinking through each line of code.
  15. Learn your debugging tools and IDE! Line-by-line step through debugging is incredibly helpful.
  16. Don’t stay stuck for more than a couple hours. Try something different.
  17. When you notice that something looks wrong, try to encode the meaning of “wrong”.
  18. Be a faster programmer. Sometimes, just trying a bunch of things is the right approach.
  19. Use smaller and simpler test problems so that you can iterate really fast.
  20. Don’t trust evaluation code. Look at the output. Bugs in evaluation code can make you waste a huge amount of time.
  21. Make lots of figures and videos. Visualizing a problem is often very effective for debugging.
  22. Test derivatives with finite differences.
  23. Test symmetries and invariances. Many problems are rotationally symmetric. Or invariant with respect to rigid body motion.
  24. Test more algorithmic properties. Optimization algorithms often guarantee a decrease in objective at each step!
  25. Log lots of info. It’s okay to save a full matrix at each time step.
  26. Prototype! The first version of the code should not be well designed or fast.
  27. Write at least the first version in a language like Python or Julia where edit-(compile)-run cycles are on the order of a second.
  28. Doing the first phases of development in a Jupyter notebook (or similar) is really fast. Iteration time is extremely important.
  29. Write a second version from scratch, maybe in a different language or style, and compare the results. You might catch some bugs.
  30. Fast tests are more useful than slow tests.
  31. Characterization tests (aka “golden master tests”, aka “freeze tests”) are useful for preventing unexpected changes. But they can’t define correctness and they are brittle.
  32. Robust tests are more useful than brittle tests. False alarms are a bummer.
  33. Automated tests are more useful than manual tests.
  34. Continuous integration is normally worth the effort.
  35. Remove randomness in testing by setting a random seed. Even if true randomness is necessary for correctness. Consistency in testing is very important.
  36. Use symbolic algebra tools to develop test cases. I frequently use Sage and sympy.
  37. Use multiprecision and arbitrary precision arithmetic to develop test cases. I’ve used mpmath and MPFR.
  38. Sign errors can often be solved with guess and check. Verify with the math later! I’ve found a lot of mild math errors or misunderstandings this way.
  39. Guess and check also works in some other areas. But, don’t flail around in the dark for very long.
  40. Ablation testing: remove some component of your system and verify that the performance degrades as expected.
  41. Look for gaps or overlaps and other problems in your meshes.
  42. Check your normal vectors. Should they point inwards or outwards?
  43. Corners are scary. Start with something smooth like a circle or sphere.
  44. The math and theory is often more important than you think.
  45. Violating function space and regularity requirements can bite you. See “corners”.
  46. Start with direct linear solvers. Iterative solvers introduce a whole new class of problems with preconditioning.
  47. Check the condition number of your matrices.
  48. Use standard tools and libraries where they are sufficient.
  49. Test single threaded first. Then on two cores. Then many. And write the CPU version before the GPU version!
  50. Finally, remember not to trust your code.

  1. We should try to find the smallest testable unit of code. For example, when writing a fast multipole method implementation, you could start by writing code for a 3x3 subdivision of the domain with only source approximation. This would test the source-side approximation code in isolation. Then, you write the target-side approximation code separately and test that. Then, you move to a tree structure and test that component separately. ↩︎

  2. As an aside, I think most test driven development (TDD) advice is insane, particularly when applied to code for numerical methods. TDD focuses on tiny, (almost?) meaningless tests. In most numerical methods work, a unit of testable work is often an entire solution to a particular physical problem, perhaps including convergence tests. It’s perfectly fine and maybe even good to write that “test” before writing the implementation of the numerical method. I do it often. But, that sometimes still means that I have several hundred lines of code that I need to write before I can see if the test passes or even runs at all. The problem isn’t really in the spirit of TDD, but in the focus in books and examples on tiny increments of work and on requiring the tests to be written first. ↩︎

Posts BIEBook