I think testing and debugging is one of the harder aspects of scientific and numerical software. It’s easy to get buried in giant pile of code and have no idea where to look for bugs. It’s even harder when actually the bug isn’t in the code but is in the concepts, math or data.

The fundamental problem with testing and debugging scientific software is that we don’t know correct intermediate values or even the correct final output. I helped write a previous post on similar topics but I wanted to write more on the topic. So, here are various suggestions that probably would’ve helped me at some point. I’d like to expand quite a bit more on individual points here in the future.

1. Don’t trust your code. Seriously, never trust your code.
2. Treasure correct code and use version control. Don’t lose your treasure.
3. Use existing correct code to maximum value because it means you can check intermediate values that would otherwise be untestable.
4. Hunt the internet for existing correct code. But don’t trust it.
5. Add one feature at a time.¹
6. Define “one feature” as narrowly as possible.²
7. Edges cases shouldn’t be an after thought. Sometimes solving an edge case can be informative about the common case.
8. It’s okay to spend a couple hours just thinking about how to design a test problem that tests only one new feature.
9. Any time you can, compare with an analytical solution.
10. The method of manufactured solutions (MMS) is amazing.
11. Use MMS even if it requires implementing some extra features. It will be worth it.
12. Check the order of accuracy/convergence rate!
13. Even on problems where the true solution is unknown, it’s possible to check for convergence rate by comparing with a very high accuracy solution.
14. Be a more careful programmer. Sometimes, It’s okay to spend thirty minutes just thinking through each line of code.
15. Learn your debugging tools and IDE! Line-by-line step through debugging is incredibly helpful.
16. Don’t stay stuck for more than a couple hours. Try something different.
17. When you notice that something looks wrong, try to encode the meaning of “wrong”.
18. Be a faster programmer. Sometimes, just trying a bunch of things is the right approach.
19. Use smaller and simpler test problems so that you can iterate really fast.
20. Don’t trust evaluation code. Look at the output. Bugs in evaluation code can make you waste a huge amount of time.
21. Make lots of figures and videos. Visualizing a problem is often very effective for debugging.
22. Test derivatives with finite differences.
23. Test symmetries and invariances. Many problems are rotationally symmetric. Or invariant with respect to rigid body motion.
24. Test more algorithmic properties. Optimization algorithms often guarantee a decrease in objective at each step!
25. Log lots of info. It’s okay to save a full matrix at each time step.
26. Prototype! The first version of the code should not be well designed or fast.
27. Write at least the first version in a language like Python or Julia where edit-(compile)-run cycles are on the order of a second.
28. Doing the first phases of development in a Jupyter notebook (or similar) is really fast. Iteration time is extremely important.
29. Write a second version from scratch, maybe in a different language or style, and compare the results. You might catch some bugs.
30. Fast tests are more useful than slow tests.
31. Characterization tests (aka “golden master tests”, aka “freeze tests”) are useful for preventing unexpected changes. But they can’t define correctness and they are brittle.
32. Robust tests are more useful than brittle tests. False alarms are a bummer.
33. Automated tests are more useful than manual tests.
34. Continuous integration is normally worth the effort.
35. Remove randomness in testing by setting a random seed. Even if true randomness is necessary for correctness. Consistency in testing is very important.
36. Use symbolic algebra tools to develop test cases. I frequently use Sage and sympy.
37. Use multiprecision and arbitrary precision arithmetic to develop test cases. I’ve used mpmath and MPFR.
38. Sign errors can often be solved with guess and check. Verify with the math later! I’ve found a lot of mild math errors or misunderstandings this way.
39. Guess and check also works in some other areas. But, don’t flail around in the dark for very long.
40. Ablation testing: remove some component of your system and verify that the performance degrades as expected.
41. Look for gaps or overlaps and other problems in your meshes.
42. Check your normal vectors. Should they point inwards or outwards?
43. Corners are scary. Start with something smooth like a circle or sphere.
44. The math and theory is often more important than you think.
45. Violating function space and regularity requirements can bite you. See “corners”.
46. Start with direct linear solvers. Iterative solvers introduce a whole new class of problems with preconditioning.
47. Check the condition number of your matrices.
48. Use standard tools and libraries where they are sufficient.
49. Test single threaded first. Then on two cores. Then many. And write the CPU version before the GPU version!
50. Finally, remember not to trust your code.

Footnotes and links

• I’ve also worked in machine learning, statistical software and deep learning a lot. I think a similar list would apply in those areas, but I’d have to add and remove a few points.
• Best Practices for Scientific Computing
• Good enough practices in scientific computing
• Sharing your code feels related
• Is it worthwhile to write unit tests for scientific research codes?
• How to test scientific software?
• How to write integration tests for numeric simulation software?
• Strategies for unit testing and test-driven development
• Why We Don’t Teach Testing (Even Though We’d Like To)
• Spinning Up as a Deep RL Reseacher (A great high-level summary of how to do effective research in Reinforcement Learning)
• Design for testability