Bad Tools Hurt: Lessons for teaching computer security skills to undergraduates

  • Jonathan Sharman Rice University
  • Claudia Acemyan
  • Philip Kortum rice university
  • Dan Wallach Rice University
Keywords: security, coding, teaching, usability, tools


Understanding why developers continue to misuse security tools is critical to designing safer software, yet the underlying reasons developers fail to write secure code are not well understood. In order to better understand how to teach these skills, we conducted two comparatively large-scale usability studies with undergraduate CS students to assess factors that affect success rates in securing web applications against cross-site request forgery (CSRF) attacks. First, we examined the impact of providing students with example code and/or a testing tool. Next, we examined the impact of working in pairs. We found that access to relevant secure code samples gave significant benefit to security outcomes. However, access to the tool alone had no significant effect on security outcomes, and surprisingly, the same held true for the tool and example code combined. These results confirm the importance of quality example code and demonstrate the potential danger of using security tools in the classroom that have not been validated for usability. No individual differences predicted one’s ability to complete the task. We also found that working in pairs had a significant positive effect on security outcomes. These results provide useful directions for teaching computer security programming skills to undergraduate students.


Download data is not yet available.


Acar, Y., Backes, M., Fahl, S., Garfinkel, S., Kim, D., Mazurek, M. L., and Stransky, C. (2017). Comparing the Usability of Cryptographic APIs. In 2017 IEEE Symposium on Security and Privacy (SP), pages 154-171.

Acar, Y., Backes, M., Fahl, S., Kim, D., Mazurek, M. L., and Stransky, C. (2016). You Get Where You're Looking for: The Impact of Information Sources on Code Security. In 2016 IEEE Symposium on Security and Privacy (SP), pages 289-305.

Acar, Y., Stransky, C., Wermke, D., Mazurek, M. L., and Fahl, S. (2017). Security Developer Studies with GitHub Users: Exploring a Convenience Sample. In Thirteenth Symposium on Usable Privacy and Security (SOUPS 2017), pages 81-95, Santa Clara, CA. USENIX Association.

Arisholm, E., Gallis, H., Dyba, T., and Sjoberg, D. I. K. (2007). Evaluating pair programming with respect to system complexity and programmer expertise. IEEE Transactions on Software Engineering, 33(2):65-86.

Bangor, A., Kortum, P., and Miller, J. (2009). Determining what individual sus scores mean: Adding an adjective rating scale. Journal of Usability Studies, 4(3):114-123.

Bangor, A., Kortum, P. T., and Miller, J. T. (2008). An empirical evaluation of the system usability scale. International Journal of Human-Computer Interaction, 24(6):574-594.

Begel, A. and Nagappan, N. (2008). Pair programming: What's in it for me? In Proceedings of the Second ACMIEEE International Symposium on Empirical Software Engineering and Measurement, ESEM '08, page 120-128, New York, NY, USA. Association for Computing Machinery.

Bernhard, M., Sharman, J., Acemyan, C. Z., Kortum, P., Wallach, D. S., and Halderman, J. A. (2019). On the Usability of HTTPS Deployment. In Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems, CHI '19, pages 310:1-310:10, New York, NY, USA. ACM.

Brooke, J. (1996). SUS: A quick and dirty usability scale. Usability evaluation in industry, page 189.

Eierman, M. and Iversen, J. (2018). Comparing testdriven development and pair programming to improve the learning of programming languages. Journal of the Midwest Association for Information Systems, 2018:23-36.

Fischer, F., Böttinger, K., Xiao, H., Stransky, C., Acar, Y., Backes, M., and Fahl, S. (2017). Stack Overflow Considered Harmful? The Impact of Copy Paste on Android Application Security. In 2017 IEEE Symposium on Security and Privacy, pages 121-136.

Gorski, P. L., Iacono, L. L., Wermke, D., Stransky, C., Moeller, S., Acar, Y., and Fahl, S. (2018). Developers Deserve Security Warnings, Too: On the Effect of Integrated Security Advice on Cryptographic API Misuse. In SOUPS @ USENIX Security Symposium.

Hanks, B., Mcdowell, C., Draper, D., and Krnjajic, M. (2004). Program quality with pair programming in cs1. In ACM Sigcse Bulletin,(36), pages 176-180.

Hannay, J., Dybå, T., Arisholm, E., and Sjøberg, D. (2009). The effectiveness of pair programming: A meta-analysis. Information and Software Technology, 51:1110-1122.

ISO 9241-11 (2018). Ergonomics of human-system interaction - Part 11: Usability: Definitions and concepts.

Kortum, P. and Acemyan, C. Z. (2013). How low can you go?: Is the system usability scale range restricted? Journal of Usability Studies, 9(1):14-24.

Krombholz, K., Mayer, W., Schmiedecker, M., and Weippl, E. (2017). 'I Have No Idea What I'm Doing' - On the Usability of Deploying HTTPS. In 26th USENIX Security Symposium (USENIX Security 17), pages 1339-1356, Vancouver, BC. USENIX Association.

Mcdowell, C., Werner, L., Bullock, H., and Fernald, J. (2002). The effects of pair-programming on performance in an introductory programming course. In ACM SIGCSE Bulletin,(34), pages 38-42.

Mcdowell, C., Werner, L., Bullock, H. E., and Fernald, J. (2003). The impact of pair programming on student performance, perception and persistence. In 25th International Conference on Software Engineering, 2003, pages 602-607.

McDowell, C., Werner, L., Bullock, H. E., and Fernald, J. (2006). Pair programming improves student retention, confidence, and program quality. Communications of the ACM, 49(8):90-95.

Mindermann, K. and Wagner, S. (2018). Usability and Security Effects of Code Examples on Crypto APIs - CryptoExamples: A platform for free, minimal, complete and secure crypto examples. 2018 16th Annual Conference on Privacy, Security and Trust (PST).

Naiakshina, A., Danilova, A., Gerlitz, E., and Smith, M. (2020). On conducting security developer studies with cs students: Examining a password-storage study with cs students, freelancers, and company developers. In Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems, pages 1-13.

Naiakshina, A., Danilova, A., Gerlitz, E., von Zezschwitz, E., and Smith, M. (2019). 'If You Want, I Can Store the Encrypted Password': A Password-Storage Field Study with Freelance Developers. In Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems, CHI '19, pages 140:1-140:12, New York, NY, USA. ACM.

Naiakshina, A., Danilova, A., Tiefenau, C., and Smith, M. (2018). Deception task design in developer password studies: Exploring a student sample. In Fourteenth Symposium on Usable Privacy and Security (SOUPS 2018), pages 297-313, Baltimore, MD. USENIX Association.

Oliveira, D. S., Lin, T., Rahman, M. S., Akefirad, R., Ellis, D., Perez, E., Bobhate, R., DeLong, L. A., Cappos, J., and Brun, Y. (2018). API Blindspots: Why Experienced Developers Write Vulnerable Code. In Fourteenth Symposium on Usable Privacy and Security (SOUPS 2018), pages 315- 328, Baltimore, MD. USENIX Association.

Patitsas, E., Berlin, J., Craig, M., and Easterbrook, S. (2016). Evidence that computer science grades are not bimodal. In ICER '16: Proceedings of the 2016 ACM Conference on International Computing Education Research, pages 113-121.

Smith, M. O., Giugliano, A., and DeOrio, A. (2018). Long term effects of pair programming. IEEE Transactions on Education, 61(3):187-194.

Ukrop, M. and Matyas, V. (2018). Why Johnny the Developer Can't Work with Public Key Certificates - An Experimental Study of OpenSSL Usability. In CT-RSA, volume 10808 of Lecture Notes in Computer Science, pages 45-64. Springer.

Weber, S., Coblenz, M., Myers, B., Aldrich, J., and Sunshine, J. (2017). Empirical studies on the security and usability impact of immutability. In 2017 IEEE Cybersecurity Development (SecDev), pages 50-53.

Werner, L., Hanks, B., and Mcdowell, C. (2004). Pair programming helps female computer science students. ACM Journal of Educational Resources in Computing, 4.

Williams, L., Kessler, R. R., Cunningham, W., and Jeffries, R. (2000). Strengthening the case for pair programming. IEEE Software, 17(4):19-25.

Williams, L. and Upchurch, R. (2001). In support of student pair-programming. In ACM SIGCSE Bulletin,(33), pages 327-331.

Williams, L., Wiebe, E., Yang, K., Ferzli, M., and Miller, C. (2002). In support of pair programming in the introductory computer science course. Computer Science Education, 12(3):197-212.

Zeier, A., Wiesmaier, A., and Heinemann, A. (2019). API Usability of Stateful Signature Schemes, pages 221-240. Springer.
How to Cite
Sharman, J., Acemyan, C., Kortum, P., & Wallach, D. (2021). Bad Tools Hurt: Lessons for teaching computer security skills to undergraduates. International Journal of Computer Science Education in Schools, 5(2), 74-92.