Bad Tools Hurt: Lessons for teaching computer security skills to undergraduates

  • Jonathan Sharman Rice University
  • Claudia Acemyan
  • Philip Kortum rice university
  • Dan Wallach Rice University
Keywords: security, coding, teaching, usability, tools

Abstract

Understanding why developers continue to misuse security tools is critical to designing safer software, yet the underlying reasons developers fail to write secure code are not well understood. In order to better understand how to teach these skills, we conducted two comparatively large-scale usability studies with undergraduate CS students to assess factors that affect success rates in securing web applications against cross-site request forgery (CSRF) attacks. First, we examined the impact of providing students with example code and/or a testing tool. Next, we examined the impact of working in pairs. We found that access to relevant secure code samples gave significant benefit to security outcomes. However, access to the tool alone had no significant effect on security outcomes, and surprisingly, the same held true for the tool and example code combined. These results confirm the importance of quality example code and demonstrate the potential danger of using security tools in the classroom that have not been validated for usability. No individual differences predicted one’s ability to complete the task. We also found that working in pairs had a significant positive effect on security outcomes. These results provide useful directions for teaching computer security programming skills to undergraduate students.

Downloads

Download data is not yet available.

References

Acar, Y., Backes, M., Fahl, S., Garfinkel, S., Kim, D., Mazurek, M. L., and Stransky, C. (2017). Comparing the Usability of Cryptographic APIs. In 2017 IEEE Symposium on Security and Privacy (SP), pages 154-171.
https://doi.org/10.1109/SP.2017.52


Acar, Y., Backes, M., Fahl, S., Kim, D., Mazurek, M. L., and Stransky, C. (2016). You Get Where You're Looking for: The Impact of Information Sources on Code Security. In 2016 IEEE Symposium on Security and Privacy (SP), pages 289-305. https://doi.org/10.1109/SP.2016.25


Acar, Y., Stransky, C., Wermke, D., Mazurek, M. L., and Fahl, S. (2017). Security Developer Studies with GitHub Users: Exploring a Convenience Sample. In Thirteenth Symposium on Usable Privacy and Security (SOUPS 2017), pages 81-95, Santa Clara, CA. USENIX Association. https://www.usenix.org/system/files/conference/soups2017/soups2017-acar.pdf


Arisholm, E., Gallis, H., Dyba, T., and Sjoberg, D. I. K. (2007). Evaluating pair programming with respect to system complexity and programmer expertise. IEEE Transactions on Software Engineering, 33(2):65-86.
https://doi.org/10.1109/TSE.2007.17


Bangor, A., Kortum, P., and Miller, J. (2009). Determining what individual sus scores mean: Adding an adjective rating scale. Journal of Usability Studies, 4(3):114-123. http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.177.1240&rep=rep1&type=pdf


Bangor, A., Kortum, P. T., and Miller, J. T. (2008). An empirical evaluation of the system usability scale. International Journal of Human-Computer Interaction, 24(6):574-594. https://doi.org/10.1080/10447310802205776


Begel, A. and Nagappan, N. (2008). Pair programming: What's in it for me? In Proceedings of the Second ACMIEEE International Symposium on Empirical Software Engineering and Measurement, ESEM '08, page 120-128, New York, NY, USA. Association for Computing Machinery. https://doi.org/10.1145/1414004.1414026


Bernhard, M., Sharman, J., Acemyan, C. Z., Kortum, P., Wallach, D. S., and Halderman, J. A. (2019). On the Usability of HTTPS Deployment. In Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems, CHI '19, pages 310:1-310:10, New York, NY, USA. ACM. https://doi.org/10.1145/3290605.3300540


Brooke, J. (1996). SUS: A quick and dirty usability scale. Usability evaluation in industry, page 189.

Eierman, M. and Iversen, J. (2018). Comparing testdriven development and pair programming to improve the learning of programming languages. Journal of the Midwest Association for Information Systems, 2018:23-36.
https://doi.org/10.17705/3jmwa.000038


Fischer, F., Böttinger, K., Xiao, H., Stransky, C., Acar, Y., Backes, M., and Fahl, S. (2017). Stack Overflow Considered Harmful? The Impact of Copy Paste on Android Application Security. In 2017 IEEE Symposium on Security and Privacy, pages 121-136. https://doi.org/10.1109/SP.2017.31


Gorski, P. L., Iacono, L. L., Wermke, D., Stransky, C., Moeller, S., Acar, Y., and Fahl, S. (2018). Developers Deserve Security Warnings, Too: On the Effect of Integrated Security Advice on Cryptographic API Misuse. In SOUPS @ USENIX Security Symposium. https://www.usenix.org/system/files/conference/soups2018/soups2018-gorski.pdf


Hanks, B., Mcdowell, C., Draper, D., and Krnjajic, M. (2004). Program quality with pair programming in cs1. In ACM Sigcse Bulletin,(36), pages 176-180. https://doi.org/10.1145/1026487.1008043


Hannay, J., Dybå, T., Arisholm, E., and Sjøberg, D. (2009). The effectiveness of pair programming: A meta-analysis. Information and Software Technology, 51:1110-1122. https://doi.org/10.1016/j.infsof.2009.02.001


ISO 9241-11 (2018). Ergonomics of human-system interaction - Part 11: Usability: Definitions and concepts.

Kortum, P. and Acemyan, C. Z. (2013). How low can you go?: Is the system usability scale range restricted? Journal of Usability Studies, 9(1):14-24. http://tecfa.unige.ch/tecfa/maltt/ergo/1415/UtopiaPeriode4/articles/JUS_Kortum_November_2013.pdf


Krombholz, K., Mayer, W., Schmiedecker, M., and Weippl, E. (2017). 'I Have No Idea What I'm Doing' - On the Usability of Deploying HTTPS. In 26th USENIX Security Symposium (USENIX Security 17), pages 1339-1356, Vancouver, BC. USENIX Association. https://www.usenix.org/system/files/conference/usenixsecurity17/sec17-krombholz.pdf



Mcdowell, C., Werner, L., Bullock, H., and Fernald, J. (2002). The effects of pair-programming on performance in an introductory programming course. In ACM SIGCSE Bulletin,(34), pages 38-42. https://doi.org/10.1145/563340.563353


Mcdowell, C., Werner, L., Bullock, H. E., and Fernald, J. (2003). The impact of pair programming on student performance, perception and persistence. In 25th International Conference on Software Engineering, 2003, pages 602-607.
https://doi.org/10.1109/ICSE.2003.1201243


McDowell, C., Werner, L., Bullock, H. E., and Fernald, J. (2006). Pair programming improves student retention, confidence, and program quality. Communications of the ACM, 49(8):90-95. https://doi.org/10.1145/1145287.1145293


Mindermann, K. and Wagner, S. (2018). Usability and Security Effects of Code Examples on Crypto APIs - CryptoExamples: A platform for free, minimal, complete and secure crypto examples. 2018 16th Annual Conference on Privacy, Security and Trust (PST). https://doi.org/10.1109/PST.2018.8514203



Naiakshina, A., Danilova, A., Gerlitz, E., and Smith, M. (2020). On conducting security developer studies with cs students: Examining a password-storage study with cs students, freelancers, and company developers. In Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems, pages 1-13. https://doi.org/10.1145/3313831.3376791


Naiakshina, A., Danilova, A., Gerlitz, E., von Zezschwitz, E., and Smith, M. (2019). 'If You Want, I Can Store the Encrypted Password': A Password-Storage Field Study with Freelance Developers. In Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems, CHI '19, pages 140:1-140:12, New York, NY, USA. ACM.
https://doi.org/10.1145/3290605.3300370


Naiakshina, A., Danilova, A., Tiefenau, C., and Smith, M. (2018). Deception task design in developer password studies: Exploring a student sample. In Fourteenth Symposium on Usable Privacy and Security (SOUPS 2018), pages 297-313, Baltimore, MD. USENIX Association. https://www.usenix.org/system/files/conference/soups2018/soups2018-naiakshina.pdf


Oliveira, D. S., Lin, T., Rahman, M. S., Akefirad, R., Ellis, D., Perez, E., Bobhate, R., DeLong, L. A., Cappos, J., and Brun, Y. (2018). API Blindspots: Why Experienced Developers Write Vulnerable Code. In Fourteenth Symposium on Usable Privacy and Security (SOUPS 2018), pages 315- 328, Baltimore, MD. USENIX Association. https://www.usenix.org/system/files/conference/soups2018/soups2018-oliveira.pdf


Patitsas, E., Berlin, J., Craig, M., and Easterbrook, S. (2016). Evidence that computer science grades are not bimodal. In ICER '16: Proceedings of the 2016 ACM Conference on International Computing Education Research, pages 113-121. https://doi.org/10.1145/2960310.2960312


Smith, M. O., Giugliano, A., and DeOrio, A. (2018). Long term effects of pair programming. IEEE Transactions on Education, 61(3):187-194. https://doi.org/10.1109/TE.2017.2773024


Ukrop, M. and Matyas, V. (2018). Why Johnny the Developer Can't Work with Public Key Certificates - An Experimental Study of OpenSSL Usability. In CT-RSA, volume 10808 of Lecture Notes in Computer Science, pages 45-64. Springer. https://doi.org/10.1007/978-3-319-76953-0_3


Weber, S., Coblenz, M., Myers, B., Aldrich, J., and Sunshine, J. (2017). Empirical studies on the security and usability impact of immutability. In 2017 IEEE Cybersecurity Development (SecDev), pages 50-53. https://doi.org/10.1109/SecDev.2017.21


Werner, L., Hanks, B., and Mcdowell, C. (2004). Pair programming helps female computer science students. ACM Journal of Educational Resources in Computing, 4. https://doi.org/10.1145/1060071.1060075


Williams, L., Kessler, R. R., Cunningham, W., and Jeffries, R. (2000). Strengthening the case for pair programming. IEEE Software, 17(4):19-25. https://doi.org/10.1109/52.854064


Williams, L. and Upchurch, R. (2001). In support of student pair-programming. In ACM SIGCSE Bulletin,(33), pages 327-331. https://doi.org/10.1145/366413.364614


Williams, L., Wiebe, E., Yang, K., Ferzli, M., and Miller, C. (2002). In support of pair programming in the introductory computer science course. Computer Science Education, 12(3):197-212. https://doi.org/10.1076/csed.12.3.197.8618


Zeier, A., Wiesmaier, A., and Heinemann, A. (2019). API Usability of Stateful Signature Schemes, pages 221-240. Springer.
https://doi.org/10.1007/978-3-030-26834-3_13
Published
2021-12-17
How to Cite
Sharman, J., Acemyan, C., Kortum, P., & Wallach, D. (2021). Bad Tools Hurt: Lessons for teaching computer security skills to undergraduates. International Journal of Computer Science Education in Schools, 5(2), 74-92. https://doi.org/10.21585/ijcses.v5i2.131